@denisixnpm/agent-rdp 0.7.10 → 0.7.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -8,121 +8,76 @@
8
8
 
9
9
  A CLI tool for AI agents to control Windows Remote Desktop sessions, built on [IronRDP](https://github.com/Devolutions/IronRDP).
10
10
 
11
+ Screenshots, mouse/keyboard input, clipboard sync, drive mapping, verified file
12
+ transfer, Windows UI Automation, OCR text location, and JSON output for every
13
+ command — across named sessions with automatic daemon lifecycle.
14
+
11
15
  ## Demo
12
16
 
13
17
  Claude Code automating SQLite database and table creation via RDP:
14
18
 
15
19
  https://github.com/user-attachments/assets/91892b39-4edb-412b-b265-55ccd75d7421
16
20
 
17
- ## Features
18
-
19
- - **Connect to RDP servers** - Full RDP protocol support with TLS and CredSSP authentication
20
- - **Take screenshots** - Capture the remote desktop as PNG or JPEG
21
- - **Mouse control** - Click, double-click, right-click, drag, scroll
22
- - **Keyboard input** - Type text, press key combinations (Ctrl+C, Alt+Tab, etc.)
23
- - **Clipboard sync** - Copy/paste text between local machine and remote Windows
24
- - **Drive mapping** - Map local directories as network drives on the remote machine
25
- - **UI Automation** - Interact with Windows applications via accessibility API (click, select, toggle, expand)
26
- - **OCR text location** - Find text on screen using OCR when UI Automation isn't available
27
- - **JSON output** - Structured output for AI agent consumption
28
- - **Session management** - Multiple named sessions with automatic daemon lifecycle
29
-
30
21
  ## Installation
31
22
 
32
- ### From npm
33
-
34
- ```bash
35
- npm install -g @denisixnpm/agent-rdp
36
- ```
37
-
38
- ### As a Claude Code skill
39
-
40
23
  ```bash
41
- npx add-skill https://github.com/denisix/agent-rdp
24
+ npm install -g @denisixnpm/agent-rdp # CLI
25
+ npx add-skill https://github.com/denisix/agent-rdp # as a Claude Code skill
42
26
  ```
43
27
 
44
- ### From a GitHub release
45
-
46
- Each release attaches a standalone binary per platform, plus the OCR models as
47
- `agent-rdp-models.tar.gz` / `agent-rdp-models.zip`. The models are
48
- architecture-independent, so they ship once rather than inside every archive.
49
-
50
- The binary alone covers everything except `locate`; that command needs the
51
- models, so extract them too and point `AGENT_RDP_MODELS_DIR` at them.
28
+ <details>
29
+ <summary>From a GitHub release (standalone binary)</summary>
52
30
 
53
- **macOS / Linux**
31
+ Each release attaches a binary per platform plus the OCR models as
32
+ `agent-rdp-models.tar.gz` / `.zip`. The models are architecture-independent, so
33
+ they ship once. The binary alone covers everything except `locate`, which needs
34
+ the models — extract them and point `AGENT_RDP_MODELS_DIR` at them.
54
35
 
55
36
  ```bash
56
37
  tar -xzf agent-rdp-linux-x64.tar.gz -C ~/.local/bin # or agent-rdp-darwin-arm64.tar.gz
57
38
  chmod +x ~/.local/bin/agent-rdp
58
39
  mkdir -p ~/.local/share/agent-rdp/models
59
40
  tar -xzf agent-rdp-models.tar.gz -C ~/.local/share/agent-rdp/models
60
- export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc to persist
41
+ export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
61
42
  ```
62
43
 
63
- On macOS the release binaries are unsigned, so Gatekeeper quarantines anything
64
- downloaded from a browser. If you get *"cannot be opened because the developer
65
- cannot be verified"*, clear the flag:
66
-
67
- ```bash
68
- xattr -d com.apple.quarantine ~/.local/bin/agent-rdp
69
- ```
70
-
71
- **Windows (PowerShell)**
72
-
73
44
  ```powershell
74
45
  Expand-Archive agent-rdp-win32-x64.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp"
75
46
  Expand-Archive agent-rdp-models.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp\models"
76
- # persist for future sessions (restart the terminal afterwards)
77
- setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models"
78
- ```
79
-
80
- Installing from npm needs none of this — the models ship inside
81
- `@denisixnpm/agent-rdp` and are located automatically.
82
-
83
- ### From source
84
-
85
- ```bash
86
- git clone https://github.com/denisix/agent-rdp
87
- cd agent-rdp
88
- bun install
89
- bun run build # Build native binary
90
- bun run build:ts # Build TypeScript
47
+ setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models" # restart the terminal after
91
48
  ```
92
49
 
93
- ## Using with AI Coding Agents
50
+ macOS release binaries are unsigned. If Gatekeeper blocks it with *"cannot be
51
+ opened because the developer cannot be verified"*:
52
+ `xattr -d com.apple.quarantine ~/.local/bin/agent-rdp`
94
53
 
95
- ### Claude Code
54
+ Installing from npm needs none of this — the models ship in the package.
55
+ </details>
96
56
 
97
- One command — installs the [SKILL.md](skills/agent-rdp/SKILL.md) workflow so Claude knows the commands, flags, and gotchas without you explaining them:
57
+ <details>
58
+ <summary>From source</summary>
98
59
 
99
60
  ```bash
100
- npx add-skill https://github.com/denisix/agent-rdp
61
+ git clone https://github.com/denisix/agent-rdp && cd agent-rdp
62
+ bun install
63
+ bun run build # native binary
64
+ bun run build:ts # TypeScript
101
65
  ```
66
+ </details>
102
67
 
103
- You don't need to install the CLI separately: the skill installs
104
- `@denisixnpm/agent-rdp` on first use if it isn't already on PATH.
105
-
106
- Or install manually:
107
-
108
- ```bash
109
- mkdir -p .claude/skills/agent-rdp
110
- curl -o .claude/skills/agent-rdp/SKILL.md \
111
- https://raw.githubusercontent.com/denisix/agent-rdp/main/skills/agent-rdp/SKILL.md
112
- ```
68
+ ## Using with AI coding agents
113
69
 
114
- Then just ask Claude Code, in plain language:
70
+ **Claude Code** `npx add-skill https://github.com/denisix/agent-rdp` installs
71
+ the [SKILL.md](skills/agent-rdp/SKILL.md) workflow so Claude knows the commands,
72
+ flags and gotchas without you explaining them. It installs the CLI on first use
73
+ if it isn't on PATH. Then ask in plain language:
115
74
 
116
75
  ```
117
76
  Connect to 192.168.1.100 as Administrator (password: secret), open Notepad,
118
77
  type "hello from Claude", and take a screenshot.
119
78
  ```
120
79
 
121
- Claude will run the underlying `agent-rdp connect`, `automate run`, `keyboard type`, and `screenshot` commands on its own.
122
-
123
- ### Codex
124
-
125
- Codex doesn't have a skill-install mechanism, but it reads `AGENTS.md` for project instructions. Point it at this tool by adding a section to your `AGENTS.md`:
80
+ **Codex** reads `AGENTS.md` instead. Add a section pointing at the tool:
126
81
 
127
82
  ```bash
128
83
  cat >> AGENTS.md <<'EOF'
@@ -135,414 +90,237 @@ https://github.com/denisix/agent-rdp for the full command reference.
135
90
  EOF
136
91
  ```
137
92
 
138
- Then prompt Codex the same way:
139
-
140
- ```
141
- codex "Connect to the Windows VM at 192.168.1.100 (user Administrator, password
142
- secret) using agent-rdp, open the Run dialog, launch calc.exe, and confirm it's
143
- open with a screenshot."
144
- ```
145
-
146
- Codex will call `agent-rdp` as a regular shell command, same as any other CLI tool.
147
-
148
93
  ## Usage
149
94
 
150
- ### Connect to an RDP Server
95
+ ### Connect
151
96
 
152
97
  ```bash
153
- # Using command line (password visible in process list - not recommended)
154
98
  agent-rdp connect --host 192.168.1.100 --username Administrator --password 'secret'
155
99
 
156
- # Using environment variables (recommended)
157
- export AGENT_RDP_USERNAME=Administrator
158
- export AGENT_RDP_PASSWORD=secret
100
+ # Environment variables (recommended - keeps the password out of the process list)
101
+ export AGENT_RDP_USERNAME=Administrator AGENT_RDP_PASSWORD=secret
159
102
  agent-rdp connect --host 192.168.1.100
160
103
 
161
- # Using stdin (most secure)
162
- echo 'secret' | agent-rdp connect --host 192.168.1.100 --username Administrator --password-stdin
104
+ # stdin (most secure)
105
+ echo 'secret' | agent-rdp connect --host 192.168.1.100 -u Administrator --password-stdin
106
+
107
+ agent-rdp disconnect
163
108
  ```
164
109
 
165
- ### Take a Screenshot
110
+ ### Screenshot
166
111
 
167
112
  ```bash
168
- # Save to file (default: ./screenshot.png)
169
- agent-rdp screenshot --output desktop.png
113
+ agent-rdp screenshot --output desktop.png # default ./screenshot.png
114
+ agent-rdp --json screenshot --output desktop.png # metadata; image always goes to disk
115
+ agent-rdp screenshot --region 100,380,600,30 -o row.png # crop; reports the offset back
116
+ ```
170
117
 
171
- # JSON metadata (path/width/height image is always written to disk)
172
- agent-rdp --json screenshot --output desktop.png
118
+ A screenshot is the last frame the server painted, not a live poll. Each one
119
+ reports `frame_age_ms` (time since the server last sent anything), and `--json`
120
+ adds `frame_seq` and `frame_hash` — two screenshots with the same value are
121
+ guaranteed pixel-identical, which is how you confirm a frame actually changed
122
+ after an action rather than hashing the saved file yourself. A large
123
+ `frame_age_ms` usually just means an idle desktop, since RDP servers send
124
+ nothing when nothing changes; a genuinely dead connection is detected within
125
+ seconds by TCP keepalive and reported as disconnected.
173
126
 
174
- # Capture just part of the screen (X,Y,WIDTH,HEIGHT in screen pixels).
175
- # The result reports the offset, so a coordinate read off the crop can be
176
- # turned back into a clickable one.
177
- agent-rdp screenshot --region 100,380,600,30 --output row.png
178
- ```
127
+ > The CLI has no `screenshot --base64`. In Node, `rdp.screenshot({ path })`
128
+ > writes to disk and returns `{ path, width, height }` prefer it over the
129
+ > base64-returning form when the caller doesn't need the bytes, since echoing a
130
+ > base64 image into an LLM context is expensive.
179
131
 
180
- > Note: the CLI no longer has `screenshot --base64`. For agent pipelines, write a file and encode it yourself, or use the Node.js API's `rdp.screenshot({ path })`, which writes to disk and returns `{ path, width, height }` without materializing base64 — prefer this over the default `rdp.screenshot()` (which returns `{ base64, width, height }`) when the caller doesn't need the raw bytes, since echoing a base64 image into an LLM context is expensive.
181
-
182
- Every screenshot also reports `frame_age_ms` (`--json`) / a printed warning
183
- (human mode, when large): milliseconds since the RDP server last sent any
184
- data. An idle-but-alive desktop can legitimately sit at a large age — RDP
185
- servers send nothing when nothing changes — but combined with `agent-rdp
186
- session info`'s `last_frame_age_ms`, a value that never resets on an
187
- otherwise-idle session is the signal to check the connection rather than
188
- trust the frame. The daemon also enables TCP keepalive on the RDP socket, so
189
- a genuinely dead connection (cable pull, black-holed network path) is now
190
- detected and disconnected within seconds instead of the OS's own multi-minute
191
- retransmission timeout — previously `screenshot` could keep returning a
192
- stale, byte-identical frame under a fresh filename for the entire time the
193
- transport was silently dead.
194
-
195
- `--json` output (and the Node SDK's `frameSeq`/`frameHash`) also carries a
196
- `frame_seq` (framebuffer generation counter) and `frame_hash` (FNV-1a hash of
197
- the captured pixels). Two screenshots with the same seq or hash are
198
- guaranteed pixel-identical — proving that directly, instead of hashing the
199
- saved file yourself or reading a clock rendered on screen, is the reliable
200
- way to confirm a frame actually changed after an action that should have
201
- repainted it.
202
-
203
- ### Getting Coordinates Right
132
+ ### Getting coordinates right
204
133
 
205
134
  **Never estimate a coordinate by looking at a screenshot.** Screenshot pixels,
206
- OCR boxes and click coordinates are all the same space, so a coordinate that
207
- comes out of the tool is exact — but one guessed from an image is not. Images
208
- get downscaled on their way into a vision model, and a click that is 30px off
209
- lands on the wrong row, the wrong cell, or a search box.
135
+ OCR boxes and click coordinates share one space, so a coordinate the tool
136
+ returns is exact — one guessed from an image is not. Images get downscaled on
137
+ their way into a vision model, and a click 30px off lands on the wrong row.
210
138
 
211
139
  Ask for the target by name and let the tool find it:
212
140
 
213
141
  ```bash
214
- # Click text, without a coordinate ever passing through your hands
215
- agent-rdp locate "Добавить" --click
216
-
217
- # Several matches? Choosing is explicit - it will not guess for you
218
- agent-rdp locate "Добавить" --click --index 1
219
-
220
- # Click a button by accessibility selector (preferred when UIA can see it)
221
- agent-rdp automate click "#SaveButton"
222
- ```
223
-
224
- To verify a value in place, read the region rather than cropping an image
225
- yourself — coordinates still come back in full-screen space:
226
-
227
- ```bash
228
- # Check one table row
229
- agent-rdp locate --all --region 100,380,600,30
230
- agent-rdp screenshot --region 100,380,600,30 -o row.png
231
-
232
- # Confirm which field the keyboard is actually in before typing
233
- agent-rdp automate focused
142
+ agent-rdp locate "Добавить" --click # coordinate never passes through your hands
143
+ agent-rdp locate "Добавить" --click --index 1 # several matches: choosing is explicit
144
+ agent-rdp automate click "#SaveButton" # preferred when UI Automation can see it
145
+ agent-rdp automate focused # confirm which field has focus before typing
234
146
  ```
235
147
 
236
148
  `agent-rdp mouse click X Y` remains available for coordinates you got from
237
149
  `locate`, `automate get`, or a deliberate calculation.
238
150
 
239
- ### Mouse Operations
151
+ ### Mouse and keyboard
240
152
 
241
153
  ```bash
242
- # Click at position
243
154
  agent-rdp mouse click 500 300
244
-
245
- # Right-click
246
155
  agent-rdp mouse right-click 500 300
247
-
248
- # Double-click
249
156
  agent-rdp mouse double-click 500 300
250
-
251
- # Move cursor
252
157
  agent-rdp mouse move 100 200
253
-
254
- # Drag from (100,100) to (500,500)
255
158
  agent-rdp mouse drag 100 100 500 500
256
- ```
257
-
258
- ### Keyboard Operations
259
159
 
260
- ```bash
261
- # Type text (supports Unicode)
262
- agent-rdp keyboard type "Hello, World!"
263
-
264
- # Paste instead of type for long or non-Latin text: clipboard + Ctrl+V as one
265
- # command. More reliable than `type` - it cannot lose individual keystrokes,
266
- # and there's no gap between setting the clipboard and pasting for focus to
267
- # move in.
268
- agent-rdp keyboard paste "Привет, мир! A very long string types can drop pieces of."
269
-
270
- # Press key combinations
271
- agent-rdp keyboard press "ctrl+c"
272
- agent-rdp keyboard press "alt+tab"
273
- agent-rdp keyboard press "ctrl+shift+esc"
274
-
275
- # Press single keys (use press command)
276
- agent-rdp keyboard press enter
277
- agent-rdp keyboard press escape
278
- agent-rdp keyboard press f5
279
-
280
- # Hold a key across other commands (shift-click, hold-and-drag, ...)
281
- agent-rdp keyboard down shift
282
- agent-rdp mouse click 200 300
160
+ agent-rdp keyboard type "Hello, World!" # Unicode, batched into one round-trip
161
+ agent-rdp keyboard type "text" --delay 20 # pace it if the remote app drops fast input
162
+ agent-rdp keyboard paste "Привет, мир!" # clipboard + Ctrl+V as one command
163
+ agent-rdp keyboard press "ctrl+c" # also alt+tab, enter, escape, f5, win+r
164
+ agent-rdp keyboard down shift # hold across other commands
283
165
  agent-rdp keyboard up shift
284
166
  ```
285
167
 
286
- ### Scroll
168
+ Prefer `keyboard paste` for long or non-Latin text: it cannot lose individual
169
+ keystrokes, and leaves no gap for focus to move between setting the clipboard
170
+ and pasting.
287
171
 
288
- Position defaults to the **screen center**, not whatever pane you're working
289
- in — pass `--at` to target a specific window or control.
172
+ ### Scroll
290
173
 
291
- Amount is a positional argument, not `--amount`:
174
+ Amount is positional (not `--amount`), and the default point is the **screen
175
+ center** rather than whatever pane you're working in — use `--at` to target one.
292
176
 
293
177
  ```bash
294
178
  agent-rdp scroll up 3
295
- agent-rdp scroll down 5
296
- agent-rdp scroll left
297
- agent-rdp scroll right
298
179
  agent-rdp scroll down 5 --at 600 400
180
+ agent-rdp scroll left # also: right
299
181
  ```
300
182
 
301
- ### Locate (OCR)
302
-
303
- Find text on screen using OCR (powered by [ocrs](https://github.com/robertknight/ocrs)). Useful when UI Automation can't access certain elements (WebView content, some dialogs).
183
+ ### Clipboard and drive mapping
304
184
 
305
185
  ```bash
306
- # Find lines containing text
307
- agent-rdp locate "Cancel"
308
-
309
- # Pattern matching (glob-style)
310
- agent-rdp locate "Save*" --pattern
311
-
312
- # Exact whole-line matching. Default mode is substring containment, so
313
- # "Провести" also matches a button reading "Провести и закрыть" - two matches,
314
- # and --click refuses to guess between them. --exact matches only the line
315
- # that reads exactly "Провести".
316
- agent-rdp locate "Провести" --exact --click
317
-
318
- # Get all text on screen
319
- agent-rdp locate --all
320
-
321
- # JSON output
322
- agent-rdp locate "OK" --json
323
-
324
- # Click the match directly (see "Getting Coordinates Right" above)
325
- agent-rdp locate "Cancel" --click
326
- agent-rdp locate "Cancel" --double-click
327
- agent-rdp locate "Cancel" --right-click
328
-
329
- # Search only part of the screen. Results stay in full-screen coordinates,
330
- # so --region composes with --click. A tight region also reads small text
331
- # more reliably than a full-screen pass.
332
- agent-rdp locate --all --region 100,380,600,30
333
-
334
- # Block until text appears (e.g. a dialog finishing its animation), instead
335
- # of polling `locate` in a loop from the outside. Composes with --click.
336
- agent-rdp locate "OK" --wait 10000 --click
337
-
338
- # Constrain matches to those near a distinctive anchor label - useful when
339
- # the same text repeats on screen (a column header in every row, a label
340
- # that also appears in a tooltip). Anchor is matched by substring; if it
341
- # isn't found at all, this returns zero matches rather than an error.
342
- agent-rdp locate "Отменить" --near "Заказ №001" --click
343
- ```
344
-
345
- Returns text lines with coordinates for clicking:
346
- ```
347
- Found 1 line(s) containing 'Cancel':
348
- 'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
186
+ agent-rdp clipboard set "Hello from CLI"
187
+ agent-rdp clipboard set --file ./script.ps1 # or `--file -` to read stdin
188
+ agent-rdp clipboard get
349
189
 
350
- To click it: agent-rdp locate 'Cancel' --click
190
+ # Drives must be mapped at connect time; multiple --drive flags are allowed
191
+ agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
192
+ --drive /home/user/documents:Documents --drive /tmp/shared:Shared
193
+ agent-rdp drive list
351
194
  ```
352
195
 
353
- Clicking is deliberately strict: no match, or several matches without
354
- `--index`, is an error rather than a guess. Prefer narrowing with `--region`
355
- over picking an index. When two buttons share a prefix ("Провести" /
356
- "Провести и закрыть"), use `--exact` so the ambiguity never arises.
196
+ Mapped drives appear on the remote as network locations. Clipboard text is
197
+ sent with CRLF line endings, so multi-line content survives
198
+ `Get-Clipboard | Set-Content`; `clipboard get` returns Windows text with its
199
+ CRLF intact.
357
200
 
358
- **Numbers with thousands separators can lose their leading digit group** —
359
- OCR has been observed reading `1 250,00` as `2250,00`, dropping the `1` and
360
- the separating space. For monetary values, crop tight with `--region` before
361
- reading, and verify with `automate get` (when UI Automation can reach the
362
- field) or a second independent read before acting on the amount.
201
+ > **Do not access `\\TSCLIENT\...` from `automate run`.** Drive redirection is
202
+ > serviced by the same task that carries the automation channel, so reading the
203
+ > share from inside the agent deadlocks the command never returns and the
204
+ > session stops responding until you reconnect. Use `file push`/`file pull`.
363
205
 
364
- ### Click-at (safe clicking of externally-computed coordinates)
206
+ ### File transfer
365
207
 
366
- When the click point comes from outside agent-rdp a vision model reading a
367
- screenshot, a manual crop `locate --click` can't help, and a raw
368
- `mouse click X Y` has no safety net. `click-at` clicks the point only if it
369
- isn't ambiguously close to more than one detected text region:
208
+ Requires `--enable-win-automation`. Copies in chunks over the automation
209
+ channel, with a SHA-256 computed independently on each end, so a truncated or
210
+ corrupted transfer fails loudly instead of leaving a plausible-looking file.
370
211
 
371
212
  ```bash
372
- # Click, refusing if another label is within 10px of the target
373
- agent-rdp click-at 665 209
374
-
375
- # Tune the detection window and gap
376
- agent-rdp click-at 665 209 --window 400x160 --min-gap 20
377
-
378
- # Variants
379
- agent-rdp click-at 665 209 --double-click
380
- agent-rdp click-at 665 209 --right-click
381
-
382
- # Cross-check: a second independent measurement for the same target (e.g. a
383
- # vision model queried twice). Clicks the midpoint if the two agree within
384
- # --max-divergence (default 40px); refuses if they don't. Formalizes the
385
- # "click the intersection of two measurements" technique that catches a
386
- # single call's misreads.
387
- agent-rdp click-at 665 209 --confirm 670,212
388
- agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
213
+ agent-rdp file push ./report.xlsx "C:\\Users\\Admin\\report.xlsx"
214
+ agent-rdp file pull "C:\\Users\\Admin\\export.csv" ./export.csv
389
215
  ```
390
216
 
391
- The check uses OCR *detection* only (bounding boxes, script-agnostic), not
392
- recognition so it works even for text OCR can't read, e.g. custom-rendered
393
- Cyrillic UIs where both UI Automation and `locate` text search fail. On
394
- refusal it lists the nearby regions and exits non-zero; nothing is clicked.
395
-
396
- ### Clipboard
217
+ Transfers are byte-exact, which also makes this the reliable way to place a
218
+ script with non-ASCII content on the remote writing one through the clipboard
219
+ or `Add-Content` re-encodes it and mangles anything outside ASCII. Limit: 128MB.
397
220
 
398
- ```bash
399
- # Set clipboard text (available when you paste on Windows)
400
- agent-rdp clipboard set "Hello from CLI"
221
+ ### Locate (OCR)
401
222
 
402
- # Get clipboard text (after copying on Windows)
403
- agent-rdp clipboard get
223
+ Finds text on screen with [ocrs](https://github.com/robertknight/ocrs). Use it
224
+ when UI Automation can't reach an element (WebView content, some dialogs).
404
225
 
405
- # With JSON output
406
- agent-rdp --json clipboard get
226
+ ```bash
227
+ agent-rdp locate "Cancel" # substring match, returns coordinates
228
+ agent-rdp locate "Save*" --pattern # glob match
229
+ agent-rdp locate "Провести" --exact # whole-line match
230
+ agent-rdp locate --all # all text on screen
231
+ agent-rdp locate "Cancel" --click # also --double-click, --right-click
232
+ agent-rdp locate "OK" --wait 10000 --click # block until it appears, then click
233
+ agent-rdp locate --all --region 100,380,600,30 # coords stay full-screen
234
+ agent-rdp locate "Отменить" --near "Заказ №001" --click # anchor to a nearby label
407
235
  ```
408
236
 
409
- ### Drive Mapping
410
-
411
- Map local directories as network drives on the remote Windows machine. Drives must be mapped at connect time. Multiple drives can be specified.
412
-
413
- ```bash
414
- # Map local directories during connection
415
- agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
416
- --drive /home/user/documents:Documents \
417
- --drive /tmp/shared:Shared
237
+ Output carries coordinates ready to click:
418
238
 
419
- # List mapped drives
420
- agent-rdp drive list
239
+ ```
240
+ Found 1 line(s) containing 'Cancel':
241
+ 'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
421
242
  ```
422
243
 
423
- On the remote Windows machine, mapped drives appear in File Explorer as network locations.
244
+ Clicking is deliberately strict: no match, or several matches without
245
+ `--index`, is an error rather than a guess. Default matching is substring
246
+ containment, so `"Провести"` also matches `"Провести и закрыть"` — use
247
+ `--exact` to avoid the ambiguity, `--near` when the same text genuinely repeats
248
+ across rows, or narrow with `--region`.
424
249
 
425
- > **Do not access `\\TSCLIENT\...` from `automate run`.** Drive redirection is
426
- > serviced by the same task that carries the automation channel, so reading
427
- > the share from inside the agent deadlocks: the command never returns and the
428
- > session stops responding until you reconnect. Use `file push`/`file pull`
429
- > below to move files instead.
250
+ **Numbers with thousands separators can lose their leading digit group** — OCR
251
+ has been observed reading `1 250,00` as `2250,00`. For monetary values, crop
252
+ tight with `--region` and verify with `automate get` or a second independent
253
+ read before acting on the amount.
430
254
 
431
- ### File Transfer
255
+ ### Click-at (safe clicking of externally-computed coordinates)
432
256
 
433
- Copies files in verified chunks over the automation channel requires
434
- `--enable-win-automation` at connect time. Both directions verify a SHA-256
435
- computed independently on each end, so a truncated or corrupted transfer
436
- fails loudly instead of leaving a plausible-looking file behind.
257
+ When the click point comes from outside agent-rdpa vision model reading a
258
+ screenshot, a manual crop `locate --click` can't help and a raw `mouse click`
259
+ has no safety net. `click-at` clicks only if the point isn't ambiguously close
260
+ to more than one detected text region:
437
261
 
438
262
  ```bash
439
- agent-rdp file push ./report.xlsx "C:\\Users\\Admin\\report.xlsx"
440
- agent-rdp file pull "C:\\Users\\Admin\\export.csv" ./export.csv
263
+ agent-rdp click-at 665 209
264
+ agent-rdp click-at 665 209 --window 400x160 --min-gap 20 # tune detection window/gap
265
+ agent-rdp click-at 665 209 --double-click # also --right-click
266
+
267
+ # Cross-check two independent measurements: clicks their midpoint if they agree
268
+ # within --max-divergence (default 40px), refuses if they don't.
269
+ agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
441
270
  ```
442
271
 
443
- Transfers are byte-exact, which also makes this the reliable way to place a
444
- script with non-ASCII content on the remote: writing one through the
445
- clipboard or `Add-Content` re-encodes it and mangles anything outside ASCII.
446
- The size limit is 128MB.
272
+ The check uses OCR *detection* only (bounding boxes, script-agnostic), not
273
+ recognition, so it works even for text OCR can't read — custom-rendered
274
+ Cyrillic UIs where both UI Automation and `locate` fail. On refusal it lists
275
+ the nearby regions and exits non-zero; nothing is clicked.
447
276
 
448
277
  ### UI Automation
449
278
 
450
- Interact with Windows applications programmatically via the Windows UI Automation API using native patterns (InvokePattern, SelectionItemPattern, TogglePattern, etc.). When enabled, a PowerShell agent is injected into the remote session that captures the accessibility tree and performs actions. Communication between the CLI and the agent uses a Dynamic Virtual Channel (DVC) for fast bidirectional IPC.
451
-
452
- For detailed documentation, see [AUTOMATION.md](https://github.com/denisix/agent-rdp/blob/main/docs/AUTOMATION.md).
279
+ Drives Windows applications through the UI Automation API using native patterns
280
+ (Invoke, SelectionItem, Toggle, ExpandCollapse). A PowerShell agent is injected
281
+ into the remote session and speaks to the daemon over a Dynamic Virtual Channel.
282
+ Full protocol details in [AUTOMATION.md](docs/AUTOMATION.md).
453
283
 
454
284
  ```bash
455
- # Connect with automation enabled
456
285
  agent-rdp connect --host 192.168.1.100 -u Admin -p secret --enable-win-automation
457
286
 
458
- # Take an accessibility tree snapshot (refs are always included)
459
- agent-rdp automate snapshot
460
-
461
- # Snapshot filtering options (like agent-browser)
462
- agent-rdp automate snapshot -i # Interactive elements only
463
- agent-rdp automate snapshot -c # Compact (remove empty structural elements)
464
- agent-rdp automate snapshot -d 3 # Limit depth to 3 levels
465
- agent-rdp automate snapshot -s "~*Notepad*" # Scope to a window/element
466
- agent-rdp automate snapshot -i -c -d 5 # Combine options
467
-
468
- # Which control has keyboard focus right now?
469
- # Use this after Tab/Enter to confirm where your keystrokes will go, instead
470
- # of typing into a cell you assume is focused.
471
- agent-rdp automate focused
472
- # edit 'Количество' = "5,000" at (110, 391) 80x18 [focusable]
473
-
474
- # Pattern-based element operations (refs use @eN format)
475
- agent-rdp automate click "#SaveButton" # Click button
476
- agent-rdp automate click "@e5" # Click by ref number from snapshot
477
- agent-rdp automate click "@e5" -d # Double-click (for file list items)
478
- agent-rdp automate select "@e10" # Select item (SelectionItemPattern)
479
- agent-rdp automate toggle "@e7" # Toggle checkbox (TogglePattern)
480
- agent-rdp automate expand "@e3" # Expand menu (ExpandCollapsePattern)
481
- agent-rdp automate context-menu "@e5" # Open context menu (Shift+F10)
482
-
483
- # Fill text fields
287
+ agent-rdp automate snapshot # full tree (refs always included)
288
+ agent-rdp automate snapshot -i -c -d 3 # interactive only, compact, depth 3
289
+ agent-rdp automate snapshot -s "~*Notepad*" # scope to a window
290
+ agent-rdp automate focused # what has keyboard focus right now
291
+
292
+ agent-rdp automate click "@e5" # also: "#SaveButton", ".Edit", "~*wild*"
293
+ agent-rdp automate click "@e5" -d # double-click
294
+ agent-rdp automate select "@e10" --item "Option 1"
295
+ agent-rdp automate toggle "@e7" --state on
296
+ agent-rdp automate expand "@e3" # also: collapse, context-menu, focus, clear
297
+ agent-rdp automate get "@e2" # includes Value: for text/multiline edits
484
298
  agent-rdp automate fill ".Edit" "Hello World"
299
+ agent-rdp automate scroll "@e4" --direction down --amount 3
300
+ agent-rdp automate wait-for "#SaveButton" --timeout 5000 --state visible
485
301
 
486
- # Window operations
487
302
  agent-rdp automate window list
488
303
  agent-rdp automate window focus "~*Notepad*"
304
+ agent-rdp automate window maximize|minimize|restore|close
489
305
 
490
- # Run PowerShell commands
491
- agent-rdp automate run "Get-Process" --wait
492
- agent-rdp automate run "Get-Process" --wait --process-timeout 5000 # With 5s timeout
493
- agent-rdp automate run "$PSVersionTable" --wait --shell pwsh.exe # Run through PowerShell 7 instead of Windows PowerShell
306
+ agent-rdp automate run "Get-Process" --wait --process-timeout 5000
307
+ agent-rdp automate run "$PSVersionTable" --wait --shell pwsh.exe
308
+ agent-rdp automate run "ping -t 127.0.0.1" --stream # returns a pid immediately
309
+ agent-rdp automate run-poll <pid> # drain output; repeat until exited
310
+ agent-rdp automate run "Add-Content C:\log.txt x" --wait --idempotency-key step-07
311
+ # a retry with the same key replays, never re-runs
494
312
 
495
- # Stream output from a long-running command instead of waiting for it to exit
496
- agent-rdp automate run "ping -t 127.0.0.1" --stream # Returns immediately with a pid
497
- agent-rdp automate run-poll <pid> # Repeat to drain output incrementally; reports exit once the process ends
313
+ agent-rdp automate status # uptime, last DVC round-trip, failure count
314
+ agent-rdp automate restart # relaunch the agent, keeping the RDP session
315
+ ```
498
316
 
499
- # Diagnostics: agent uptime, last DVC round-trip time, consecutive failures
500
- agent-rdp automate status
317
+ **Selectors:** `@e5` (snapshot ref), `#SaveButton` (automation ID), `.Edit`
318
+ (Win32 class), `~*pattern*` (wildcard name), `File` (exact name).
501
319
 
502
- # Relaunch the agent without a full RDP reconnect - use when the agent died
503
- # mid-session or never came up after connect, but the RDP session is fine.
504
- # Requires --enable-win-automation to have been passed at connect time.
505
- agent-rdp automate restart
506
- ```
320
+ **Snapshot format**, with `disabled` shown on interactive elements a free
321
+ pre-action state check (a disabled "Отменить проведение" tells you the document
322
+ isn't posted before you click anything):
507
323
 
508
- An `automation indeterminate` error means the DVC reply was lost, not that
509
- the action necessarily failed or succeeded. The agent now keeps the results of
510
- its recent requests, so when a reply goes missing the daemon asks it what
511
- actually happened and usually returns the real outcome instead - or, if the
512
- agent has no record of the request, says outright that it never ran and
513
- retrying is safe. Only when the agent is still busy does a genuine
514
- `indeterminate` remain; check state before retrying in that case, or a
515
- retried click/fill can apply twice. Read-only commands (`snapshot`, `get`,
516
- `status`, `wait-for`, `window list`) say so explicitly, since those can always
517
- be retried safely.
518
-
519
- **Long-running commands.** `run --wait` and `wait-for` are given the time
520
- they ask for: the transport deadline, the CLI's socket timeout and the
521
- watchdog all extend to cover `--process-timeout`/`--timeout`. Note the agent
522
- processes one command at a time, so a long `run --wait` blocks every other
523
- `automate` command until it finishes — for anything beyond about a minute,
524
- prefer `run --stream` plus `run-poll`, which returns immediately and lets you
525
- drain output while the session stays responsive.
526
-
527
- Snapshots include `disabled` on interactive elements — a free pre-action
528
- state check. A disabled "Отменить проведение" menu item, for example, tells
529
- you the document isn't posted before you ever click anything.
530
-
531
- **Keyboard navigation inside a panel (arrow keys + Enter) can land on the
532
- wrong item.** Observed in 1C side panels ("Функции" etc.): Up/Down selection
533
- followed by Enter is not reliably deterministic. Prefer `automate` refs when
534
- the panel is exposed to UI Automation; when it isn't, use two independent
535
- coordinate measurements and [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates)
536
- rather than arrow-key navigation.
537
-
538
- **Selector Types:**
539
- - `@e5` or `@5` - Reference number from snapshot (e prefix recommended)
540
- - `#SaveButton` - Automation ID
541
- - `.Edit` - Win32 class name
542
- - `~*pattern*` - Wildcard name match
543
- - `File` - Element name (exact match)
544
-
545
- **Snapshot Output Format:**
546
324
  ```
547
325
  - Window "Notepad" [ref=e1, id=Notepad]
548
326
  - MenuBar "Application" [ref=e2]
@@ -550,81 +328,111 @@ rather than arrow-key navigation.
550
328
  - Edit "Text Editor" [ref=e5, value="Hello"]
551
329
  ```
552
330
 
553
- ### Session Management
331
+ **`automation indeterminate`** means the reply was lost, not that the action
332
+ failed. The agent journals recent results, so the daemon asks what actually
333
+ happened and usually returns the real outcome — or reports that the request
334
+ never ran and is safe to retry. A surviving `indeterminate` means the agent is
335
+ still busy: check state before retrying, or a click/fill can apply twice.
336
+ Read-only commands (`snapshot`, `get`, `status`, `wait-for`, `window list`) say
337
+ so explicitly, since those are always safe to retry.
338
+
339
+ **Retrying `run` safely.** Give a mutating `run` an `--idempotency-key`. A
340
+ retry that reuses the key — after `indeterminate`, an IPC timeout or
341
+ `daemon_unresponsive` — gets the recorded result of the first execution back
342
+ (`replayed: true`) instead of running the command again; reusing a key for a
343
+ *different* command is refused (`idempotency_key_reused`). The journal is
344
+ per agent process (last 64 results) and empty after a reconnect, so after
345
+ `connect` verify the side effect instead of retrying blindly.
346
+
347
+ **`run` exit codes mean something.** The child runs with
348
+ `$ErrorActionPreference='Stop'`, so a cmdlet that fails non-terminatingly —
349
+ `Add-Content` to a locked file, `Set-Content` to a bad path — exits 1 with the
350
+ error on stderr instead of reporting success for a write that never happened.
351
+ A script that wants continue-on-error sets `$ErrorActionPreference='Continue'`
352
+ on its first line. Do not redirect `*>` into a file the script itself writes:
353
+ the child holds it open and the script's own writes fail with "being used by
354
+ another process" — `run --wait` captures output without a file, and
355
+ `run --stream` + `run-poll` covers long jobs.
356
+
357
+ **Long-running commands** get the time they ask for: the transport deadline,
358
+ the CLI socket timeout and the watchdog all extend to cover
359
+ `--process-timeout`/`--timeout`. But the agent handles one command at a time,
360
+ so a long `run --wait` blocks every other `automate` call — past about a
361
+ minute, prefer `run --stream` plus `run-poll`. Streamed output is captured to
362
+ files on the remote side, so nothing is lost if the process exits between
363
+ polls; a finished process stays pollable for 10 minutes and repeat polls return
364
+ `exited: true` with empty chunks. (If you redirect inside the command instead,
365
+ note that Windows PowerShell 5.1's `>` writes UTF-16LE.)
366
+
367
+ **`daemon_not_running` vs `daemon_unresponsive`.** The first means no daemon
368
+ process exists — reconnect, and `<session>/daemon.log` says why it exited. The
369
+ second means the process is alive but did not answer a health check within
370
+ 10s: it is busy (a long `run --wait`, a file transfer). Wait and retry; do not
371
+ reconnect, that discards a working session. The message includes the tail of
372
+ daemon.log. `connect` replaces a daemon that stays unresponsive, killing the
373
+ stuck one first.
374
+
375
+ **`daemon_version_mismatch`** means the daemon was started by a different
376
+ agent-rdp version than the CLI — it kept running across an upgrade, and is
377
+ still serving the old code, including the automation agent it embeds. Run
378
+ `agent-rdp connect ...` again: it replaces the daemon (and the SDK does the
379
+ same on its own). `session info` shows both versions. This is worth knowing
380
+ about because it is how "upgraded, but the old bug still reproduces" happens.
381
+
382
+ **Arrow-key navigation inside a panel can land on the wrong item.** Observed in
383
+ 1C side panels: Up/Down then Enter is not reliably deterministic. Prefer
384
+ `automate` refs; when the panel isn't exposed to UI Automation, use two
385
+ independent measurements with [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates).
386
+
387
+ ### Sessions and the web viewer
554
388
 
555
389
  ```bash
556
- # List active sessions
557
390
  agent-rdp session list
558
-
559
- # Get current session info
560
391
  agent-rdp session info
561
-
562
- # Close a session
563
- agent-rdp session close
564
-
565
- # Use a named session
566
- agent-rdp --session work connect --host work-pc.local ...
392
+ agent-rdp --session work connect --host work-pc.local ... # named session
567
393
  agent-rdp --session work screenshot
568
- ```
569
394
 
570
- ### Disconnect
395
+ agent-rdp wait 3000 # pause; useful right after connect, before the first input
571
396
 
572
- ```bash
573
- agent-rdp disconnect
397
+ # Web viewer (needs streaming enabled at connect)
398
+ agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
399
+ agent-rdp view --port 9224
574
400
  ```
575
401
 
576
- ### Web Viewer
402
+ There is no `session close` — use `disconnect`.
577
403
 
578
- Open the web-based viewer to see the remote desktop in your browser:
404
+ ### Diagnostics and bug reports
579
405
 
580
406
  ```bash
581
- # Open viewer (connects to default streaming port 9224)
582
- agent-rdp view
583
-
584
- # Specify a different port
585
- agent-rdp view --port 9224
407
+ agent-rdp diagnose # -> ./agent-rdp-diagnostics-<session>-<ts>.zip
408
+ agent-rdp diagnose --output report.zip
586
409
  ```
587
410
 
588
- The viewer requires WebSocket streaming to be enabled. Start a session with streaming:
411
+ The zip holds `daemon.log` (+ `.prev`), `transcript.jsonl` (one redacted line
412
+ per request: what was asked, outcome, duration), `diagnostics/` (failure
413
+ captures), a current screenshot, the remote automation agent's own log, and
414
+ `info.json` (versions, OS, daemon state, which `AGENT_RDP_*` variables are
415
+ set). It is built from disk first and the daemon second, so it works when the
416
+ daemon is dead or unresponsive — those parts are listed as skipped. The RDP
417
+ password is never logged, and is blanked if it appears anyway.
589
418
 
590
- ```bash
591
- agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
592
- agent-rdp view
593
- ```
419
+ Failure captures are automatic: when `locate` finds nothing, `click-at` refuses,
420
+ an `automate` command errors, or a waited `run` exits non-zero, the daemon saves
421
+ `diagnostics/<ts>-<kind>-<code>.png` plus a `.json` with the request, the error
422
+ and — for OCR misses — every line OCR *did* read. At most one capture per 5s,
423
+ the newest 20 kept. Set `AGENT_RDP_DIAGNOSTICS=0` to turn the transcript and
424
+ captures off.
594
425
 
595
- ## JSON Output
426
+ ## JSON output
596
427
 
597
- All commands support `--json` for structured output:
428
+ Add `--json` to any command:
598
429
 
599
- ```bash
600
- agent-rdp --json screenshot --output desktop.png
601
- ```
602
-
603
- **Success response:**
604
430
  ```json
605
- {
606
- "success": true,
607
- "data": {
608
- "type": "screenshot",
609
- "path": "desktop.png",
610
- "width": 1920,
611
- "height": 1080
612
- }
613
- }
431
+ { "success": true, "data": { "type": "screenshot", "path": "desktop.png", "width": 1920, "height": 1080 } }
432
+ { "success": false, "error": { "code": "not_connected", "message": "Not connected to an RDP server" } }
614
433
  ```
615
434
 
616
- **Error response:**
617
- ```json
618
- {
619
- "success": false,
620
- "error": {
621
- "code": "not_connected",
622
- "message": "Not connected to an RDP server"
623
- }
624
- }
625
- ```
626
-
627
- ## Environment Variables
435
+ ## Environment variables
628
436
 
629
437
  | Variable | Description |
630
438
  |----------|-------------|
@@ -634,173 +442,109 @@ agent-rdp --json screenshot --output desktop.png
634
442
  | `AGENT_RDP_PASSWORD` | RDP password |
635
443
  | `AGENT_RDP_SESSION` | Session name (default: "default") |
636
444
  | `AGENT_RDP_STREAM_PORT` | WebSocket streaming port (0 = disabled) |
637
- | `AGENT_RDP_MODELS_DIR` | Override the OCR models directory (set automatically by the npm wrapper; useful for standalone binary installs) |
445
+ | `AGENT_RDP_MODELS_DIR` | OCR models directory (set automatically by the npm wrapper; needed for standalone binary installs) |
446
+ | `AGENT_RDP_DIAGNOSTICS` | Set to `0` to disable the request transcript and failure captures |
638
447
 
639
448
  ## Node.js API
640
449
 
641
- Use agent-rdp programmatically from Node.js/TypeScript:
642
-
643
450
  ```typescript
644
451
  import { RdpSession } from 'agent-rdp';
645
452
 
646
453
  const rdp = new RdpSession({ session: 'default' });
647
-
648
454
  await rdp.connect({
649
- host: '192.168.1.100',
650
- username: 'Administrator',
651
- password: 'secret',
652
- width: 1280,
653
- height: 800,
455
+ host: '192.168.1.100', username: 'Administrator', password: 'secret',
456
+ width: 1280, height: 800,
654
457
  drives: [{ path: '/tmp/share', name: 'Share' }],
655
- enableWinAutomation: true, // Enable UI Automation
458
+ enableWinAutomation: true,
656
459
  });
657
460
 
658
- // Screenshot - prefer `path` so a large base64 string never has to be
659
- // held in memory or echoed into an agent's context
660
- const { path, width, height } = await rdp.screenshot({ format: 'png', path: 'screenshot.png' });
661
-
662
- // Or get raw base64 directly (e.g. for further in-process processing)
663
- const { base64 } = await rdp.screenshot({ format: 'png' });
461
+ // Screenshot - prefer `path` so a large base64 string is never held in memory
462
+ const { path, width, height } = await rdp.screenshot({ format: 'png', path: 'shot.png' });
463
+ const { base64 } = await rdp.screenshot({ format: 'png' }); // or raw, for in-process use
664
464
 
665
- // Mouse
666
- await rdp.mouse.click({ x: 100, y: 200 });
667
- await rdp.mouse.rightClick({ x: 100, y: 200 });
668
- await rdp.mouse.doubleClick({ x: 100, y: 200 });
669
- await rdp.mouse.move({ x: 150, y: 250 });
465
+ await rdp.mouse.click({ x: 100, y: 200 }); // also rightClick, doubleClick, move
670
466
  await rdp.mouse.drag({ from: { x: 100, y: 100 }, to: { x: 500, y: 500 } });
671
467
 
672
- // Keyboard
673
468
  await rdp.keyboard.type({ text: 'Hello World' });
674
- await rdp.keyboard.paste('Привет, мир!'); // Reliable for long/non-Latin text
469
+ await rdp.keyboard.paste('Привет, мир!'); // reliable for long/non-Latin text
675
470
  await rdp.keyboard.press({ keys: 'ctrl+c' });
676
- await rdp.keyboard.press({ keys: 'enter' }); // Single keys use press()
677
- await rdp.keyboard.down('shift'); // Hold across other commands
678
- await rdp.keyboard.up('shift');
679
-
680
- // Scroll
681
- await rdp.scroll.up(); // Default amount: 3
682
- await rdp.scroll.down({ amount: 5 }); // Custom amount
683
- await rdp.scroll.up({ x: 500, y: 300 }); // Scroll at position
471
+ await rdp.keyboard.down('shift'); await rdp.keyboard.up('shift');
684
472
 
685
- // Clipboard
473
+ await rdp.scroll.down({ amount: 5 }); // default 3; { x, y } to target a point
686
474
  await rdp.clipboard.set({ text: 'text to copy' });
687
475
  const text = await rdp.clipboard.get();
688
476
 
689
- // Locate text using OCR
690
- const matches = await rdp.locate({ text: 'Cancel' });
691
- if (matches.length > 0) {
692
- await rdp.mouse.click({ x: matches[0].center_x, y: matches[0].center_y });
693
- }
694
-
695
- // Get all text on screen
696
- const allText = await rdp.locate({ all: true });
697
-
698
- // Click a match directly - the coordinate never leaves the process
477
+ // OCR - click directly so the coordinate never leaves the process
699
478
  await rdp.locate({ text: 'Cancel', click: 'left' });
700
-
701
- // Exact whole-line match - won't also hit "Провести и закрыть"
702
479
  await rdp.locate({ text: 'Провести', exact: true, click: 'left' });
480
+ await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' }); // block until it appears
481
+ const allText = await rdp.locate({ all: true });
703
482
 
704
- // Block until text appears, instead of polling in a loop
705
- await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' });
706
-
707
- // Safe click of an externally-computed point (e.g. a vision-model bbox):
708
- // refuses if it's ambiguously close to more than one detected label
483
+ // Safe click of an externally-computed point (e.g. a vision-model bbox)
709
484
  const result = await rdp.clickAt(665, 209);
710
485
  if (!result.clicked) console.log('Ambiguous:', result.nearby);
711
486
 
712
- // Automation (requires --enable-win-automation at connect)
487
+ // File transfer (needs enableWinAutomation)
488
+ await rdp.files.push('./report.xlsx', 'C:\\Users\\Admin\\report.xlsx');
489
+ await rdp.files.pull('C:\\Users\\Admin\\export.csv', './export.csv');
490
+
491
+ // UI Automation
713
492
  const snapshot = await rdp.automation.snapshot({ interactive: true });
714
- const focused = await rdp.automation.focused(); // What has keyboard focus right now
715
- await rdp.automation.click('@e5'); // Click button by ref
716
- await rdp.automation.click('@e5', { doubleClick: true }); // Double-click
717
- await rdp.automation.select('@e10'); // Select item
718
- await rdp.automation.toggle('@e7'); // Toggle checkbox
719
- await rdp.automation.expand('@e3'); // Expand menu
720
- await rdp.automation.contextMenu('@e5'); // Open context menu
721
- await rdp.automation.fill('#input', 'text'); // Fill text field
722
- await rdp.automation.run('notepad.exe'); // Run command
493
+ const focused = await rdp.automation.focused();
494
+ await rdp.automation.click('@e5', { doubleClick: true });
495
+ await rdp.automation.fill('#input', 'text');
496
+ await rdp.automation.run('notepad.exe');
723
497
  await rdp.automation.waitFor('#SaveButton', { timeout: 5000 });
724
-
725
- // Window management
726
- const windows = await rdp.automation.listWindows();
727
498
  await rdp.automation.focusWindow('~*Notepad*');
728
- await rdp.automation.maximizeWindow();
729
499
 
730
- // Drives
731
500
  const drives = await rdp.drives.list();
732
-
733
- // Session info
734
501
  const info = await rdp.getInfo();
735
-
736
- // Disconnect
737
502
  await rdp.disconnect();
738
503
  ```
739
504
 
740
- ### WebSocket Streaming
505
+ The process can stay alive after `disconnect()` unless the caller cleans up —
506
+ call `rdp.close()` or exit explicitly.
741
507
 
742
- Enable WebSocket streaming for real-time screen capture and bidirectional clipboard support:
743
-
744
- ```typescript
745
- const rdp = new RdpSession({
746
- session: 'viewer',
747
- streamPort: 9224, // Enable streaming
748
- });
749
-
750
- await rdp.connect({...});
751
-
752
- // Connect your WebSocket client to receive JPEG frames
753
- const streamUrl = rdp.getStreamUrl(); // "ws://localhost:9224"
754
- ```
755
-
756
- For the complete WebSocket protocol specification (message types, clipboard flow, input handling), see [WEBSOCKET.md](https://github.com/denisix/agent-rdp/blob/main/docs/WEBSOCKET.md).
508
+ **WebSocket streaming** for real-time capture: construct with
509
+ `{ streamPort: 9224 }`, then `rdp.getStreamUrl()` returns `ws://localhost:9224`.
510
+ Message types, clipboard flow and input handling are specified in
511
+ [WEBSOCKET.md](docs/WEBSOCKET.md).
757
512
 
758
513
  ## Architecture
759
514
 
760
- agent-rdp uses a daemon-per-session architecture:
761
-
762
- 1. **CLI** (`agent-rdp`) - Parses commands and communicates with the daemon
763
- 2. **Daemon** - Maintains the RDP connection and processes commands
764
- 3. **IPC** - Unix sockets (macOS/Linux) or TCP (Windows)
765
-
766
- The daemon is automatically started on the first command and persists until explicitly closed or the session times out.
515
+ A daemon per session: the **CLI** parses commands and talks to a **daemon** over
516
+ Unix sockets (macOS/Linux) or TCP (Windows); the daemon owns the RDP connection
517
+ and processes commands. It starts on the first command and persists until
518
+ closed. See [CLAUDE.md](CLAUDE.md) for the internals.
767
519
 
768
520
  ## Limitations
769
521
 
770
- ### UI Automation
771
-
772
- - **WebViews**: UI Automation cannot interact with WebView content (e.g., Windows Start menu search, Edge browser content, Electron apps). Use `Win+R` or `automate run` to launch programs directly instead of clicking through menus.
773
- - **UAC Dialogs**: User Account Control elevation prompts run on a secure desktop and are not accessible via UI Automation. There is no good workaround - the remote user must interact with UAC manually, or UAC must be disabled (not recommended for security reasons).
774
-
775
- ### OCR Fallback
776
-
777
- When UI Automation cannot access certain elements, the `locate` command provides OCR-based text detection:
778
-
779
- ```bash
780
- agent-rdp locate "Button Text" # Find text and get coordinates
781
- agent-rdp mouse click <x> <y> # Click at returned coordinates
782
- ```
783
-
784
- This is not highly reliable (OCR can misread characters, miss text, or return imprecise coordinates), but may work for simple cases like dialog buttons.
785
-
786
- ### Screenshot Coordinate Detection
787
-
788
- **Claude models** (in non-computer-use mode, such as Claude Code) are poor at estimating pixel coordinates from screenshots. Do not ask Claude to look at a screenshot and guess where to click - it will likely be inaccurate.
789
-
790
- **Gemini models** are generally good at pixel coordinate estimation from images.
791
-
792
- If you need vision-based coordinate detection with Claude, implement your own harness using Claude's [Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use) which is specifically designed for this purpose.
522
+ - **WebViews** — UI Automation cannot see WebView content (Start menu search,
523
+ Edge, Electron apps). Launch programs with `automate run` or Win+R instead of
524
+ clicking through menus.
525
+ - **UAC dialogs** run on a secure desktop and are invisible to UI Automation.
526
+ There is no good workaround short of the remote user acting manually.
527
+ - **OCR** can misread characters, miss text, or return imprecise coordinates.
528
+ Use it only when UI Automation can't reach an element, and verify before
529
+ clicking anything destructive.
530
+ - **Claude models in non-computer-use mode** (Claude Code included) are poor at
531
+ estimating pixel coordinates from screenshots — don't ask for a guess from an
532
+ image. Gemini models are generally good at it; for Claude, use the
533
+ [Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use).
793
534
 
794
535
  ## Requirements
795
536
 
796
- - Rust 1.75 or later
797
- - Target RDP server must offer **TLS** for RDP. agent-rdp uses `rustls`, which does not implement TLS 1.0/1.1, so the server must support TLS 1.2 or later — legacy targets (e.g. Windows Server 2008 R2) are not supported and fail with a TLS handshake error.
798
- - NLA (`UserAuthentication=1`) is **recommended but not required** — TLS is what matters. The stock Windows defaults work as shipped.
537
+ Rust 1.75+. The target must offer **TLS** for RDP — agent-rdp uses `rustls`,
538
+ which doesn't implement TLS 1.0/1.1, so legacy hosts (e.g. Windows Server
539
+ 2008 R2) fail with a handshake error. **Stock Windows defaults work as
540
+ shipped.**
799
541
 
800
- ### RDP security layer settings
542
+ <details>
543
+ <summary>RDP security layer settings</summary>
801
544
 
802
- All six combinations measured against Windows Server 2022, varying only these two
803
- values under `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp`:
545
+ All six combinations measured against Windows Server 2022, varying only these
546
+ two values under
547
+ `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp`:
804
548
 
805
549
  | `SecurityLayer` | `UserAuthentication` | Meaning | Result |
806
550
  |---|---|---|---|
@@ -812,12 +556,12 @@ values under `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations
812
556
  | `0` (RDP) | `0` | legacy RC4 only | fails |
813
557
 
814
558
  The rule: **agent-rdp works wherever the host offers TLS.** With NLA on that is
815
- always the case, because NLA forces CredSSP/TLS regardless of `SecurityLayer`.
559
+ always the case, since NLA forces CredSSP/TLS regardless of `SecurityLayer`.
816
560
  Both failing rows are NLA-off hosts that refuse TLS outright.
817
561
 
818
562
  If `connect` reports *"server only supports Standard RDP Security"*, the host is
819
- in one of those two rows. Either enable NLA (preferred — it also gives you
820
- pre-authentication) or force TLS:
563
+ in one of those rows. Enable NLA (preferred — it also gives pre-authentication)
564
+ or force TLS:
821
565
 
822
566
  ```powershell
823
567
  $k = 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp'
@@ -826,18 +570,24 @@ Set-ItemProperty -Path $k -Name SecurityLayer -Value 2 # force TLS
826
570
  ```
827
571
 
828
572
  New connections normally pick this up immediately; restart `TermService` only if
829
- they don't (a restart drops existing sessions).
573
+ they don't (that drops existing sessions).
830
574
 
831
575
  Neither failing case is fixable client-side. With `SecurityLayer=1` and NLA off
832
576
  the server returns `SSL_NOT_ALLOWED_BY_SERVER` even though the client advertises
833
- `PROTOCOL_SSL` — verified by testing a build that requested SSL *only*, which was
834
- rejected identically. And Standard RDP Security (`SecurityLayer=0`, NLA off) is
835
- refused by IronRDP itself, which implements no RC4 transport layer; it uses a
836
- well-known key derivation, so credentials sent over it are recoverable in transit.
577
+ `PROTOCOL_SSL` — verified by testing a build that requested SSL *only*, rejected
578
+ identically. Standard RDP Security (`SecurityLayer=0`, NLA off) is refused by
579
+ IronRDP itself, which implements no RC4 transport; it uses a well-known key
580
+ derivation, so credentials sent over it are recoverable in transit.
581
+ </details>
837
582
 
838
583
  ## Credits
839
584
 
840
- Originally created by [Nick Yu](https://github.com/thisnick) ([thisnick/agent-rdp](https://github.com/thisnick/agent-rdp)). This fork ([denisix/agent-rdp](https://github.com/denisix/agent-rdp), published to npm as [`@denisixnpm/agent-rdp`](https://www.npmjs.com/package/@denisixnpm/agent-rdp)) is maintained independently with additional fixes and features; see [CHANGELOG.md](packages/agent-rdp/CHANGELOG.md) for what's changed.
585
+ Originally created by [Nick Yu](https://github.com/thisnick)
586
+ ([thisnick/agent-rdp](https://github.com/thisnick/agent-rdp)). This fork
587
+ ([denisix/agent-rdp](https://github.com/denisix/agent-rdp), published as
588
+ [`@denisixnpm/agent-rdp`](https://www.npmjs.com/package/@denisixnpm/agent-rdp))
589
+ is maintained independently; see
590
+ [CHANGELOG.md](packages/agent-rdp/CHANGELOG.md) for what's changed.
841
591
 
842
592
  ## License
843
593