ai-remote 0.4.5 → 0.4.6

package/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: ai-remote
+description: Drive a remote Windows, Linux or macOS desktop from the command line with `npx ai-remote` (also installed as `remotectl`) — open one session, then screenshot it, click, type, press keys, run shell commands over SSH, and watch it in a window. Use this skill whenever the user wants to control, inspect, automate or screenshot a machine over RDP/VNC/SSH — including when they never name the tool and just say "click the button on that Windows box", "what's on the screen of 192.168.x.x", "log into the VM and run this", or "take a screenshot of the remote desktop".
+---
+
+# Driving a remote desktop with ai-remote
+
+## The one thing to get right
+
+`open` is the only command that names a machine. It starts a session that
+outlives the command, and **every command after it takes no host, no account
+and no password** — they attach to the session that is already running.
+
+```bash
+export AI_REMOTE_PASSWORD=...          # never a flag; argv is world-readable
+npx ai-remote open 192.168.1.20 -u owner
+
+npx ai-remote shot -o screen.png       # not `shot 192.168.1.20`
+npx ai-remote click 640,400
+npx ai-remote close
+```
+
+Writing `ai-remote shot 192.168.1.20` is the single most common mistake, because
+that *was* the grammar in older versions and it is all over the internet. The
+CLI now refuses it with a message explaining the change rather than doing
+something surprising, so if you see that error, drop the host and try again.
+
+State carries between commands, and that is the entire point: a tool that
+reconnects between acting and looking can never see the result of its own
+action. Reuse also makes each command effectively instant — a keystroke is
+~0.15s against ~5s for a fresh RDP handshake.
+
+## Working loop
+
+Act, then look. The session holds the framebuffer, so a screenshot taken after a
+click shows what the click did.
+
+```bash
+npx ai-remote key MetaLeft             # open the Start menu
+npx ai-remote shot -o after.png        # see what happened
+```
+
+`shot` waits for the screen to stop changing before it captures, so there is
+normally no need to sleep first. When an action starts something slow (an app
+launching, a page loading), put an explicit wait in a `do` script instead of
+guessing from outside.
+
+Prefer `do` when the next few steps are already known — one round trip instead
+of four, and the timing between steps stays inside the session rather than
+depending on how fast the shell gets round to the next command:
+
+```bash
+npx ai-remote do "key MetaLeft; wait 800; type notepad; wait 1200; key Enter; wait 1500; shot s.png"
+```
+
+Steps, separated by `;`:
+
+| Step | Meaning |
+|---|---|
+| `move X,Y` | move the pointer |
+| `click X,Y[,right\|middle]` | click, optionally another button |
+| `dblclick X,Y` | double click |
+| `scroll X,Y,DX,DY` | scroll by notches at a position |
+| `type TEXT` | type into whatever has focus |
+| `key Code[+Code...]` | press keys; `+` is a chord |
+| `cad` | Ctrl+Alt+Del |
+| `wait MS` | pause |
+| `shot FILE` | write a screenshot mid-script |
+
+## Screenshots and coordinates
+
+**Coordinates are always in the desktop's own pixels, never the screenshot's.**
+This matters the moment you shrink a screenshot to save tokens:
+
+```bash
+npx ai-remote shot --max-edge 800 --json -o s.png
+# {"ok":true,...,"width":800,"height":500,"file":"/abs/path/s.png"}
+```
+
+That image is 800x500 but the desktop is still 1280x800. A button you see at
+(400, 250) in the image is at (640, 400) on the desktop. Scale back before
+clicking:
+
+```
+desktop_x = image_x * desktop_width / image_width
+```
+
+`ai-remote status --json` reports the true desktop size (`width`, `height`), and
+so does `shot` when you do not pass `--max-edge`. When in doubt, take the
+screenshot full size and click the coordinates you read off it.
+
+`--max-edge` is worth using anyway: a 1280x800 PNG is several hundred KB of
+image tokens and a 800px one is a fraction of that, usually with no loss of
+anything you needed to see.
+
+## Keys
+
+Key codes are DOM-style physical codes: `MetaLeft`, `Enter`, `Escape`, `Tab`,
+`F5`, `KeyA`, `Digit1`, `ArrowDown`, `ControlLeft`, `AltLeft`, `ShiftLeft`.
+
+```bash
+npx ai-remote key ControlLeft+KeyA     # a chord: Ctrl+A
+npx ai-remote key MetaLeft KeyR        # in sequence: Meta, then R
+```
+
+`+` holds the modifiers down for the last key. Separate arguments are pressed
+one after another. An unknown code is ignored rather than guessed at, so if
+nothing happens, check the spelling against the list above.
+
+## Running commands instead of clicking
+
+When the machine has SSH, a command is faster and far more reliable than driving
+the GUI — no coordinates, no waiting, and real output to read:
+
+```bash
+npx ai-remote exec "hostname"
+npx ai-remote exec "dir C:\\Users"
+```
+
+`exec` reuses the session's own account and password, so it needs no flags. Its
+stdout is the command's output, and it exits 0 when the command did and 1 when
+it did not — the command's own status is in `exitStatus` under `--json` if you
+need the number. `ai-remote shell` opens an interactive terminal (Ctrl-] to
+leave) — that one is for a human, not for an agent, because it never returns.
+
+Reach for the GUI when the task is genuinely graphical. Reach for `exec` for
+anything a shell can do.
+
+## Sessions
+
+The session opened last is the one commands drive. With several open, name them:
+
+```bash
+npx ai-remote open 192.168.1.20 -u owner --session desk
+npx ai-remote open 192.168.1.30 -u owner --session build
+
+npx ai-remote list                  # * marks the one commands will use
+npx ai-remote shot --session desk   # a name, a host, or host:port
+npx ai-remote close --all           # or `close` for the current one
+```
+
+Nothing times out and nothing closes itself — a session that an agent left
+mid-task is a feature, not a leak — so **close what you open** when the task is
+done, or tell the user it is still running.
+
+## Watching
+
+By default `open` also shows the desktop in a real window, and whoever opens it
+can drive it straight away. Control is not exclusive — you and the person at the
+window can both work the same desktop at once, because your input goes over the
+session's own socket and never through the viewer. Expect a human to move the
+pointer out from under you: if a step depends on what was on screen, take a
+fresh `shot` rather than trusting one from before you paused.
+
+- Leave the window on when a human is present. Watching it happen is the reason
+  it exists.
+- `--watch-only` starts the window as an observer instead, for a session someone
+  should supervise rather than share. Taking control is then one click in the
+  toolbar, and the gate is enforced in the session rather than in the page.
+- `--headless` (or `--no-view`) on a server or in CI. The framebuffer is the
+  authority either way, so a session nobody watches is not a lesser session.
+- `ai-remote view` adds a window to a headless session later (`view --watch-only`
+  for an observer one); `view --close` takes it away without ending the session.
+
+## Reading results
+
+Add `--json` to any command for one object on stdout; diagnostics go to stderr,
+so the two never interleave.
+
+```bash
+npx ai-remote status --json
+# {"ok":true,"host":"...","name":"desk","width":1280,"height":800,
+#  "connected":true,"viewer":"http://127.0.0.1:7373/?t=...","viewers":1,
+#  "shell":"closed","pid":54594}
+```
+
+Exit codes carry meaning, which is usually faster than parsing the message:
+
+| Code | Meaning |
+|---|---|
+| 0 | fine |
+| 10 | unreachable |
+| 11 | credentials rejected |
+| 12 | bad usage, or no session open |
+| 13 | timed out |
+
+## When it does not work
+
+**"No session is open."** — Nothing is running. `open` first. Commands never
+start a session implicitly, because they have no host to start one with.
+
+**Credentials rejected (11), or a session that stops before it is ready.** —
+Almost always `AI_REMOTE_PASSWORD` missing from *this* shell. It is deliberately
+not a flag, and it does not carry across terminals; export it again. `open` says
+so explicitly when it sees the host ask for NLA and finds no password.
+
+**Unreachable (10).** — `ai-remote probe <host>` is the one command besides
+`open` that still takes an address: it answers whether anything is listening
+before you spend a handshake finding out.
+
+**A session that stopped by itself.** — The host closed the connection, or the
+machine went away. `ai-remote list` shows what is really running; stale entries
+clean themselves up when it does.
+
+## Getting the desktop's own size right
+
+RDP lets the client choose the desktop size, and the host may or may not honour
+it. Ask for what suits the task at `open` time (`-W 1920 -H 1080`), then trust
+what `status` reports rather than what you asked for.
+
+## Cheat sheet
+
+```bash
+export AI_REMOTE_PASSWORD=...
+npx ai-remote open HOST[:PORT] -u USER [-d DOMAIN] [--session NAME] [--headless] [--watch-only]
+npx ai-remote shot [-o FILE] [--max-edge N] [--json]
+npx ai-remote click X,Y [--button right|middle] [--double]
+npx ai-remote type "text"
+npx ai-remote key ControlLeft+KeyA
+npx ai-remote do "key MetaLeft; wait 800; type notepad; shot s.png"
+npx ai-remote exec "command"
+npx ai-remote view [--watch-only] [--close]
+npx ai-remote status | list | close [--all]
+npx ai-remote probe [HOST[:PORT]]
+```
+
+Every command takes `--session NAME` to pick between open sessions, and `--json`
+for machine-readable output. `ai-remote --help` is the authority; this file is
+the map.
+
+## Working inside this repository
+
+This repo *is* ai-remote. When testing a change, drive the local build rather
+than the published package, or you will be exercising the last release instead
+of your edit:
+
+```bash
+node packages/core/scripts/build.mjs        # after editing packages/core/src/cli/*
+node packages/core/dist/cli.mjs open HOST -u USER --headless
+```
+
+The CLI's own source is [`packages/core/src/cli/`](src/cli/):
+`cli.ts` is the grammar, `sessions.ts` decides which session a command drives,
+`daemon.ts` is the process that holds the connection, and `viewer.ts` plus
+`window.ts` are the window. `pnpm test` covers them.