@desklink/axi 0.3.0 → 0.4.0

package/README.md

@@ -4,7 +4,7 @@
   <a href="https://www.npmjs.com/package/@desklink/axi"><img alt="npm" src="https://img.shields.io/npm/v/@desklink/axi?style=flat" /></a>
   <a href="https://github.com/umeranjum17/desklink/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/umeranjum17/desklink/ci.yml?style=flat&branch=main" /></a>
   <a href="https://github.com/umeranjum17/desklink/blob/main/LICENSE"><img alt="Apache 2.0" src="https://img.shields.io/badge/license-Apache--2.0-666?style=flat" /></a>
-  <img alt="Linux and macOS" src="https://img.shields.io/badge/Linux%20%7C%20macOS-111?style=flat" />
+  <img alt="Linux and macOS (preview)" src="https://img.shields.io/badge/Linux%20%7C%20macOS%20%28preview%29-111?style=flat" />
 </p>
 
 <p align="center">
@@ -21,16 +21,17 @@ npm install -g @desklink/axi
 ```
 
 Or build from the repository root with `npm install && npm run build`.
+Delete copied `*.tsbuildinfo` before rebuilding a moved worktree.
 The engine is resolved by `@desklink/host`; see its [installation guide](../desktop-host/README.md#install-and-run)
-for supported platforms and [macOS setup](../desktop-host/README.md#macos).
-Set `DESKLINK_AXI_ENGINE` for a source build.
+for supported platforms and [macOS setup](../desktop-host/README.md#macos-preview).
+Set `DESKLINK_AXI_ENGINE` for a matching source build. An override pointing new AXI at a 0.3.x engine is unsupported; host and engine versions must match.
 
-A Linux/macOS desktop AXI. It runs a private bridge per session, captures read-only frames through `@desklink/host` protocol v3, and sends input only through a WebRTC `control` channel. The bridge's Unix socket lives in `$XDG_RUNTIME_DIR/desklink-axi` (or the OS temp directory), with `DESKLINK_AXI_SESSION` selecting an isolated socket. It opens a separate engine session; don't drive the desktop while someone else is controlling it.
+A Linux desktop AXI with macOS in preview. It runs a private bridge per session, captures read-only frames through `@desklink/host` protocol v3, and sends input only through a WebRTC `control` channel. The bridge's Unix socket lives in `$XDG_RUNTIME_DIR/desklink-axi` (or the OS temp directory), with `DESKLINK_AXI_SESSION` selecting an isolated socket. It opens a separate engine session; don't drive the desktop while someone else is controlling it.
 
-The engine is resolved by `@desklink/host`; set `DESKLINK_AXI_ENGINE` to an executable built from source if no packaged binary is installed. On macOS use `PATH=/opt/homebrew/bin:$PATH`, Rust >= 1.97.1, delete copied `*.tsbuildinfo` before rebuilding a moved worktree, and `DESKLINK_VPX_STATIC_DIR=/opt/homebrew` (the install **prefix**, not `/opt/homebrew/lib`). For macOS start the bridge from a stable signed app, not `desklink-axi start` in Terminal/SSH: `open -n /absolute/DesklinkHost.app --args axi-bridge /absolute/path/to/packages/axi/bin/desklink-axi.js --control`. Install or update that app using `packages/desktop-host/release/install-mac-app.sh`; grant Screen Recording and Accessibility to it once. An SSH-launched Node helper does not inherit those grants. OCR needs the optional `tesseract` executable (`brew install tesseract`, `sudo pacman -S tesseract` or `sudo apt install tesseract-ocr`); on macOS AXI also finds Homebrew's standard install paths when LaunchServices omits them from PATH. `look` works without it.
+For macOS source prerequisites, signing and permission attribution, follow the [host setup guide](../desktop-host/README.md#macos-preview). To launch AXI through its signed local harness, use `open -n /absolute/DesklinkHost.app --args axi-bridge /absolute/path/to/packages/axi/bin/desklink-axi.js` from the logged-in Mac session. OCR needs the optional `tesseract` executable (`brew install tesseract`, `sudo pacman -S tesseract` or `sudo apt install tesseract-ocr`); on macOS AXI also finds Homebrew's standard install paths when LaunchServices omits them from PATH. `look` works without it.
 
 ```
-desklink-axi start --control --source x11 --display :97
+desklink-axi start --source x11 --display :97
 desklink-axi screen --query "Budget"
 desklink-axi tree --query "Save"
 desklink-axi type "hello" --into @a3.10
@@ -44,7 +45,11 @@ desklink-axi look @r1
 desklink-axi stop
 ```
 
-`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` defaults to view-only; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/portal-token` (otherwise `~/.local/state/desklink-axi/portal-token`), atomically replaced with mode 0600 after each grant. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard read/write require `start --control`; view-only clipboard calls return a structured error naming that flag. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
+`start --cursor hidden` opts into cursor-free capture; omit it for existing behavior. Use `desklink-axi cursor` to drain source-coordinate cursor samples (`x`, `y`, `visible`, `timestamp_us`) from the live session; `dropped:N` reports older samples overwritten after its 10,000-sample buffer fills. Linux portals require cursor metadata and refuse hidden capture when it is unavailable; see the [cursor protocol](../desktop-host/docs/PROTOCOL.md#cursor-free-capture).
+
+`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` grants all available access to its same-user paired consumer; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/portal-token` (otherwise `~/.local/state/desklink-axi/portal-token`), atomically replaced with mode 0600 after each grant. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard need no app grants; unavailable OS backends fail per action. Legacy `start --control` is accepted and ignored. See the [clipboard protocol](../desktop-host/docs/PROTOCOL.md#clipboard-over-the-local-protocol) for X11 targets, transfer limits and selection lifetime. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
+To guide the person without input, use `desklink-axi point 400,300 --label "Here" --timeout 3000`, or `desklink-axi point --clear`. See the [host protocol reference](../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources, coordinates, and limits.
+
 ## Browser lane (opt-in, loopback-only)
 
 For web pages the browser lane gives semantic control through [playwright-core](https://www.npmjs.com/package/playwright-core) over CDP; the desktop lane stays for native apps and whole-desktop tasks. Nothing is attached implicitly:
@@ -60,16 +65,23 @@ desklink-axi browser select 1 / open <url> / navigate <url> / close 1 / detach
 
 Non-loopback endpoints are refused; `launch` renders on `$DISPLAY` (X11) so it never follows `WAYLAND_DISPLAY` to another desktop, and `--arg <chromium flag>` (repeatable) passes extra flags such as `--arg=--window-size=1280,640`. Snapshot refs are re-derived on every command and rejected as `stale-ref` when the page changed. `upload` only sets files on `<input type=file>`; for OS-level dialogs it returns `not-supported` and names the desktop lane (`desklink-axi click/type`), which is also the route for real native-dialog navigation — a DOM upload is not equivalent to completing a native file dialog. `detach` stops the browser only if this lane launched it; a passed `--profile` directory is kept, a lane-created temp profile is removed.
 
+On Linux, a lane-launched browser's recorded PID and start time must match
+before connection or cleanup. Cached sessions from older versions without a
+recorded start time are refused; use a fresh `DESKLINK_AXI_SESSION` and profile.
+Cleanup verifies helper PID/start-time pairs against the private launch marker
+before signalling them. Browsers attached through `--cdp` remain caller-owned.
+
 ## Batch outcomes
 
 `batch` steps are `[verb, ...args]` arrays or `{verb, args?, name?, wait?}` objects with verbs `click, type, press, scroll, wait, snapshot, assert, tree, marks`. A named snapshot step binds its OCR items so later steps can target `@name.n`; `assert` steps make success depend on observed screen text; per-step lines separate `input: applied` (transport acknowledgement) from `effect: change observed | none within timeout`, and the footer says `effects: asserted[n]` or `effects: unverified (input acknowledgements only)` — an input acknowledgement is never a verified outcome, so wrong-focus typing or a no-op click fails the batch instead of reporting success.
 
 ## Proofs and measurements
 
-From the repository root with a task-built engine (`DESKLINK_AXI_ENGINE`) on Linux:
+From the repository root with a task-built engine (`DESKLINK_AXI_ENGINE`) on
+Linux, following the [private-lab guidance](../../README.md#develop):
 
 - `node packages/axi/test/smoke.mjs` — clean-tree rebuild of host+axi, then `batch` and `click` on a private Xvfb; asserts no task-owned survivors.
-- `node packages/axi/test/flow.mjs` — full desktop proof (frame/damage, input, chords, batch outcomes, injected-failure cleanup).
+- `node packages/axi/test/flow.mjs` — full desktop proof (frame/damage, point coordinates/timeout/clear without pointer or focus changes, click-through, input, chords, batch outcomes, injected-failure cleanup).
 - `node packages/axi/test/browser-flow.mjs` — the open/form/two-tab fixture through the browser lane with semantic refs on a private Xvfb; also proves selected-vs-front, stale-ref rejection, DOM upload and the not-supported fallback.
 - `node packages/axi/test/bench.mjs [runs]` — comparative p50/p95 (default 20 measured runs after 3 warmups) for the browser lane vs the AXI fallback over the open/form/tabs/native fixtures; text-token proxy is `ceil(stdout chars/4)`, a proxy, not a tokenizer.
 
@@ -99,7 +111,7 @@ Measured on a private 1280×720 Xvfb client: `diff → click 100,100 → look @r
 
 ## Accessibility tree (`tree`) — Linux AT-SPI
 
-`tree` reads the desktop's AT-SPI accessibility tree over D-Bus and returns bounded TOON rows of `ref,role,name,value,x,y,w,h,states`. It needs no `--control`; acting on refs does. Refs are generation-tagged (`@a<gen>.<n>`), stay usable across newer snapshots, and are revalidated at use time: a ref whose role, name, or screen extents changed — or whose object is gone — returns `stale-ref` instead of acting blind.
+`tree` reads the desktop's AT-SPI accessibility tree over D-Bus and returns bounded TOON rows of `ref,role,name,value,x,y,w,h,states`. Read and input actions share the same paired session. Refs are generation-tagged (`@a<gen>.<n>`), stay usable across newer snapshots, and are revalidated at use time: a ref whose role, name, or screen extents changed — or whose object is gone — returns `stale-ref` instead of acting blind.
 
 - Capability probe: the bus is resolved from `AT_SPI_BUS_ADDRESS`, else `org.a11y.Bus.GetAddress` on the session bus. Unavailable → structured `a11y-unavailable` error; every other command keeps working. macOS AX is **not implemented** and labeled as such.
 - Values: text and value widgets expose their content in the `value` column (Text/Value interfaces), so a typed entry can be verified without pixels.

package/dist/bridge.js

@@ -8,6 +8,7 @@ import { PeerConnection } from 'node-datachannel';
 import { saveToken, takeToken, tokenPath } from './token.js';
 import { A11yClient, A11yError, axRefPattern, isActionable } from './atspi.js';
 import { dedupeCandidates, drawMarks, estimateMarksCost } from './marks.js';
+import { CursorEventBuffer, formatCursorSamples } from './cursor-events.js';
 export const socketPath = join(process.env.XDG_RUNTIME_DIR ?? join(tmpdir(), `desklink-axi-${process.getuid?.() ?? 'user'}`), 'desklink-axi', `${process.env.DESKLINK_AXI_SESSION ?? 'default'}.sock`);
 export async function call(command, args = []) {
     if (command === 'stop' && !existsSync(socketPath))
@@ -106,7 +107,9 @@ export async function serve(args) {
         throw new Error('session already running');
     const source = args.includes('--source') ? args[args.indexOf('--source') + 1] : 'auto';
     const display = args.includes('--display') ? args[args.indexOf('--display') + 1] : undefined;
-    const control = args.includes('--control');
+    const cursor = args.includes('--cursor') ? args[args.indexOf('--cursor') + 1] : 'embedded';
+    if (cursor !== 'embedded' && cursor !== 'hidden')
+        throw new Error('cursor: use embedded or hidden');
     const timeout = args.includes('--timeout') ? Number(args[args.indexOf('--timeout') + 1]) : 120000;
     if (process.platform !== 'linux' && process.platform !== 'darwin')
         throw new Error('Linux or macOS only');
@@ -119,6 +122,7 @@ export async function serve(args) {
             : 'desktop engine unavailable; set DESKLINK_AXI_ENGINE');
     mkdirSync(join(socketPath, '..'), { recursive: true, mode: 0o700 });
     const events = [];
+    const cursorSamples = new CursorEventBuffer();
     const tokenFailure = {};
     const checkToken = () => { if (tokenFailure.current)
         throw new Error(`cannot persist portal restore token: ${tokenFailure.current.message}`); };
@@ -128,7 +132,9 @@ export async function serve(args) {
     let peer;
     let offerReady = false;
     const engine = await EngineClient.start(executable.command, executable.args, { requestTimeoutMs: 125000, onEvent: event => {
-            if (event.event !== 'session.frame.changed')
+            if (event.event === 'session.cursor')
+                cursorSamples.push(event);
+            else if (event.event !== 'session.frame.changed')
                 events.push(event);
             if (event.event === 'session.restoreToken' && tokenFile) {
                 try {
@@ -157,12 +163,13 @@ export async function serve(args) {
             source: process.platform === 'darwin'
                 ? { kind: 'display', ...(display === undefined ? {} : { display_id: Number(display) }) }
                 : source === 'x11' || (source === 'auto' && display) ? { kind: 'x11', display } : { kind: 'portal' },
-            permissions: control ? ['view', 'control', 'clipboard'] : ['view'], loopbackTcp: true,
-            agentIndicator: control,
+            loopbackTcp: true,
+            agentIndicator: true, // serve rejects platforms other than Linux/macOS above.
+            cursor,
             ...(tokenFile ? { restoreToken: takeToken(tokenFile) } : {}),
         }, timeout);
         checkToken();
-        peer = control ? new PeerConnection('desklink-axi', { iceServers: [], bindAddress: '127.0.0.1', enableIceTcp: true }) : undefined;
+        peer = new PeerConnection('desklink-axi', { iceServers: [], bindAddress: '127.0.0.1', enableIceTcp: true });
         let channel;
         let open = false;
         let nextSeq = 1;
@@ -267,8 +274,6 @@ export async function serve(args) {
             return [...lines.values()];
         };
         const act = async (message) => {
-            if (!control)
-                throw new Error('input-unavailable: start --control');
             for (let i = 0; i < 100 && !open; i++)
                 await new Promise(r => setTimeout(r, 50));
             if (!open || !channel)
@@ -313,7 +318,11 @@ export async function serve(args) {
             let waitSeen = baseline;
             routeNote = undefined;
             if (command === 'start')
-                return `session: open source=${source} ${display ?? ''} size=${opened.geometry.encoded.width}x${opened.geometry.encoded.height} permissions=view${control ? ',control' : ''}`;
+                return `session: open source=${source} ${display ?? ''} size=${opened.geometry.encoded.width}x${opened.geometry.encoded.height}`;
+            if (command === 'cursor') {
+                const batch = cursorSamples.take(opened.sessionId);
+                return formatCursorSamples(batch.samples, batch.dropped);
+            }
             if (command === 'health') {
                 const state = events.filter(event => event.event === 'session.state' || event.event === 'session.capture.stopped').at(-1);
                 if (state?.event === 'session.capture.stopped')
@@ -424,8 +433,6 @@ export async function serve(args) {
                 return 'session: stopped';
             }
             if (command === 'clipboard') {
-                if (!control)
-                    throw new Error('input-unavailable: clipboard requires start --control');
                 if (args[0] === 'read') {
                     const result = await engine.readClipboard(opened.sessionId);
                     return `clipboard: ${args.includes('--full') ? result.text : result.text.slice(0, 1000)} (${result.text.length} chars)`;
@@ -435,6 +442,21 @@ export async function serve(args) {
                     return 'clipboard: written';
                 }
             }
+            if (command === 'point') {
+                if (args.includes('--clear')) {
+                    if (args.length !== 1)
+                        throw new Error('point: --clear accepts no other arguments');
+                    await engine.point(opened.sessionId, { clear: true });
+                    return 'point: cleared';
+                }
+                if (!/^\d+,\d+$/.test(args[0] ?? ''))
+                    throw new Error('point: expected x,y desktop pixels, or --clear');
+                const [x, y] = args[0].split(',').map(Number);
+                const timeout = args.includes('--timeout') ? Number(args[args.indexOf('--timeout') + 1]) : 3000;
+                const label = args.includes('--label') ? args[args.indexOf('--label') + 1] : '';
+                await engine.point(opened.sessionId, { x: x, y: y, label, timeoutMs: timeout });
+                return `point: shown at=${x},${y} timeout=${timeout}ms`;
+            }
             const action = ['click', 'drag', 'type', 'press', 'scroll'].includes(command);
             const actionWait = action && args.includes('--wait') ? args[args.indexOf('--wait') + 1] : 'none';
             const actionDelay = action && /^\d+$/.test(actionWait) ? boundedDelay(actionWait) : undefined;
@@ -629,6 +651,15 @@ export async function serve(args) {
             if (fields.some(field => !['ref', 'text', 'x', 'y', 'w', 'h', 'conf', 'line'].includes(field)))
                 throw new Error('fields: valid fields are ref,text,x,y,w,h,conf,line');
             const frame = await capture(pendingDamage ? waitSeen : baseline);
+            // A frame reply can already be older than the latest change notice. Its
+            // stillness must not erase motion from a newer frame. Classify at capture
+            // time too: synchronous OCR can outlast the motion freshness window.
+            const moving = [...motion].filter(([, state]) => state.count >= 3 && Date.now() - state.at < 600 && (frame.still_ms < 150 || state.seq > frame.seq));
+            if (frame.still_ms >= 150)
+                for (const [region, state] of motion) {
+                    if (state.seq <= frame.seq)
+                        motion.delete(region);
+                }
             if (pendingDamage)
                 frame.damage = [...new Set([...pendingDamage, ...frame.damage])];
             if (command === 'look') {
@@ -717,7 +748,7 @@ export async function serve(args) {
                 }
             }
             if (command === 'home')
-                return `session: open source=${source} ${display ?? ''} size=${frame.width}x${frame.height} permissions=view${control ? ',control' : ''}\nframe: ${frame.seq} settled=${frame.still_ms >= 150} unseen=${frame.changed ? frame.damage.length : 0} region\nwindows: unavailable on this compositor\ntext[${Math.min(text.length, 12)} of ${text.length}]{ref,text,x,y}:\n${text.slice(0, 12).map(t => `  ${t.ref},${JSON.stringify(t.text)},${t.x},${t.y}`).join('\n')}\nhelp[2]:\n  desklink-axi diff\n  desklink-axi screen --query "<words>"`;
+                return `session: open source=${source} ${display ?? ''} size=${frame.width}x${frame.height}\nframe: ${frame.seq} settled=${frame.still_ms >= 150} unseen=${frame.changed ? frame.damage.length : 0} region\nwindows: unavailable on this compositor\ntext[${Math.min(text.length, 12)} of ${text.length}]{ref,text,x,y}:\n${text.slice(0, 12).map(t => `  ${t.ref},${JSON.stringify(t.text)},${t.x},${t.y}`).join('\n')}\nhelp[2]:\n  desklink-axi diff\n  desklink-axi screen --query "<words>"`;
             if (command === 'screen') {
                 const query = args.includes('--query') ? args[args.indexOf('--query') + 1] : undefined;
                 const items = query ? text.filter(t => t.text.toLowerCase().includes(query.toLowerCase())) : text;
@@ -747,11 +778,8 @@ export async function serve(args) {
                     return false;
                 });
                 const gone = unmatched.length;
-                if (frame.still_ms >= 150)
-                    motion.clear();
                 const animating = [];
                 const visible = [];
-                const moving = [...motion].filter(([, state]) => state.count >= 3 && Date.now() - state.at < 600);
                 for (const region of [...frame.damage, ...moving.map(([key]) => key).filter(key => !frame.damage.some(region => overlaps(key, region)))]) {
                     const state = moving.find(([key]) => overlaps(key, region))?.[1];
                     const [x, y, w, h] = region.split(',').map(Number);
@@ -770,8 +798,6 @@ export async function serve(args) {
                 const changed = visible.length ? `${visible.length} region since frame ${frame.previous}` : `none since frame ${frame.previous}`;
                 return `changed: ${changed}\nregions[${visible.length}]{ref,box}:\n${visible.map((d, i) => `  @r${i + 1},"${d}"`).join('\n')}\nappeared[${Math.min(appeared.length, 20)} of ${appeared.length}]{${fields.join(',')}}:\n${appeared.slice(0, 20).map(t => `  ${fields.map(field => JSON.stringify(field === 'w' ? t.words.at(-1).x + t.words.at(-1).w - t.x : field === 'h' ? Math.max(...t.words.map(w => w.y + w.h)) - t.y : t[field])).join(',')}`).join('\n')}\ngone: ${gone} text items${animating.length ? `\nanimating[${animating.length}]{box}:\n${animating.map(box => `  "${box}"`).join('\n')}\nanimating: ${animating.join('; ')}` : ''}\nhelp[2]:\n  desklink-axi look @r1\n  desklink-axi screen --query "<words>"`;
             }
-            if (frame.still_ms >= 150)
-                motion.clear();
             return `changed: none since frame ${frame.seq} (still ${frame.still_ms}ms)\nhelp[1]:\n  desklink-axi screen --query "<words>"`;
         };
         let commands = Promise.resolve();

package/dist/browser-lane.js

@@ -70,6 +70,8 @@ async function orderedPages(state, context) {
 }
 async function connect() {
     const state = readState();
+    if (state.launched)
+        assertOwnedBrowser(state);
     const { chromium } = await import('playwright-core');
     const browser = await chromium.connectOverCDP(`http://${state.endpoint}`).catch((error) => {
         throw new AxiError(`browser: cannot reach ${state.endpoint}: ${error.message.split('\n')[0]}`, 'check that the browser is still running; re-attach with browser attach --cdp');
@@ -152,12 +154,23 @@ const attach = {
         return 0;
     },
 };
-function childrenOf(pid) {
+function processStarted(pid) {
     try {
-        return readFileSync(`/proc/${pid}/task/${pid}/children`, 'utf8').trim().split(/\s+/).filter(Boolean).map(Number);
+        const stat = readFileSync(`/proc/${pid}/stat`, 'utf8');
+        const fields = stat.slice(stat.lastIndexOf(') ') + 2).split(' ');
+        return fields[0] === 'Z' ? undefined : fields[19];
     }
     catch {
-        return [];
+        return undefined;
+    }
+}
+function assertOwnedBrowser(state) {
+    if (process.platform !== 'linux')
+        return; // /proc identity is Linux-only.
+    // The fingerprint was recorded from our ChildProcess at spawn. Chromium may
+    // hide/scrub /proc/environ after startup; it is not the root PID's identity.
+    if (!state.pid || !state.pidStarted || processStarted(state.pid) !== state.pidStarted) {
+        throw new AxiError('browser: refusing stale or unowned process identity', 'use a fresh task session; no process was signalled or contacted');
     }
 }
 // Chromium helpers (crashpad handlers double-fork, xdg-settings spawns during
@@ -180,22 +193,31 @@ function markerStrays(marker) {
         if (!Number.isInteger(pid) || pid <= 1 || pid === process.pid)
             continue;
         try {
-            if (uid !== undefined && statSync(`/proc/${pid}`).uid !== uid)
+            const started = processStarted(pid);
+            if (!started || (uid !== undefined && statSync(`/proc/${pid}`).uid !== uid))
                 continue;
-            if (readFileSync(`/proc/${pid}/environ`, 'utf8').split('\0').includes(needle))
-                found.push(pid);
+            if (readFileSync(`/proc/${pid}/environ`, 'utf8').split('\0').includes(needle) && processStarted(pid) === started)
+                found.push({ pid, started });
         }
         catch { /* vanished or unreadable */ }
     }
     return found;
 }
 async function stopLaunched(state) {
-    const stop = async (pid) => {
+    const signal = (pid, started, value) => {
+        if (process.platform === 'linux' && (!started || processStarted(pid) !== started))
+            return;
         try {
-            process.kill(pid, 'SIGTERM');
+            process.kill(pid, value);
         }
         catch { /* already gone */ }
-        for (let i = 0; i < 20; i++) {
+    };
+    const stop = async (pid, started = processStarted(pid)) => {
+        const same = () => process.platform !== 'linux' || (started !== undefined && processStarted(pid) === started);
+        if (!same())
+            return;
+        signal(pid, started, 'SIGTERM');
+        for (let i = 0; i < 20 && same(); i++) {
             try {
                 process.kill(pid, 0);
             }
@@ -204,17 +226,17 @@ async function stopLaunched(state) {
             }
             await new Promise(r => setTimeout(r, 50));
         }
-        try {
-            process.kill(pid, 'SIGKILL');
-        }
-        catch { /* already gone */ }
+        if (same())
+            signal(pid, started, 'SIGKILL');
     };
-    if (state.pid) {
-        // Record the child tree before any signal; kill only these exact PIDs.
-        const kids = [...new Set([...childrenOf(state.pid), ...childrenOf(state.pid).flatMap(childrenOf)])];
-        await stop(state.pid);
+    if (state.pid && (process.platform !== 'linux' || processStarted(state.pid))) {
+        assertOwnedBrowser(state);
+        // This launch supplied the private marker at spawn. Snapshot matching
+        // PID/start-time pairs before signalling, never descendants of a reused PID.
+        const kids = state.browserSession ? markerStrays(state.browserSession) : [];
+        await stop(state.pid, state.pidStarted);
         for (const kid of kids)
-            await stop(kid);
+            await stop(kid.pid, kid.started);
     }
     // Helpers that reparented out of the recorded tree (crashpad handlers,
     // xdg-settings spawned during shutdown) still carry the launch marker:
@@ -227,20 +249,12 @@ async function stopLaunched(state) {
             if (!strays.length)
                 break;
             if (Date.now() >= deadline) {
-                for (const pid of strays) {
-                    try {
-                        process.kill(pid, 'SIGKILL');
-                    }
-                    catch { /* gone */ }
-                }
+                for (const child of strays)
+                    signal(child.pid, child.started, 'SIGKILL');
                 break;
             }
-            for (const pid of strays) {
-                try {
-                    process.kill(pid, termed ? 'SIGKILL' : 'SIGTERM');
-                }
-                catch { /* gone */ }
-            }
+            for (const child of strays)
+                signal(child.pid, child.started, termed ? 'SIGKILL' : 'SIGTERM');
             termed = true;
             await new Promise(r => setTimeout(r, 100));
         }
@@ -295,10 +309,11 @@ const launch = {
             ...(parsed.flags.headless ? ['--headless=new'] : []),
             ...(Array.isArray(parsed.flags.arg) ? parsed.flags.arg.map(String) : typeof parsed.flags.arg === 'string' ? [parsed.flags.arg] : []),
         ];
-        const browserSession = randomBytes(8).toString('hex');
+        const browserSession = randomBytes(16).toString('hex');
         const log = openSync(join(profile, 'chromium.log'), 'a');
         const child = spawn(executable, args, { detached: true, stdio: ['ignore', 'ignore', log], env: { ...process.env, DESKLINK_AXI_BROWSER_SESSION: browserSession } });
         closeSync(log);
+        const pidStarted = child.pid ? processStarted(child.pid) : undefined;
         const portFile = join(profile, 'DevToolsActivePort');
         for (let i = 0; i < 300 && (!existsSync(portFile) || !readFileSync(portFile, 'utf8').trim()) && child.pid; i++)
             await new Promise(r => setTimeout(r, 100));
@@ -307,17 +322,12 @@ const launch = {
                 child.kill('SIGKILL');
             }
             catch { /* already gone */ }
-            for (const pid of markerStrays(browserSession)) {
-                try {
-                    process.kill(pid, 'SIGKILL');
-                }
-                catch { /* gone */ }
-            }
+            await stopLaunched({ endpoint: '', launched: true, pid: child.pid, pidStarted, browserSession });
             const tail = existsSync(join(profile, 'chromium.log')) ? readFileSync(join(profile, 'chromium.log'), 'utf8').split('\n').filter(Boolean).at(-1) ?? '' : '';
             throw new AxiError(`browser: Chromium did not open its debugging port${tail ? `: ${tail.slice(0, 200)}` : ''}`, 'check the executable; see chromium.log in the profile; try --headless on displays without a compositor');
         }
         const port = readFileSync(portFile, 'utf8').split('\n')[0].trim();
-        writeState({ endpoint: `127.0.0.1:${port}`, launched: true, pid: child.pid, profile, profileCreated: typeof parsed.flags.profile !== 'string', browserSession });
+        writeState({ endpoint: `127.0.0.1:${port}`, launched: true, pid: child.pid, pidStarted, profile, profileCreated: typeof parsed.flags.profile !== 'string', browserSession });
         child.unref();
         print(`browser: launched 127.0.0.1:${port} pid=${child.pid} profile=${profile}`);
         return 0;

package/dist/cursor-events.js

@@ -0,0 +1,37 @@
+export const CURSOR_EVENT_BUFFER_CAPACITY = 10_000;
+export class CursorEventBuffer {
+    samples = new Array(CURSOR_EVENT_BUFFER_CAPACITY);
+    head = 0;
+    size = 0;
+    dropped = 0;
+    push(event) {
+        if (this.size === CURSOR_EVENT_BUFFER_CAPACITY) {
+            this.samples[this.head] = event.params;
+            this.head = (this.head + 1) % CURSOR_EVENT_BUFFER_CAPACITY;
+            this.dropped++;
+            return;
+        }
+        this.samples[(this.head + this.size) % CURSOR_EVENT_BUFFER_CAPACITY] = event.params;
+        this.size++;
+    }
+    take(sessionId) {
+        const samples = [];
+        for (let index = 0; index < this.size; index++) {
+            const slot = (this.head + index) % CURSOR_EVENT_BUFFER_CAPACITY;
+            const sample = this.samples[slot];
+            if (sample?.sessionId === sessionId)
+                samples.push(sample);
+            this.samples[slot] = undefined;
+        }
+        const dropped = this.dropped;
+        this.head = 0;
+        this.size = 0;
+        this.dropped = 0;
+        return { samples, dropped };
+    }
+}
+export function formatCursorSamples(samples, dropped) {
+    return samples.length
+        ? `cursor[${samples.length}] dropped:${dropped} {sessionId,x,y,visible,timestamp_us}:\n${samples.map(sample => `  ${JSON.stringify(sample)}`).join('\n')}`
+        : `cursor: no pending samples; dropped:${dropped}`;
+}

package/dist/index.js

@@ -20,9 +20,10 @@ else if (process.argv[2] === '--session-hook') {
 }
 else {
     const definitions = [
-        ['start', [], ['control', 'source', 'display', 'timeout']], ['stop', [], []], ['health', [], []],
+        ['start', [], ['source', 'display', 'timeout', 'cursor']], ['stop', [], []], ['health', [], []], ['cursor', [], []],
         ['screen', [], ['query', 'region', 'full', 'fields']], ['diff', [], ['include-animating', 'fields']],
         ['tree', [], ['query', 'full', 'fields']], ['marks', [], ['region', 'out']],
+        ['point', [], ['label', 'timeout', 'clear']],
         ['look', [], ['region', 'out']], ['click', ['target'], ['button', 'double', 'wait']],
         ['type', ['text'], ['into', 'submit', 'wait']], ['press', ['key'], ['wait']],
         ['scroll', ['direction'], ['at', 'amount', 'wait']], ['drag', ['from', 'to'], ['wait']],
@@ -34,7 +35,7 @@ else {
     Object.assign(commands, browserCommands());
     for (const [name, required, flags] of definitions)
         commands[name] = {
-            spec: { name, summary: `${name} on the live desktop`, args: required.map(arg => ({ name: arg, required: true, description: arg })), flags: flags.map(flag => ({ name: flag, type: ['control', 'full', 'double', 'submit', 'include-animating'].includes(flag) ? 'boolean' : 'string', values: flag === 'source' ? ['auto', 'portal', 'x11', 'display'] : flag === 'button' ? ['left', 'right'] : undefined, description: flag })), examples: [`desklink-axi ${name} ${required.map(arg => `<${arg}>`).join(' ')}`] },
+            spec: { name, summary: `${name} on the live desktop`, args: name === 'point' ? [{ name: 'x,y', required: false, description: 'selected desktop pixels; omit for --clear' }] : required.map(arg => ({ name: arg, required: true, description: arg })), flags: flags.map(flag => ({ name: flag, type: ['full', 'double', 'submit', 'include-animating', 'clear'].includes(flag) ? 'boolean' : 'string', values: flag === 'source' ? ['auto', 'portal', 'x11', 'display'] : flag === 'button' ? ['left', 'right'] : undefined, description: flag })), examples: name === 'point' ? ['desklink-axi point 400,300 --label "Here" --timeout 3000', 'desklink-axi point --clear'] : [`desklink-axi ${name} ${required.map(arg => `<${arg}>`).join(' ')}`] },
             async run(parsed) {
                 if (name === 'scroll' && !['up', 'down'].includes(parsed.positionals[0]))
                     throw new UsageError('scroll direction must be up or down');
@@ -67,5 +68,8 @@ else {
             },
         };
     const registry = { tool: 'desklink-axi', root: homeCommand, rootHelp, commands };
-    process.exit(await dispatch(registry, process.argv.slice(2)));
+    // 0.3 scripts may still pass --control; paired sessions always have access.
+    const argv = process.argv.slice(2);
+    const flagsEnd = argv.indexOf('--');
+    process.exit(await dispatch(registry, argv.filter((arg, index) => !(argv[0] === 'start' && arg === '--control' && (flagsEnd < 0 || index < flagsEnd)))));
 }

package/dist/skill/content.js

@@ -2,7 +2,7 @@ import { homedir } from 'node:os';
 import { emitList } from '../output/toon.js';
 export const DESCRIPTION = "See and drive a live Linux or macOS desktop and an opted-in browser through desklink; text and changes by default, pixels on request";
 export const SPEC_VERSION = 'axi/1.0-2026-07';
-const commands = ['start', 'stop', 'health', 'screen', 'tree', 'marks', 'diff', 'look', 'click', 'type', 'press', 'scroll', 'drag', 'batch', 'wait', 'clipboard read', 'clipboard write', 'setup hooks', 'browser attach', 'browser launch', 'browser tabs', 'browser select', 'browser open', 'browser navigate', 'browser close', 'browser snapshot', 'browser click', 'browser fill', 'browser press', 'browser upload', 'browser detach'];
-export function renderHome(binPath) { return `desklink-axi: ${binPath.replace(homedir(), '~')} — ${DESCRIPTION}\nhelp[5]:\n  desklink-axi start --control --source x11 --display :97\n  desklink-axi screen --query "<words>"\n  desklink-axi tree --query "<words>"\n  desklink-axi look @r1\n  desklink-axi browser snapshot`; }
-export function rootHelpText() { return [`desklink-axi: ${DESCRIPTION}`, emitList('commands', commands.map(command => ({ command, summary: command.startsWith('browser') ? `${command} on the attached browser` : `${command} on the live desktop` })), ['command', 'summary']), 'flags[1]{flag,description}:\n  --help,show help for any command', 'examples[2]:\n  desklink-axi start --control --source x11 --display :97\n  desklink-axi screen --query "<words>"'].join('\n'); }
-export function renderSkill() { return `---\nname: desklink-axi\ndescription: "${DESCRIPTION}"\n---\n\n# desklink-axi\n\n${DESCRIPTION} (${SPEC_VERSION}).\n\n\`\`\`\n${renderHome('desklink-axi')}\n\`\`\`\n\nTwo lanes: the desktop lane (capture, OCR refs, AT-SPI tree, input) and the browser lane (semantic web control through an opted-in Chromium). On the desktop, \`tree --query "<words>"\` reads the AT-SPI accessibility tree into generation-tagged \`@a<gen>.<n>\` refs (values included), and \`marks\` overlays numbered candidates for icon-only controls. For web pages use the browser lane: \`desklink-axi browser attach --cdp 127.0.0.1:<port>\` for a browser you started with \`--remote-debugging-port\` (loopback only), or \`desklink-axi browser launch\` for an isolated task-owned profile. Then \`browser tabs\`, \`browser snapshot\` (compact accessibility refs, no screenshots), \`browser click/fill/upload @e<N>\`, \`browser select <tab>\`. Page selection (selected) and the browser's actually activated tab (front) are reported separately. OS-level dialogs are not-supported in the browser lane: fall back to the desktop lane (click/type on the dialog).\n\nRead with screen --query or --region before screen --full. Actions acknowledge input immediately; add --wait change|settle|<milliseconds> when needed. Batch sequential steps in one socket round trip: desklink-axi batch '[["press","Meta+l"],["type","https://example.com"],["press","Return"],["wait","change"]]'. Steps may be {verb,args?,name?,wait?} objects; a named snapshot binds OCR refs for later @name.n steps, and assert steps (\`["assert","text"]\`, \`["assert","--absent","text"]\`) make the batch outcome depend on observed screen text. The footer reports effects: asserted[n] or effects: unverified (input acknowledgements only) — an input ack is never a verified outcome.\n\nEvery command supports --help. Exit 0 success, 1 operational error, 2 usage error. No screen text at startup unless requested.\n`; }
+const commands = ['start', 'stop', 'health', 'screen', 'tree', 'marks', 'diff', 'look', 'point', 'click', 'type', 'press', 'scroll', 'drag', 'batch', 'wait', 'clipboard read', 'clipboard write', 'setup hooks', 'browser attach', 'browser launch', 'browser tabs', 'browser select', 'browser open', 'browser navigate', 'browser close', 'browser snapshot', 'browser click', 'browser fill', 'browser press', 'browser upload', 'browser detach'];
+export function renderHome(binPath) { return `desklink-axi: ${binPath.replace(homedir(), '~')} — ${DESCRIPTION}\nhelp[5]:\n  desklink-axi start --source x11 --display :97\n  desklink-axi screen --query "<words>"\n  desklink-axi tree --query "<words>"\n  desklink-axi look @r1\n  desklink-axi browser snapshot`; }
+export function rootHelpText() { return [`desklink-axi: ${DESCRIPTION}`, emitList('commands', commands.map(command => ({ command, summary: command.startsWith('browser') ? `${command} on the attached browser` : `${command} on the live desktop` })), ['command', 'summary']), 'flags[1]{flag,description}:\n  --help,show help for any command', 'examples[2]:\n  desklink-axi start --source x11 --display :97\n  desklink-axi screen --query "<words>"'].join('\n'); }
+export function renderSkill() { return `---\nname: desklink-axi\ndescription: "${DESCRIPTION}"\n---\n\n# desklink-axi\n\n${DESCRIPTION} (${SPEC_VERSION}).\n\n\`\`\`\n${renderHome('desklink-axi')}\n\`\`\`\n\nTwo lanes: the desktop lane (capture, OCR refs, AT-SPI tree, input) and the browser lane (semantic web control through an opted-in Chromium). On the desktop, \`tree --query "<words>"\` reads the AT-SPI accessibility tree into generation-tagged \`@a<gen>.<n>\` refs (values included), and \`marks\` overlays numbered candidates for icon-only controls. For web pages use the browser lane: \`desklink-axi browser attach --cdp 127.0.0.1:<port>\` for a browser you started with \`--remote-debugging-port\` (loopback only), or \`desklink-axi browser launch\` for an isolated task-owned profile. Then \`browser tabs\`, \`browser snapshot\` (compact accessibility refs, no screenshots), \`browser click/fill/upload @e<N>\`, \`browser select <tab>\`. Page selection (selected) and the browser's actually activated tab (front) are reported separately. OS-level dialogs are not-supported in the browser lane: fall back to the desktop lane (click/type on the dialog).\n\nRead with screen --query or --region before screen --full. Actions acknowledge input immediately; add --wait change|settle|<milliseconds> when needed. Batch sequential steps in one socket round trip: desklink-axi batch '[["press","Meta+l"],["type","https://example.com"],["press","Return"],["wait","change"]]'. Steps may be {verb,args?,name?,wait?} objects; a named snapshot binds OCR refs for later @name.n steps, and assert steps (\`["assert","text"]\`, \`["assert","--absent","text"]\`) make the batch outcome depend on observed screen text. The footer reports effects: asserted[n] or effects: unverified (input acknowledgements only) — an input ack is never a verified outcome.\n\nGuide without input using \`point 400,300 --label "Here" --timeout 3000\`; use \`point --clear\` to remove the cue. See the [host protocol reference](../../../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources and limits. Every command supports --help. Exit 0 success, 1 operational error, 2 usage error. No screen text at startup unless requested.\n`; }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@desklink/axi",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Agent-facing desktop control CLI",
   "type": "module",
   "bin": {
@@ -24,15 +24,15 @@
     "skill:check": "node scripts/gen-skill.mjs --check"
   },
   "devDependencies": {
-    "@types/node": "^22.10.0",
-    "@types/pngjs": "^6.0.5",
-    "typescript": "^5.6.0"
+    "@types/node": "22.20.4",
+    "@types/pngjs": "6.0.5",
+    "typescript": "5.9.3"
   },
   "dependencies": {
-    "@desklink/host": "0.3.0",
-    "node-datachannel": "^0.33.4",
+    "@desklink/host": "0.4.0",
+    "node-datachannel": "0.33.4",
     "playwright-core": "1.62.1",
-    "pngjs": "^7.0.0"
+    "pngjs": "7.0.0"
   },
   "license": "Apache-2.0"
 }

package/skills/desklink-axi/SKILL.md

@@ -10,7 +10,7 @@ See and drive a live Linux or macOS desktop and an opted-in browser through desk
 ```
 desklink-axi: desklink-axi — See and drive a live Linux or macOS desktop and an opted-in browser through desklink; text and changes by default, pixels on request
 help[5]:
-  desklink-axi start --control --source x11 --display :97
+  desklink-axi start --source x11 --display :97
   desklink-axi screen --query "<words>"
   desklink-axi tree --query "<words>"
   desklink-axi look @r1
@@ -21,4 +21,4 @@ Two lanes: the desktop lane (capture, OCR refs, AT-SPI tree, input) and the brow
 
 Read with screen --query or --region before screen --full. Actions acknowledge input immediately; add --wait change|settle|<milliseconds> when needed. Batch sequential steps in one socket round trip: desklink-axi batch '[["press","Meta+l"],["type","https://example.com"],["press","Return"],["wait","change"]]'. Steps may be {verb,args?,name?,wait?} objects; a named snapshot binds OCR refs for later @name.n steps, and assert steps (`["assert","text"]`, `["assert","--absent","text"]`) make the batch outcome depend on observed screen text. The footer reports effects: asserted[n] or effects: unverified (input acknowledgements only) — an input ack is never a verified outcome.
 
-Every command supports --help. Exit 0 success, 1 operational error, 2 usage error. No screen text at startup unless requested.
+Guide without input using `point 400,300 --label "Here" --timeout 3000`; use `point --clear` to remove the cue. See the [host protocol reference](../../../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources and limits. Every command supports --help. Exit 0 success, 1 operational error, 2 usage error. No screen text at startup unless requested.