@desklink/axi 0.3.1 → 0.4.0

package/README.md

@@ -24,14 +24,14 @@ Or build from the repository root with `npm install && npm run build`.
 Delete copied `*.tsbuildinfo` before rebuilding a moved worktree.
 The engine is resolved by `@desklink/host`; see its [installation guide](../desktop-host/README.md#install-and-run)
 for supported platforms and [macOS setup](../desktop-host/README.md#macos-preview).
-Set `DESKLINK_AXI_ENGINE` for a source build.
+Set `DESKLINK_AXI_ENGINE` for a matching source build. An override pointing new AXI at a 0.3.x engine is unsupported; host and engine versions must match.
 
 A Linux desktop AXI with macOS in preview. It runs a private bridge per session, captures read-only frames through `@desklink/host` protocol v3, and sends input only through a WebRTC `control` channel. The bridge's Unix socket lives in `$XDG_RUNTIME_DIR/desklink-axi` (or the OS temp directory), with `DESKLINK_AXI_SESSION` selecting an isolated socket. It opens a separate engine session; don't drive the desktop while someone else is controlling it.
 
-For macOS source prerequisites, signing and permission attribution, follow the [host setup guide](../desktop-host/README.md#macos-preview). To launch AXI through its signed local harness, use `open -n /absolute/DesklinkHost.app --args axi-bridge /absolute/path/to/packages/axi/bin/desklink-axi.js --control` from the logged-in Mac session. OCR needs the optional `tesseract` executable (`brew install tesseract`, `sudo pacman -S tesseract` or `sudo apt install tesseract-ocr`); on macOS AXI also finds Homebrew's standard install paths when LaunchServices omits them from PATH. `look` works without it.
+For macOS source prerequisites, signing and permission attribution, follow the [host setup guide](../desktop-host/README.md#macos-preview). To launch AXI through its signed local harness, use `open -n /absolute/DesklinkHost.app --args axi-bridge /absolute/path/to/packages/axi/bin/desklink-axi.js` from the logged-in Mac session. OCR needs the optional `tesseract` executable (`brew install tesseract`, `sudo pacman -S tesseract` or `sudo apt install tesseract-ocr`); on macOS AXI also finds Homebrew's standard install paths when LaunchServices omits them from PATH. `look` works without it.
 
 ```
-desklink-axi start --control --source x11 --display :97
+desklink-axi start --source x11 --display :97
 desklink-axi screen --query "Budget"
 desklink-axi tree --query "Save"
 desklink-axi type "hello" --into @a3.10
@@ -47,7 +47,7 @@ desklink-axi stop
 
 `start --cursor hidden` opts into cursor-free capture; omit it for existing behavior. Use `desklink-axi cursor` to drain source-coordinate cursor samples (`x`, `y`, `visible`, `timestamp_us`) from the live session; `dropped:N` reports older samples overwritten after its 10,000-sample buffer fills. Linux portals require cursor metadata and refuse hidden capture when it is unavailable; see the [cursor protocol](../desktop-host/docs/PROTOCOL.md#cursor-free-capture).
 
-`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` defaults to view-only; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/portal-token` (otherwise `~/.local/state/desklink-axi/portal-token`), atomically replaced with mode 0600 after each grant. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard read/write require `start --control`; view-only clipboard calls return a structured error naming that flag. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
+`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` grants all available access to its same-user paired consumer; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/portal-token` (otherwise `~/.local/state/desklink-axi/portal-token`), atomically replaced with mode 0600 after each grant. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard need no app grants; unavailable OS backends fail per action. Legacy `start --control` is accepted and ignored. See the [clipboard protocol](../desktop-host/docs/PROTOCOL.md#clipboard-over-the-local-protocol) for X11 targets, transfer limits and selection lifetime. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
 To guide the person without input, use `desklink-axi point 400,300 --label "Here" --timeout 3000`, or `desklink-axi point --clear`. See the [host protocol reference](../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources, coordinates, and limits.
 
 ## Browser lane (opt-in, loopback-only)
@@ -111,7 +111,7 @@ Measured on a private 1280×720 Xvfb client: `diff → click 100,100 → look @r
 
 ## Accessibility tree (`tree`) — Linux AT-SPI
 
-`tree` reads the desktop's AT-SPI accessibility tree over D-Bus and returns bounded TOON rows of `ref,role,name,value,x,y,w,h,states`. It needs no `--control`; acting on refs does. Refs are generation-tagged (`@a<gen>.<n>`), stay usable across newer snapshots, and are revalidated at use time: a ref whose role, name, or screen extents changed — or whose object is gone — returns `stale-ref` instead of acting blind.
+`tree` reads the desktop's AT-SPI accessibility tree over D-Bus and returns bounded TOON rows of `ref,role,name,value,x,y,w,h,states`. Read and input actions share the same paired session. Refs are generation-tagged (`@a<gen>.<n>`), stay usable across newer snapshots, and are revalidated at use time: a ref whose role, name, or screen extents changed — or whose object is gone — returns `stale-ref` instead of acting blind.
 
 - Capability probe: the bus is resolved from `AT_SPI_BUS_ADDRESS`, else `org.a11y.Bus.GetAddress` on the session bus. Unavailable → structured `a11y-unavailable` error; every other command keeps working. macOS AX is **not implemented** and labeled as such.
 - Values: text and value widgets expose their content in the `value` column (Text/Value interfaces), so a typed entry can be verified without pixels.

package/dist/bridge.js

@@ -107,7 +107,6 @@ export async function serve(args) {
         throw new Error('session already running');
     const source = args.includes('--source') ? args[args.indexOf('--source') + 1] : 'auto';
     const display = args.includes('--display') ? args[args.indexOf('--display') + 1] : undefined;
-    const control = args.includes('--control');
     const cursor = args.includes('--cursor') ? args[args.indexOf('--cursor') + 1] : 'embedded';
     if (cursor !== 'embedded' && cursor !== 'hidden')
         throw new Error('cursor: use embedded or hidden');
@@ -164,13 +163,13 @@ export async function serve(args) {
             source: process.platform === 'darwin'
                 ? { kind: 'display', ...(display === undefined ? {} : { display_id: Number(display) }) }
                 : source === 'x11' || (source === 'auto' && display) ? { kind: 'x11', display } : { kind: 'portal' },
-            permissions: control ? ['view', 'control', 'clipboard'] : ['view'], loopbackTcp: true,
-            agentIndicator: control,
+            loopbackTcp: true,
+            agentIndicator: true, // serve rejects platforms other than Linux/macOS above.
             cursor,
             ...(tokenFile ? { restoreToken: takeToken(tokenFile) } : {}),
         }, timeout);
         checkToken();
-        peer = control ? new PeerConnection('desklink-axi', { iceServers: [], bindAddress: '127.0.0.1', enableIceTcp: true }) : undefined;
+        peer = new PeerConnection('desklink-axi', { iceServers: [], bindAddress: '127.0.0.1', enableIceTcp: true });
         let channel;
         let open = false;
         let nextSeq = 1;
@@ -275,8 +274,6 @@ export async function serve(args) {
             return [...lines.values()];
         };
         const act = async (message) => {
-            if (!control)
-                throw new Error('input-unavailable: start --control');
             for (let i = 0; i < 100 && !open; i++)
                 await new Promise(r => setTimeout(r, 50));
             if (!open || !channel)
@@ -321,7 +318,7 @@ export async function serve(args) {
             let waitSeen = baseline;
             routeNote = undefined;
             if (command === 'start')
-                return `session: open source=${source} ${display ?? ''} size=${opened.geometry.encoded.width}x${opened.geometry.encoded.height} permissions=view${control ? ',control' : ''}`;
+                return `session: open source=${source} ${display ?? ''} size=${opened.geometry.encoded.width}x${opened.geometry.encoded.height}`;
             if (command === 'cursor') {
                 const batch = cursorSamples.take(opened.sessionId);
                 return formatCursorSamples(batch.samples, batch.dropped);
@@ -436,8 +433,6 @@ export async function serve(args) {
                 return 'session: stopped';
             }
             if (command === 'clipboard') {
-                if (!control)
-                    throw new Error('input-unavailable: clipboard requires start --control');
                 if (args[0] === 'read') {
                     const result = await engine.readClipboard(opened.sessionId);
                     return `clipboard: ${args.includes('--full') ? result.text : result.text.slice(0, 1000)} (${result.text.length} chars)`;
@@ -656,6 +651,15 @@ export async function serve(args) {
             if (fields.some(field => !['ref', 'text', 'x', 'y', 'w', 'h', 'conf', 'line'].includes(field)))
                 throw new Error('fields: valid fields are ref,text,x,y,w,h,conf,line');
             const frame = await capture(pendingDamage ? waitSeen : baseline);
+            // A frame reply can already be older than the latest change notice. Its
+            // stillness must not erase motion from a newer frame. Classify at capture
+            // time too: synchronous OCR can outlast the motion freshness window.
+            const moving = [...motion].filter(([, state]) => state.count >= 3 && Date.now() - state.at < 600 && (frame.still_ms < 150 || state.seq > frame.seq));
+            if (frame.still_ms >= 150)
+                for (const [region, state] of motion) {
+                    if (state.seq <= frame.seq)
+                        motion.delete(region);
+                }
             if (pendingDamage)
                 frame.damage = [...new Set([...pendingDamage, ...frame.damage])];
             if (command === 'look') {
@@ -744,7 +748,7 @@ export async function serve(args) {
                 }
             }
             if (command === 'home')
-                return `session: open source=${source} ${display ?? ''} size=${frame.width}x${frame.height} permissions=view${control ? ',control' : ''}\nframe: ${frame.seq} settled=${frame.still_ms >= 150} unseen=${frame.changed ? frame.damage.length : 0} region\nwindows: unavailable on this compositor\ntext[${Math.min(text.length, 12)} of ${text.length}]{ref,text,x,y}:\n${text.slice(0, 12).map(t => `  ${t.ref},${JSON.stringify(t.text)},${t.x},${t.y}`).join('\n')}\nhelp[2]:\n  desklink-axi diff\n  desklink-axi screen --query "<words>"`;
+                return `session: open source=${source} ${display ?? ''} size=${frame.width}x${frame.height}\nframe: ${frame.seq} settled=${frame.still_ms >= 150} unseen=${frame.changed ? frame.damage.length : 0} region\nwindows: unavailable on this compositor\ntext[${Math.min(text.length, 12)} of ${text.length}]{ref,text,x,y}:\n${text.slice(0, 12).map(t => `  ${t.ref},${JSON.stringify(t.text)},${t.x},${t.y}`).join('\n')}\nhelp[2]:\n  desklink-axi diff\n  desklink-axi screen --query "<words>"`;
             if (command === 'screen') {
                 const query = args.includes('--query') ? args[args.indexOf('--query') + 1] : undefined;
                 const items = query ? text.filter(t => t.text.toLowerCase().includes(query.toLowerCase())) : text;
@@ -774,11 +778,8 @@ export async function serve(args) {
                     return false;
                 });
                 const gone = unmatched.length;
-                if (frame.still_ms >= 150)
-                    motion.clear();
                 const animating = [];
                 const visible = [];
-                const moving = [...motion].filter(([, state]) => state.count >= 3 && Date.now() - state.at < 600);
                 for (const region of [...frame.damage, ...moving.map(([key]) => key).filter(key => !frame.damage.some(region => overlaps(key, region)))]) {
                     const state = moving.find(([key]) => overlaps(key, region))?.[1];
                     const [x, y, w, h] = region.split(',').map(Number);
@@ -797,8 +798,6 @@ export async function serve(args) {
                 const changed = visible.length ? `${visible.length} region since frame ${frame.previous}` : `none since frame ${frame.previous}`;
                 return `changed: ${changed}\nregions[${visible.length}]{ref,box}:\n${visible.map((d, i) => `  @r${i + 1},"${d}"`).join('\n')}\nappeared[${Math.min(appeared.length, 20)} of ${appeared.length}]{${fields.join(',')}}:\n${appeared.slice(0, 20).map(t => `  ${fields.map(field => JSON.stringify(field === 'w' ? t.words.at(-1).x + t.words.at(-1).w - t.x : field === 'h' ? Math.max(...t.words.map(w => w.y + w.h)) - t.y : t[field])).join(',')}`).join('\n')}\ngone: ${gone} text items${animating.length ? `\nanimating[${animating.length}]{box}:\n${animating.map(box => `  "${box}"`).join('\n')}\nanimating: ${animating.join('; ')}` : ''}\nhelp[2]:\n  desklink-axi look @r1\n  desklink-axi screen --query "<words>"`;
             }
-            if (frame.still_ms >= 150)
-                motion.clear();
             return `changed: none since frame ${frame.seq} (still ${frame.still_ms}ms)\nhelp[1]:\n  desklink-axi screen --query "<words>"`;
         };
         let commands = Promise.resolve();

package/dist/index.js

@@ -20,7 +20,7 @@ else if (process.argv[2] === '--session-hook') {
 }
 else {
     const definitions = [
-        ['start', [], ['control', 'source', 'display', 'timeout', 'cursor']], ['stop', [], []], ['health', [], []], ['cursor', [], []],
+        ['start', [], ['source', 'display', 'timeout', 'cursor']], ['stop', [], []], ['health', [], []], ['cursor', [], []],
         ['screen', [], ['query', 'region', 'full', 'fields']], ['diff', [], ['include-animating', 'fields']],
         ['tree', [], ['query', 'full', 'fields']], ['marks', [], ['region', 'out']],
         ['point', [], ['label', 'timeout', 'clear']],
@@ -35,7 +35,7 @@ else {
     Object.assign(commands, browserCommands());
     for (const [name, required, flags] of definitions)
         commands[name] = {
-            spec: { name, summary: `${name} on the live desktop`, args: name === 'point' ? [{ name: 'x,y', required: false, description: 'selected desktop pixels; omit for --clear' }] : required.map(arg => ({ name: arg, required: true, description: arg })), flags: flags.map(flag => ({ name: flag, type: ['control', 'full', 'double', 'submit', 'include-animating', 'clear'].includes(flag) ? 'boolean' : 'string', values: flag === 'source' ? ['auto', 'portal', 'x11', 'display'] : flag === 'button' ? ['left', 'right'] : undefined, description: flag })), examples: name === 'point' ? ['desklink-axi point 400,300 --label "Here" --timeout 3000', 'desklink-axi point --clear'] : [`desklink-axi ${name} ${required.map(arg => `<${arg}>`).join(' ')}`] },
+            spec: { name, summary: `${name} on the live desktop`, args: name === 'point' ? [{ name: 'x,y', required: false, description: 'selected desktop pixels; omit for --clear' }] : required.map(arg => ({ name: arg, required: true, description: arg })), flags: flags.map(flag => ({ name: flag, type: ['full', 'double', 'submit', 'include-animating', 'clear'].includes(flag) ? 'boolean' : 'string', values: flag === 'source' ? ['auto', 'portal', 'x11', 'display'] : flag === 'button' ? ['left', 'right'] : undefined, description: flag })), examples: name === 'point' ? ['desklink-axi point 400,300 --label "Here" --timeout 3000', 'desklink-axi point --clear'] : [`desklink-axi ${name} ${required.map(arg => `<${arg}>`).join(' ')}`] },
             async run(parsed) {
                 if (name === 'scroll' && !['up', 'down'].includes(parsed.positionals[0]))
                     throw new UsageError('scroll direction must be up or down');
@@ -68,5 +68,8 @@ else {
             },
         };
     const registry = { tool: 'desklink-axi', root: homeCommand, rootHelp, commands };
-    process.exit(await dispatch(registry, process.argv.slice(2)));
+    // 0.3 scripts may still pass --control; paired sessions always have access.
+    const argv = process.argv.slice(2);
+    const flagsEnd = argv.indexOf('--');
+    process.exit(await dispatch(registry, argv.filter((arg, index) => !(argv[0] === 'start' && arg === '--control' && (flagsEnd < 0 || index < flagsEnd)))));
 }

package/dist/skill/content.js

@@ -3,6 +3,6 @@ import { emitList } from '../output/toon.js';
 export const DESCRIPTION = "See and drive a live Linux or macOS desktop and an opted-in browser through desklink; text and changes by default, pixels on request";
 export const SPEC_VERSION = 'axi/1.0-2026-07';
 const commands = ['start', 'stop', 'health', 'screen', 'tree', 'marks', 'diff', 'look', 'point', 'click', 'type', 'press', 'scroll', 'drag', 'batch', 'wait', 'clipboard read', 'clipboard write', 'setup hooks', 'browser attach', 'browser launch', 'browser tabs', 'browser select', 'browser open', 'browser navigate', 'browser close', 'browser snapshot', 'browser click', 'browser fill', 'browser press', 'browser upload', 'browser detach'];
-export function renderHome(binPath) { return `desklink-axi: ${binPath.replace(homedir(), '~')} — ${DESCRIPTION}\nhelp[5]:\n  desklink-axi start --control --source x11 --display :97\n  desklink-axi screen --query "<words>"\n  desklink-axi tree --query "<words>"\n  desklink-axi look @r1\n  desklink-axi browser snapshot`; }
-export function rootHelpText() { return [`desklink-axi: ${DESCRIPTION}`, emitList('commands', commands.map(command => ({ command, summary: command.startsWith('browser') ? `${command} on the attached browser` : `${command} on the live desktop` })), ['command', 'summary']), 'flags[1]{flag,description}:\n  --help,show help for any command', 'examples[2]:\n  desklink-axi start --control --source x11 --display :97\n  desklink-axi screen --query "<words>"'].join('\n'); }
+export function renderHome(binPath) { return `desklink-axi: ${binPath.replace(homedir(), '~')} — ${DESCRIPTION}\nhelp[5]:\n  desklink-axi start --source x11 --display :97\n  desklink-axi screen --query "<words>"\n  desklink-axi tree --query "<words>"\n  desklink-axi look @r1\n  desklink-axi browser snapshot`; }
+export function rootHelpText() { return [`desklink-axi: ${DESCRIPTION}`, emitList('commands', commands.map(command => ({ command, summary: command.startsWith('browser') ? `${command} on the attached browser` : `${command} on the live desktop` })), ['command', 'summary']), 'flags[1]{flag,description}:\n  --help,show help for any command', 'examples[2]:\n  desklink-axi start --source x11 --display :97\n  desklink-axi screen --query "<words>"'].join('\n'); }
 export function renderSkill() { return `---\nname: desklink-axi\ndescription: "${DESCRIPTION}"\n---\n\n# desklink-axi\n\n${DESCRIPTION} (${SPEC_VERSION}).\n\n\`\`\`\n${renderHome('desklink-axi')}\n\`\`\`\n\nTwo lanes: the desktop lane (capture, OCR refs, AT-SPI tree, input) and the browser lane (semantic web control through an opted-in Chromium). On the desktop, \`tree --query "<words>"\` reads the AT-SPI accessibility tree into generation-tagged \`@a<gen>.<n>\` refs (values included), and \`marks\` overlays numbered candidates for icon-only controls. For web pages use the browser lane: \`desklink-axi browser attach --cdp 127.0.0.1:<port>\` for a browser you started with \`--remote-debugging-port\` (loopback only), or \`desklink-axi browser launch\` for an isolated task-owned profile. Then \`browser tabs\`, \`browser snapshot\` (compact accessibility refs, no screenshots), \`browser click/fill/upload @e<N>\`, \`browser select <tab>\`. Page selection (selected) and the browser's actually activated tab (front) are reported separately. OS-level dialogs are not-supported in the browser lane: fall back to the desktop lane (click/type on the dialog).\n\nRead with screen --query or --region before screen --full. Actions acknowledge input immediately; add --wait change|settle|<milliseconds> when needed. Batch sequential steps in one socket round trip: desklink-axi batch '[["press","Meta+l"],["type","https://example.com"],["press","Return"],["wait","change"]]'. Steps may be {verb,args?,name?,wait?} objects; a named snapshot binds OCR refs for later @name.n steps, and assert steps (\`["assert","text"]\`, \`["assert","--absent","text"]\`) make the batch outcome depend on observed screen text. The footer reports effects: asserted[n] or effects: unverified (input acknowledgements only) — an input ack is never a verified outcome.\n\nGuide without input using \`point 400,300 --label "Here" --timeout 3000\`; use \`point --clear\` to remove the cue. See the [host protocol reference](../../../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources and limits. Every command supports --help. Exit 0 success, 1 operational error, 2 usage error. No screen text at startup unless requested.\n`; }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@desklink/axi",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "Agent-facing desktop control CLI",
   "type": "module",
   "bin": {
@@ -29,7 +29,7 @@
     "typescript": "5.9.3"
   },
   "dependencies": {
-    "@desklink/host": "0.3.1",
+    "@desklink/host": "0.4.0",
     "node-datachannel": "0.33.4",
     "playwright-core": "1.62.1",
     "pngjs": "7.0.0"

package/skills/desklink-axi/SKILL.md

@@ -10,7 +10,7 @@ See and drive a live Linux or macOS desktop and an opted-in browser through desk
 ```
 desklink-axi: desklink-axi — See and drive a live Linux or macOS desktop and an opted-in browser through desklink; text and changes by default, pixels on request
 help[5]:
-  desklink-axi start --control --source x11 --display :97
+  desklink-axi start --source x11 --display :97
   desklink-axi screen --query "<words>"
   desklink-axi tree --query "<words>"
   desklink-axi look @r1