@desklink/axi 0.5.0 → 0.5.1

package/README.md

@@ -45,9 +45,9 @@ desklink-axi look @r1
 desklink-axi stop
 ```
 
-`start --cursor hidden` opts into cursor-free capture; omit it for existing behavior. Use `desklink-axi cursor` to drain source-coordinate cursor samples (`x`, `y`, `visible`, `timestamp_us`) from the live session; `dropped:N` reports older samples overwritten after its 10,000-sample buffer fills. Linux portals require cursor metadata and refuse hidden capture when it is unavailable; see the [cursor protocol](../desktop-host/docs/PROTOCOL.md#cursor-free-capture).
+`start --cursor hidden` opts into cursor-free capture; omit it for existing behavior. Use `desklink-axi cursor` to drain source-coordinate cursor samples (`x`, `y`, `visible`, `timestamp_us`, plus `hotspot` where the compositor reports it) from the live session; `dropped:N` reports older samples overwritten after its 10,000-sample buffer fills. On Wayland a hidden session still runs when positions are unavailable and says why in `cursor.limitation`; see the [cursor protocol](../desktop-host/docs/PROTOCOL.md#cursor-free-capture).
 
-`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` grants all available access to its same-user paired consumer; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/portal-token` (otherwise `~/.local/state/desklink-axi/portal-token`), atomically replaced with mode 0600 after each grant. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard need no app grants; unavailable OS backends fail per action. Legacy `start --control` is accepted and ignored. See the [clipboard protocol](../desktop-host/docs/PROTOCOL.md#clipboard-over-the-local-protocol) for X11 targets, transfer limits and selection lifetime. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
+`health` reports capture state, latest frame sequence, and a stream-stop reason without OCR. `start` grants all available access to its same-user paired consumer; Linux uses the portal unless `--source x11 --display :N` is given, and macOS uses the main display unless `--source display --display <id>` is given. The portal can require a desktop consent prompt; its single-use restore token is kept in `$XDG_STATE_HOME/desklink-axi/<session>.portal-token` (otherwise `~/.local/state/desklink-axi/<session>.portal-token`), atomically replaced with mode 0600 after each grant; `<session>` is `DESKLINK_AXI_SESSION`, the same name the session's socket and browser state use, so two lanes never hand each other's grant to their own consumer. Set `DESKLINK_AXI_RESTORE_TOKEN_FILE` to use a different path. macOS Screen Recording and Accessibility belong to the app that launches the bridge and its engine child. Do not run it unattended against someone else's display. Input and clipboard need no app grants; unavailable OS backends fail per action. Legacy `start --control` is accepted and ignored. See the [clipboard protocol](../desktop-host/docs/PROTOCOL.md#clipboard-over-the-local-protocol) for X11 targets, transfer limits and selection lifetime. `stop` releases held input. Actions acknowledge `input: applied` immediately, without waiting for a frame. Every action (`click`, `type`, `press`, `scroll`, `drag`) accepts `--wait settle|change|none|<milliseconds>`; frame timeout after applied input returns `input: applied; frame: timed out` with success status. `batch` takes a JSON array of 1–30 `[verb, ...args]` steps (click/type/press/scroll/wait/snapshot/assert/tree/marks), runs them in order in one socket round trip, stops on the first failed step and returns a compact per-step result. OCR refs survive advancing frame numbers only while their text and position still match a fresh crop; moved or ambiguous text requires `screen --query`. `screen --query` reuses a prior matching text location for a targeted crop before falling back to full OCR, and reports estimated text tokens. Start with `screen --query` or `--region` before `--full`. For web pages prefer DOM/browser automation when available; AXI is for native apps or whole-desktop tasks. `wait change` and `wait settle` block on engine frame signals rather than polling. `diff` reports repeatedly changing regions separately as `animating`; `diff --include-animating` includes them among ordinary changes. `look` writes a PNG crop. `--help` lists the commands and their flags. Errors are structured and the CLI does not prompt.
 To guide the person without input, use `desklink-axi point 400,300 --label "Here" --timeout 3000`, or `desklink-axi point --clear`. See the [host protocol reference](../desktop-host/docs/PROTOCOL.md#point-on-the-host-desktop) for supported sources, coordinates, and limits.
 
 ## Browser lane (opt-in, loopback-only)
@@ -105,6 +105,8 @@ For a remote signed Mac bridge, start it once **inside the permission-granted ap
 
 For an isolated integration proof run `DESKLINK_AXI_ENGINE=/path/to/engine CARGO_TARGET_DIR=/tmp/task-owned-target node packages/axi/test/flow.mjs` from the repository root. It launches and kills its own Xvfb process by exact PID. `test/a11y-flow.mjs` is the accessibility + marks proof: it also starts a private D-Bus session bus, a private AT-SPI bus, and a GTK3 fixture app (all task-owned, private `XDG_RUNTIME_DIR`), asserts semantic-ref driving of form controls and a file dialog, and prints per-command p50/p95 with `--measure 20`. It skips itself when `busctl`/`dbus-daemon`/`at-spi2-core`/GTK are absent and never touches the user's accessibility bus.
 
+For the 200-word OCR and query proof, add `DESKLINK_OCR_SCALE=1` or `DESKLINK_OCR_SCALE=1.5` and `DESKLINK_OCR_EVIDENCE=/absolute/output/directory` to that command. The latter uses a private 3840×2160 display at 1.5 scale. Each run saves the ground truth, OCR, word diff, screenshot and every visible-label query. Add `DESKLINK_OCR_UPDATE_README=1` to the 1x run to replace the README transcript directly with CLI output; no transcript lines are edited by hand. Use `DESKLINK_OCR_SHOWCASE_ONLY=1` with the 1x command to replay the edit and regenerate the transcript without repeating the label-query checks. This proof needs Chromium and Tesseract. Known limitation: a text insertion caret visible in the captured pixels may be read as part of nearby item text (for example a trailing `|`), because frames carry pixels without caret identity. Full-frame OCR upscales 3x in software before Tesseract, so a 3840x2160 frame transiently needs a 11520x6480 (~74.6M px, ~298MB RGBA) buffer plus PNG encode plus Tesseract on the enlarged image.
+
 To check the CLI against the AXI spec, run `axi-axi validate "node bin/desklink-axi.js" --dir .` from `packages/axi`.
 
 Measured on a private 1280×720 Xvfb client: `diff → click 100,100 → look @r1 → type abc` returned 79 + 79 + 39 + 30 = **227 text tokens** (`@anthropic-ai/tokenizer` 0.0.4), plus ~5 tokens for the 64×64 crop. Four full screenshots would cost ~4×1,229 = **4,916 image tokens** under the same published width×height/750 estimate (~21× difference). This is a short synthetic task, not a production-workload benchmark; the tokenizer is legacy, so absolute counts are approximate.

package/dist/bridge.js

@@ -249,21 +249,57 @@ export async function serve(args) {
         };
         const ocr = async (frame, box = [0, 0, frame.width, frame.height]) => {
             const imagePath = join(dir, 'ocr.png');
-            await image(frame, box, imagePath);
-            const { spawnSync } = await import('node:child_process');
+            // UI glyphs at native size are often below Tesseract's useful x-height.
+            // Enlarge the captured pixels, rather than correcting recognized words.
+            const scale = 3;
+            const input = await cropBuffer(frame, box);
+            const { PNG } = await import('pngjs');
+            const enlarged = new PNG({ width: input.width * scale, height: input.height * scale });
+            for (let y = 0; y < enlarged.height; y++) {
+                const sy = Math.max(0, Math.min(input.height - 1, (y + 0.5) / scale - 0.5));
+                const top = Math.floor(sy), bottom = Math.min(input.height - 1, top + 1), fy = sy - top;
+                for (let x = 0; x < enlarged.width; x++) {
+                    const sx = Math.max(0, Math.min(input.width - 1, (x + 0.5) / scale - 0.5));
+                    const left = Math.floor(sx), right = Math.min(input.width - 1, left + 1), fx = sx - left;
+                    const to = (y * enlarged.width + x) * 4;
+                    for (let c = 0; c < 3; c++) {
+                        const a = input.data[(top * input.width + left) * 4 + c];
+                        const b = input.data[(top * input.width + right) * 4 + c];
+                        const d = input.data[(bottom * input.width + left) * 4 + c];
+                        const e = input.data[(bottom * input.width + right) * 4 + c];
+                        enlarged.data[to + c] = Math.round((a * (1 - fx) + b * fx) * (1 - fy) + (d * (1 - fx) + e * fx) * fy);
+                    }
+                    enlarged.data[to + 3] = 255;
+                }
+            }
+            writeFileSync(imagePath, PNG.sync.write(enlarged));
             const homebrew = ['/opt/homebrew/bin/tesseract', '/usr/local/bin/tesseract'].find(existsSync);
-            const result = spawnSync(process.platform === 'darwin' && homebrew ? homebrew : 'tesseract', [imagePath, 'stdout', 'tsv'], { encoding: 'utf8' });
-            if (result.error || result.status !== 0)
-                throw new Error('ocr-unavailable: install tesseract (brew install tesseract, sudo pacman -S tesseract or sudo apt install tesseract-ocr)');
+            // Keep receiving capture and input events while recognition is running.
+            const output = await new Promise((resolve, reject) => {
+                const child = spawn(process.platform === 'darwin' && homebrew ? homebrew : 'tesseract', [imagePath, 'stdout', 'tsv'], { stdio: ['ignore', 'pipe', 'ignore'] });
+                let result = '';
+                child.stdout.setEncoding('utf8');
+                child.stdout.on('data', chunk => { result += chunk; });
+                const unavailable = () => reject(new Error('ocr-unavailable: install tesseract (brew install tesseract, sudo pacman -S tesseract or sudo apt install tesseract-ocr)'));
+                child.once('error', unavailable);
+                child.once('close', code => { if (code === 0)
+                    resolve(result);
+                else
+                    unavailable(); });
+            });
             const lines = new Map();
-            for (const cols of result.stdout.split('\n').slice(1).map(line => line.split('\t'))) {
+            const rows = output.split('\n').slice(1).map(line => line.split('\t'));
+            for (const cols of rows) {
                 if (cols.length < 12 || Number(cols[10]) < 0 || !cols[11]?.trim())
                     continue;
+                const token = cols[11].trim();
+                if (!token)
+                    continue;
                 const key = cols.slice(1, 5).join(':');
                 if (!lines.has(key))
                     lines.set(key, { ref: `@${frame.seq}.${lines.size + 1}`, text: '', x: Number(cols[6]), y: Number(cols[7]), conf: Number(cols[10]), line: key, words: [] });
                 const item = lines.get(key);
-                const word = { text: cols[11].trim(), x: box[0] + Number(cols[6]), y: box[1] + Number(cols[7]), w: Number(cols[8]), h: Number(cols[9]) };
+                const word = { text: token, x: box[0] + Math.floor(Number(cols[6]) / scale), y: box[1] + Math.floor(Number(cols[7]) / scale), w: Math.ceil(Number(cols[8]) / scale), h: Math.ceil(Number(cols[9]) / scale) };
                 if (!item.text) {
                     item.x = word.x;
                     item.y = word.y;
@@ -464,14 +500,14 @@ export async function serve(args) {
                 const current = await capture(baseline);
                 observed = undefined; // Validation must not consume the diff baseline.
                 if (current.seq !== baseline) {
+                    // Cropping through a widget border or a caret changes segmentation.
+                    // Validate against the same full-screen context that produced the ref.
+                    const currentText = await ocr(current);
                     for (const ref of args.filter(arg => /^@\d+\.\d+$/.test(arg))) {
                         const item = text.find(t => t.ref === ref);
                         if (!item)
                             throw new Error(`stale-ref: ${ref}; run screen --query`);
-                        const x = Math.max(0, item.x - 32), y = Math.max(0, item.y - 32);
-                        const right = Math.min(current.width, item.words.at(-1).x + item.words.at(-1).w + 32);
-                        const bottom = Math.min(current.height, Math.max(...item.words.map(w => w.y + w.h)) + 32);
-                        if (!(await ocr(current, [x, y, right - x, bottom - y])).some(t => t.text === item.text && Math.abs(t.x - item.x) < 12 && Math.abs(t.y - item.y) < 12))
+                        if (!currentText.some(t => t.text === item.text && Math.abs(t.x - item.x) < 12 && Math.abs(t.y - item.y) < 12))
                             throw new Error(`stale-ref: ${ref} moved or changed; run screen --query`);
                     }
                 }
@@ -760,12 +796,15 @@ export async function serve(args) {
                 const prev = text;
                 let kept = [...prev];
                 const dirty = [];
+                // Recognize complete lines in full context before selecting damage. A
+                // cropped caret or widget border must not turn unchanged words into fragments.
+                const currentText = await ocr(frame);
                 for (const region of frame.damage) {
                     const [rx, ry, rw, rh] = region.split(',').map(Number);
                     const x = Math.max(0, rx - 32), y = Math.max(0, ry - 32);
                     const right = Math.min(frame.width, rx + rw + 32), bottom = Math.min(frame.height, ry + rh + 32);
                     kept = kept.filter(t => !t.words.some(w => w.x < right && w.x + w.w > x && w.y < bottom && w.y + w.h > y));
-                    dirty.push(...await ocr(frame, [x, y, right - x, bottom - y]));
+                    dirty.push(...currentText.filter(t => t.words.some(w => w.x < right && w.x + w.w > x && w.y < bottom && w.y + w.h > y)));
                 }
                 text = [...kept, ...dirty.filter((t, i) => !dirty.slice(0, i).some(p => p.text === t.text && Math.abs(p.x - t.x) < 12 && Math.abs(p.y - t.y) < 12))]
                     .map((t, i) => ({ ...t, ref: `@${frame.seq}.${i + 1}` }));
@@ -795,7 +834,7 @@ export async function serve(args) {
                 }
                 if (observed)
                     observed.damage = visible;
-                const changed = visible.length ? `${visible.length} region since frame ${frame.previous}` : `none since frame ${frame.previous}`;
+                const changed = visible.length ? `${visible.length} region${visible.length === 1 ? '' : 's'} since frame ${frame.previous}` : `none since frame ${frame.previous}`;
                 return `changed: ${changed}\nregions[${visible.length}]{ref,box}:\n${visible.map((d, i) => `  @r${i + 1},"${d}"`).join('\n')}\nappeared[${Math.min(appeared.length, 20)} of ${appeared.length}]{${fields.join(',')}}:\n${appeared.slice(0, 20).map(t => `  ${fields.map(field => JSON.stringify(field === 'w' ? t.words.at(-1).x + t.words.at(-1).w - t.x : field === 'h' ? Math.max(...t.words.map(w => w.y + w.h)) - t.y : t[field])).join(',')}`).join('\n')}\ngone: ${gone} text items${animating.length ? `\nanimating[${animating.length}]{box}:\n${animating.map(box => `  "${box}"`).join('\n')}\nanimating: ${animating.join('; ')}` : ''}\nhelp[2]:\n  desklink-axi look @r1\n  desklink-axi screen --query "<words>"`;
             }
             return `changed: none since frame ${frame.seq} (still ${frame.still_ms}ms)\nhelp[1]:\n  desklink-axi screen --query "<words>"`;

package/dist/cursor-events.js

@@ -32,6 +32,6 @@ export class CursorEventBuffer {
 }
 export function formatCursorSamples(samples, dropped) {
     return samples.length
-        ? `cursor[${samples.length}] dropped:${dropped} {sessionId,x,y,visible,timestamp_us}:\n${samples.map(sample => `  ${JSON.stringify(sample)}`).join('\n')}`
+        ? `cursor[${samples.length}] dropped:${dropped} {sessionId,x,y,visible,timestamp_us,hotspot?}:\n${samples.map(sample => `  ${JSON.stringify(sample)}`).join('\n')}`
         : `cursor: no pending samples; dropped:${dropped}`;
 }

package/dist/token.js

@@ -2,8 +2,12 @@ import { closeSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, write
 import { homedir } from 'node:os';
 import { dirname, join } from 'node:path';
 import { randomUUID } from 'node:crypto';
+// Scoped by the session, exactly like the socket and the browser state beside
+// it: one shared name per user let a second lane's grant overwrite the first
+// lane's single-use token, and each lane's consumer then read the other's.
 export function tokenPath() {
-    return process.env.DESKLINK_AXI_RESTORE_TOKEN_FILE ?? join(process.env.XDG_STATE_HOME ?? join(homedir(), '.local', 'state'), 'desklink-axi', 'portal-token');
+    return process.env.DESKLINK_AXI_RESTORE_TOKEN_FILE
+        ?? join(process.env.XDG_STATE_HOME ?? join(homedir(), '.local', 'state'), 'desklink-axi', `${process.env.DESKLINK_AXI_SESSION ?? 'default'}.portal-token`);
 }
 // A portal token is single-use. Remove it from the reusable name before open;
 // a failed open must prompt again rather than replay a possibly consumed grant.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@desklink/axi",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "description": "Agent-facing desktop control CLI",
   "type": "module",
   "bin": {
@@ -29,7 +29,7 @@
     "typescript": "5.9.3"
   },
   "dependencies": {
-    "@desklink/host": "0.5.0",
+    "@desklink/host": "0.5.1",
     "node-datachannel": "0.33.4",
     "playwright-core": "1.62.1",
     "pngjs": "7.0.0"