@aryacode/dsh-plugin-arya-computer-use 1.0.4 → 1.0.5

package/README.md

@@ -36,7 +36,7 @@ Replaces [dsh-computer-use-win](https://github.com/Yu-tao-Li/dsh-computer-use-wi
 | 3 | Web lane (`arya_browser` row), sign-in profiles, attach-to-your-Chrome, **wincu retirement** | planned |
 | 4 | Lazy preset, notifications/clipboard/trace, npm publish | planned |
 
-Lite 16 tool surface: `snapshot, find, get, is, click, set_value, type, press, scroll, list_windows, launch, wait, screenshot, ocr, batch, tools` (superset of wincu Lite 12; `set_value` fills a field through the accessibility value pattern without touching the pointer; `type` follows the input lane (background by default, real keyboard only when that conversation opts in); `tools` manages headed/overlay/session state).
+Lite 16 tool surface: `snapshot, find, get, is, click, set_value, type, press, scroll, list_windows, launch, wait, screenshot, ocr, batch, tools` (superset of wincu Lite 12; `set_value` fills a field through the accessibility value pattern without touching the pointer — **the headless text path, use it for text entry**; `type` synthesizes real keystrokes, which on Windows is physical input, so it moves the pointer and focuses the window; `tools` manages headed/overlay/session state).
 
 Full plan: [docs/PLAN.md](docs/PLAN.md).
 
@@ -77,7 +77,7 @@ The published package is `@aryacode/dsh-plugin-arya-computer-use` (npm, public).
 | Show a ghost cursor | on | Draw a synthetic cursor where the agent acts; your pointer is never followed |
 | Agent name | `Arya` | Drives the ghost-cursor tag and the progress-panel header (`Arya · Step 3 · acting`), 1–16 characters |
 | Banner text | `Arya is controlling your computer` | One line, 1–120 characters |
-| Real mouse and keyboard | off | Opted-in headed input for one conversation; the background lane never moves the pointer |
+| Real mouse and keyboard | off | Opted-in headed input for one conversation: **every** action then drives the real pointer and keyboard. It releases on `session_end` and expires after 5 minutes. Text entry never needs it — `set_value` is headless; `type` is physical on its own. |
 | One conversation drives at a time | on | Refuses desktop tools in other conversations while a drive is live |
 | Stop shortcut | Esc | Global kill switch; synthetic keystrokes are ignored, so it needs a real keypress |
 | Tool preset | `lite` | `lite` lists all 16 desktop tools; `lazy` starts with tools/snapshot/click and reveals the rest on demand |

package/client.js

@@ -37,7 +37,7 @@ window.__ModuleLoader__.load({
         progressPosition: '浮窗位置', progress_topRight: '右上', progress_topLeft: '左上', progress_bottomRight: '右下', progress_bottomLeft: '左下', progress_custom: '自定义',
         progressX: 'X 坐标', progressY: 'Y 坐标', progressShortcut: '浮窗解锁快捷键', progressShortcutHint: '按下后临时解锁浮窗以便拖动。留空表示不设置。',
         toolSurfaceTitle: '工具范围', toolPreset: '工具预设', toolPresetHint: 'lite 直接列出全部 16 个工具；lazy 只暴露 tools / snapshot / click，其余按需加载。修改后需重启 MCP 生效。', toolPresetLite: 'lite · 全部 16 个', toolPresetLazy: 'lazy · 按需加载',
-  behaviorTitle: '操作方式', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input, which briefly focuses the target window.',
+  behaviorTitle: '操作方式', headedDefault: '真实鼠标和键盘', headedDefaultHint: '默认关闭：输入通过后台无障碍模式完成，不会移动你的指针，也不会抢占焦点。开启后所有操作都使用真实键盘和鼠标，并会短暂聚焦目标窗口。type 工具始终使用真实按键并移动指针；set_value 在后台写入文本。',
         overlayDefault: '引擎会话叠加层', overlayDefaultHint: '启动桌面会话时启用引擎自带的光标叠加层。Arya 的虚拟光标不依赖它。',
         driverLock: '单会话驱动', driverLockHint: '同一时间只允许一个对话操作桌面；其他对话调用桌面工具会收到 BUSY 提示。关闭后最近使用的对话自动接管。',
         browserTitle: '网页通道', browserTools: '工具范围', browserToolsHint: 'agent-browser 向 MCP 暴露的工具集。修改后需重启 MCP 生效。',
@@ -89,7 +89,7 @@ window.__ModuleLoader__.load({
         progressPosition: 'Overlay position', progress_topRight: 'Top right', progress_topLeft: 'Top left', progress_bottomRight: 'Bottom right', progress_bottomLeft: 'Bottom left', progress_custom: 'Custom',
         progressX: 'X coordinate', progressY: 'Y coordinate', progressShortcut: 'Overlay unlock shortcut', progressShortcutHint: 'Press to temporarily unlock the overlay so you can drag it. Empty means none.',
         toolSurfaceTitle: 'Tool surface', toolPreset: 'Tool preset', toolPresetHint: 'lite lists all 16 tools up front; lazy exposes tools / snapshot / click and loads the rest on demand. Changes apply after the next MCP restart.', toolPresetLite: 'lite · all 16', toolPresetLazy: 'lazy · load on demand',
-  behaviorTitle: 'Control style', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input, which briefly focuses the target window.',
+  behaviorTitle: 'Control style', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input for every action, which briefly focuses the target window. The type tool always uses real keystrokes and moves the pointer; set_value writes text in the background.',
         overlayDefault: 'Engine session overlay', overlayDefaultHint: "Starts the desktop session with the engine's own cursor overlay. Arya's ghost cursor does not depend on it; leave it off unless you want the engine marker too.",
         driverLock: 'One conversation drives at a time', driverLockHint: 'While one conversation is driving, desktop calls from other conversations receive a BUSY notice. Turn off to let the most recent conversation take over automatically.',
         browserTitle: 'Web lane', browserTools: 'Tool surface', browserToolsHint: 'Which agent-browser tool profiles the MCP server exposes. Changing this applies on the next MCP restart.',

package/docs/PLAN.md

@@ -69,6 +69,7 @@ Web lane tools (second row `arya_browser` → `mcp__arya_browser__*`): open, sna
 - First desktop call in a conversation → adapter runs `agent-desktop session start` (with `--cursor` when overlay enabled) → session id; every call injects `--session <id> --agent-id <conversation-id>` → refs are namespaced per conversation (no cross-conversation collisions).
 - Turn end / Esc / session end → `cursor-overlay disable` + `session end` (renderer self-reclaims ≤3000 ms; held input released via SIGTERM semantics).
 - **Headless default.** Per-conversation toggle: `arya_tools {operation:"headed", on:true}` — adapter injects `--headed` for that conversation. Skill documents when headed is required: `type`, right-click, double/triple-click, focus, cursor-moving input, notifications, shell surfaces (start-menu/tray/action-center).
+  - **Headed is a latch, so it must expire.** `runCli` appends `--headed` to *every* command, so a latch left on turns all later clicks/scrolls/launches into physical input — that is a mouse/keyboard takeover. `type` therefore passes `--headed` **per call** (headless `type` is `POLICY_DENIED` on Windows, so it is physical by nature and must not latch the conversation), the latch is released by `session_end`, and `HEADED_LEASE_MS` (5 min) self-releases it. Text entry should use `set_value`, which is headless.
 - State root: `AGENT_DESKTOP_HOME=%LOCALAPPDATA%\dsh-plugin-arya-desktop\state` (set in row env).
 
 ## 5. Takeover UI (parity checklist)
@@ -88,7 +89,7 @@ Overlay cost: ~355 ms per headless action (click 427→782 ms). Setting default:
 
 - Elevated/admin windows: UIPI boundary fails closed natively (agent-desktop) — same practical effect as wincu's admin protection; UAC secure desktop unaddressable by both.
 - Occlusion/hidden/disabled targets: agent-desktop actionability checks (stronger than wincu POINT_COVERED: identity corroboration immediately before each write; `AMBIGUOUS_TARGET` refuses identical twins instead of mis-clicking).
-- Foreground borrow-and-return: adapter restores the user's foreground window after headed actions.
+- Foreground: headed actions **steal** the foreground and `ensureForeground()` promotes with a headed `mouse-click --xy` at the target's bounds center. **Borrow-and-return is NOT implemented** — no code restores the user's foreground window, so a headed action leaves the target focused. Mitigated (not fixed) by scoping `type` to one headed call, releasing the latch on `session_end`, and the 5-minute lease. A real restore is open work.
 - Batch: native `desktop_batch` + adapter guard (1–20 steps, stop on first error/dialog/new-window, per-step delivery disposition).
 - Blocked system shortcuts (alt+f4, win+l, alt+tab, …) refused natively without `--force`; `--force` not exposed by default.
 - Delivery honesty: `delivered_verified / delivered_unverified / not_delivered` + `retry: safe|unsafe` — skill teaches: never retry unsafe; read `details.post_state`.

package/mcp/adapter-server.mjs

@@ -62,6 +62,9 @@ const SETTINGS_FILE = join(PLUGIN_DIR, 'control', 'settings.json');
 // The overlay host reads this to frame the app window that is being driven and
 // to place the ghost cursor where the harness acts — never the real pointer.
 const TARGET_FILE = join(PLUGIN_DIR, 'control', 'target.json');
+// The host takeover lane persists its paused/stopped series here. The adapter
+// watches it so the user's Esc stop also drops the physical-input latch.
+const TAKEOVER_STATE_FILE = join(PLUGIN_DIR, 'control', 'state.json');
 
 const VERSION = '0.1.0';
 const MAX_BATCH = 20;
@@ -70,6 +73,7 @@ const state = {
   sessionId: `arya-${randomBytes(4).toString('hex')}`,
   sessionStarted: false,
   headedOverride: null, // tools {operation:'headed'} per-conversation toggle
+  headedExpiresAt: 0, // headedOverride self-releases: a latch left on is a mouse/keyboard takeover
   overlayOverride: null,
   headedDefault: false,
   overlayDefault: true,
@@ -92,13 +96,26 @@ function notifyToolsChanged() {
   // swaps its registered tool generation.
   send({ jsonrpc: '2.0', method: 'notifications/tools/list_changed' });
 }
-const effectiveHeaded = () => state.headedOverride ?? state.headedDefault;
+// A headed latch left on makes EVERY later call physical (runCli appends
+// --headed to all of them), so it releases on its own. Per-call --headed is the
+// normal path; the latch is only an escape hatch for surfaces that need a
+// sequence of physical actions.
+const HEADED_LEASE_MS = 5 * 60 * 1000;
+const effectiveHeaded = () => {
+  if (state.headedOverride === true && Date.now() > state.headedExpiresAt) {
+    state.headedOverride = null;
+    state.headedExpiresAt = 0;
+  }
+  return state.headedOverride ?? state.headedDefault;
+};
 const effectiveOverlay = () => state.overlayOverride ?? state.overlayDefault;
 
 setInterval(() => {
   try {
     const settings = JSON.parse(readFileSync(SETTINGS_FILE, 'utf8'));
-    state.headedDefault = settings.headedDefault !== false;
+    // Strict: a settings file that omits the key must mean OFF. `!== false`
+    // read a missing key as ON, which is a silent mouse/keyboard takeover.
+    state.headedDefault = settings.headedDefault === true;
     state.overlayDefault = settings.overlayDefault !== false;
     const preset = settings.toolPreset === 'lazy' ? 'lazy' : 'lite';
     if (preset !== state.toolPreset) {
@@ -109,6 +126,17 @@ setInterval(() => {
   } catch {
     // Settings file not written yet (plugin not loaded): keep the defaults.
   }
+  try {
+    // Esc / stop lives in the host lane. A stopped series means the user pushed
+    // back: release the headed latch so the next series starts headless.
+    const takeover = JSON.parse(readFileSync(TAKEOVER_STATE_FILE, 'utf8'));
+    if (takeover?.paused === true && state.headedOverride === true) {
+      state.headedOverride = null;
+      state.headedExpiresAt = 0;
+    }
+  } catch {
+    // No takeover state yet.
+  }
 }, 1000).unref();
 
 /* ------------------------------------------------------- overlay target */
@@ -323,8 +351,8 @@ const TOOLS = [
   tool('get', 'Read one control: property text|value|title|bounds|role|states.', { ref_id: str, property: str }),
   tool('is', 'Check one control: visible|enabled|checked|focused|expanded|selected.', { ref_id: str, property: str }),
   tool('click', 'Press a control via its semantic action (no pixels). Verify by re-snapshot.', { ref_id: str, timeout_ms: int }),
-  tool('set_value', 'Set a control value via UIA SetValue (no keystrokes). Some edits reject this — use type.', { ref_id: str, value: str, timeout_ms: int }),
-  tool('type', 'Type keystrokes into a text target. Background accessibility input by default; with headed on it briefly focuses the window and uses real keystrokes.', { ref_id: str, text: str, interval_ms: int, timeout_ms: int }),
+  tool('set_value', 'Set a control value via UIA SetValue (no keystrokes, no pointer movement). This is the headless text path — use it first for any text entry. Some edits reject it — only then use type.', { ref_id: str, value: str, timeout_ms: int }),
+  tool('type', 'Type real keystrokes into a text target. On Windows keystroke synthesis is physical: this moves the pointer and briefly focuses the window. Prefer set_value (headless UIA write) for text entry.', { ref_id: str, text: str, interval_ms: int, timeout_ms: int }),
   tool('press', 'Send a key chord (e.g. ctrl+s) to an app/window. Foreground queue: for non-frontmost apps expect delivered_unverified.', { combo: str, app: str, window_id: str, force: bool }),
   tool('scroll', 'Scroll a scrollable control: direction up|down|left|right, amount in lines.', { ref_id: str, direction: str, amount: int, timeout_ms: int }),
   tool('list_windows', 'List top-level windows (ids, titles, pids, responsiveness).', { app: str, timeout_ms: int }),
@@ -396,13 +424,15 @@ async function callTool(name, args = {}) {
     case 'type':
       return enqueue(async () => {
         await ensureSession();
-        const headed = effectiveHeaded();
-        if (headed) {
-          const promote = await ensureForeground(args.ref_id);
-          if (promote) return envelopeResult(promote);
-        }
-        const env = parseEnvelope(await runCli(['type', args.ref_id, args.text ?? '', ...flagArgs([['--interval-ms', args.interval_ms], ['--timeout-ms', args.timeout_ms]])], { headed }));
-        if (!headed && env && env.ok !== true) env.hint = 'Background delivery cannot synthesize keystrokes for this target. Prefer set_value, or enable real input for this conversation: tools {operation:"headed",on:true} (moves the pointer), then type again.';
+        // Engine fact on Windows: headless `type` is POLICY_DENIED — keystroke
+        // synthesis is headed-only. `type` is therefore physical by nature, so
+        // --headed is scoped to THIS call and the conversation is never latched
+        // into it. Latching was what made every later click/scroll move the
+        // user's real pointer.
+        const promote = await ensureForeground(args.ref_id);
+        if (promote) return envelopeResult(promote);
+        const env = parseEnvelope(await runCli(['type', args.ref_id, args.text ?? '', ...flagArgs([['--interval-ms', args.interval_ms], ['--timeout-ms', args.timeout_ms]])], { headed: true }));
+        if (env && env.ok !== true) env.hint = 'Keystroke synthesis is physical: it moves the pointer and focuses the window. Prefer set_value, which writes through the UIA value pattern headless.';
         return env;
       });
     case 'press':
@@ -460,7 +490,8 @@ async function callTool(name, args = {}) {
         }
         if (op === 'headed') {
           state.headedOverride = args.on === true;
-          return { content: [{ type: 'text', text: JSON.stringify({ ok: true, headed: effectiveHeaded(), headedDefault: state.headedDefault }) }] };
+          state.headedExpiresAt = args.on === true ? Date.now() + HEADED_LEASE_MS : 0;
+          return { content: [{ type: 'text', text: JSON.stringify({ ok: true, headed: effectiveHeaded(), headedDefault: state.headedDefault, lease_ms: args.on === true ? HEADED_LEASE_MS : 0, note: 'Headed drives the real mouse and keyboard for every action in this conversation. It releases on session_end and expires on its own. Prefer set_value for text so headed is not needed at all.' }) }] };
         }
         if (op === 'overlay') {
           state.overlayOverride = args.on === true;
@@ -506,6 +537,9 @@ async function callTool(name, args = {}) {
           await runCli(['session', 'gc']);
           state.sessionStarted = false;
           state.sessionId = `arya-${randomBytes(4).toString('hex')}`;
+          // Ending the driver session also releases the physical-input latch.
+          state.headedOverride = null;
+          state.headedExpiresAt = 0;
           return { content: [{ type: 'text', text: JSON.stringify({ ok: env.ok === true, note: 'new session starts on next action', engine: env }) }] };
         }
         return {
@@ -583,7 +617,8 @@ const INSTRUCTIONS = [
   'Arya Computer Use: drive Windows apps through semantic UIA refs, not pixels.',
   'Loop: snapshot (skeleton/compact first) -> act on @refs -> re-snapshot to verify.',
   'Refs go stale after window changes: on STALE_REF/AMBIGUOUS_TARGET, re-snapshot.',
-  'type uses background accessibility input by default and never moves the pointer; tools {operation:"headed",on:true} switches this conversation to real keyboard/mouse input when a surface needs it.',
+  'set_value is the headless text path (UIA value pattern, pointer stays put). type synthesizes real keystrokes: it moves the pointer and focuses the window, scoped to that one call.',
+  'Headed mode (tools {operation:"headed",on:true}) makes EVERY action in this conversation physical input. Use it only when background input is refused; it releases on session_end and expires after 5 minutes.',
   'Chromium windows settle slowly: snapshot with timeout_ms 10000-25000.',
   'Verify stateful actions by re-read; delivered_verified = confirmed, delivered_unverified retry:unsafe = do not blind-repeat.',
   'On ACTION_FAILED post_action_verification the action landed: read data.post_state, do NOT retry.',

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aryacode/dsh-plugin-arya-computer-use",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "description": "Computer use for DSH/AryaCode: agent-desktop UIA engine, agent-browser web lane, takeover UI with a ghost cursor",
   "license": "MIT",
   "type": "module",
@@ -58,6 +58,9 @@
     "bundle": {
       "patch": "./cordis.patch.yml"
     },
+    "engines": {
+      "dsh": ">=0.2.1-rc.3 <0.3.0-0"
+    },
     "client": {
       "platform": "web",
       "inject": [

package/skills/arya-desktop/SKILL.md

@@ -18,11 +18,11 @@ Screenshots are evidence, not input.
    thin and grow: pass `timeout_ms: 10000`–`25000` and re-snapshot after the
    first contact.
 2. **Act** — one action on one ref: `click`, `set_value`, `type`, `press`,
-   `scroll`. Prefer `set_value` for edits: it writes through the accessibility
-   value pattern and never touches the pointer. `type` needs real keystrokes, so
-   in the default background mode it only works on targets that accept
-   synthetic text; when it fails, either use `set_value` or enable headed input
-   for this conversation with `tools {operation:"headed",on:true}`.
+   `scroll`. **Use `set_value` for every text entry**: it writes through the
+   accessibility value pattern, headless, and never touches the pointer. `type`
+   synthesizes real keystrokes — on Windows that is physical input, so it moves
+   the pointer and briefly focuses the window. It is the last resort, only for
+   an edit that rejects `set_value`.
 3. **Verify** — re-`snapshot` (or `get`/`is` on the ref). The envelope tells
    you what happened:
    - `delivered_verified` — confirmed by readback.
@@ -43,11 +43,14 @@ Screenshots are evidence, not input.
   `lease_directory_untrusted`, tell the user: restart AryaCode as Administrator.
 - **Headed toggle.** Default is headless (no mouse/keyboard takeover): semantic
   UIA input goes straight to the control and never moves the user's pointer.
-  When headless cannot do the job (cursor-moving input, notifications, start
-  menu, tray, surfaces that reject background input), call
-  `tools {operation:"headed",on:true}` for this conversation, then `on:false`
-  when done. Headed input briefly focuses the target window and drives the real
-  keyboard/mouse, so warn the user before switching it on.
+  Headed is an **escape hatch for surfaces that refuse background input**
+  (notifications, start menu, tray, right-click, drag, double-click). It is not
+  the answer to a text field — `set_value` does that headless. Turning it on
+  makes **every** action in this conversation physical: clicks move the real
+  pointer, keystrokes go to the foreground window. So: warn the user before
+  switching it on, call `tools {operation:"headed",on:false}` when the surface
+  is done, and end with `tools {operation:"session_end"}`, which releases it.
+  A latch left on also expires by itself after 5 minutes.
 - **Launch.** Bare names resolve only under System32 (`notepad` works,
   `chrome` does not). Use the absolute exe path. If many instances run, the
   engine refuses to guess — that is correct; ask the user.