@aryacode/dsh-plugin-arya-computer-use 1.0.4 → 1.0.6

package/README.md

@@ -8,7 +8,7 @@ Replaces [dsh-computer-use-win](https://github.com/Yu-tao-Li/dsh-computer-use-wi
 
 | wincu complaint | Arya Computer Use |
 | --- | --- |
-| Slow (PowerShell + UIA backend) | Native Rust engine: snapshot ~117 ms, ref action ~197 ms on a Windows devbox |
+| Slow (PowerShell + UIA backend) | Native Rust engine: ~90 ms per call measured on this devbox (snapshot 90 ms, ref action 85 ms, `list-windows` 56 ms). Every CLI call costs ~85 ms of process start plus UIA init, so the number of calls, not the tool, sets the latency — `batch` runs up to 20 steps in one call (6 steps: 332 ms engine vs ~510 ms step by step) |
 | Pixel alignment off | Ref-based semantic UIA — coordinates are never the targeting mechanism; identical twins are refused, not mis-clicked |
 | Chinese UI/docs | English throughout |
 | (kept) DSH integration + setup UI | Settings page, chat step rows, takeover overlay, Esc kill, tool presets |
@@ -31,12 +31,12 @@ Replaces [dsh-computer-use-win](https://github.com/Yu-tao-Li/dsh-computer-use-wi
 | Phase | Deliverable | Status |
 | --- | --- | --- |
 | 0 | Evaluation spike: raw `agent-desktop mcp` row, drive Notepad/Explorer/Chrome — speed ✓, readable trees ✓, refs-not-pixels ✓ | ✅ accepted, spike removed |
-| 1 | Adapter MCP server: Lite 16 tools, per-conversation session lifecycle, headed toggle + foreground promotion, screenshot→file path, WinRT OCR, skill | ✅ built, installed, driven in-host |
+| 1 | Adapter MCP server: Lite 22 tools, per-conversation session lifecycle, headed toggle + foreground promotion, screenshot→file path, WinRT OCR, skill | ✅ built, installed, driven in-host |
 | 2 | Takeover UI: window rim + banner + Esc kill, configurable ghost cursor, settings page, chat step rows, driver lock, English UI | ✅ built + verified live |
 | 3 | Web lane (`arya_browser` row), sign-in profiles, attach-to-your-Chrome, **wincu retirement** | planned |
 | 4 | Lazy preset, notifications/clipboard/trace, npm publish | planned |
 
-Lite 16 tool surface: `snapshot, find, get, is, click, set_value, type, press, scroll, list_windows, launch, wait, screenshot, ocr, batch, tools` (superset of wincu Lite 12; `set_value` fills a field through the accessibility value pattern without touching the pointer; `type` follows the input lane (background by default, real keyboard only when that conversation opts in); `tools` manages headed/overlay/session state).
+Lite 22 tool surface: `snapshot, find, get, is, click, focus, double_click, right_click, expand, collapse, toggle, set_value, type, press, scroll, list_windows, launch, wait, screenshot, ocr, batch, tools` (superset of wincu Lite 12; `set_value` fills a field through the accessibility value pattern without touching the pointer — **the headless text path, use it for text entry**; `expand`/`collapse`/`toggle` are semantic UIA patterns and stay headless; `focus`, `double_click` and `right-click` have no headless path on Windows, so each one drives the real pointer for that single call and never latches the conversation; `type` likewise synthesizes real keystrokes, which on Windows is physical input; `tools` manages headed/overlay/session state).
 
 Full plan: [docs/PLAN.md](docs/PLAN.md).
 
@@ -47,7 +47,7 @@ package.json          Cordis bundle manifest (dsh.bundle.patch) + pinned engine
 cordis.patch.yml      rows: arya-computer-use (Host), mcp-arya-desktop, mcp-arya-browser
 index.js              Host plugin entry (settings, driver lock, takeover mgmt, status route)
 client.js             Settings page + chat step rows (client plugin)
-mcp/adapter-server.mjs  adapter MCP server (Lite 16 tool surface)
+mcp/adapter-server.mjs  adapter MCP server (Lite 22 tool surface)
 scripts/windows-takeover.cs  window rim + banner + progress + ghost cursor + global Esc
 scripts/ocr.ps1       WinRT OCR helper (Phase 1)
 skills/arya-desktop/  observe-act loop guide (Phase 1)
@@ -77,10 +77,11 @@ The published package is `@aryacode/dsh-plugin-arya-computer-use` (npm, public).
 | Show a ghost cursor | on | Draw a synthetic cursor where the agent acts; your pointer is never followed |
 | Agent name | `Arya` | Drives the ghost-cursor tag and the progress-panel header (`Arya · Step 3 · acting`), 1–16 characters |
 | Banner text | `Arya is controlling your computer` | One line, 1–120 characters |
-| Real mouse and keyboard | off | Opted-in headed input for one conversation; the background lane never moves the pointer |
+| Real mouse and keyboard | off | Opted-in headed input for one conversation: **every** action then drives the real pointer and keyboard. It releases on `session_end` and expires after 5 minutes. Text entry never needs it — `set_value` is headless; `type` is physical on its own. |
 | One conversation drives at a time | on | Refuses desktop tools in other conversations while a drive is live |
-| Stop shortcut | Esc | Global kill switch; synthetic keystrokes are ignored, so it needs a real keypress |
-| Tool preset | `lite` | `lite` lists all 16 desktop tools; `lazy` starts with tools/snapshot/click and reveals the rest on demand |
+| Progress panel detail | `summary` | What the on-screen step panel prints: `summary` = short action lines, `source` = the assistant's raw text stream (the old default — it paints model text on your desktop) |
+| Stop shortcut | Esc | Global kill switch; synthetic keystrokes are ignored, so it needs a real keypress. It now also gates the driver: desktop input tools return `COMPUTER_USE_STOPPED_BY_USER` for that turn (reads keep working) |
+| Tool preset | `lite` | `lite` lists all 22 desktop tools; `lazy` starts with tools/snapshot/click and reveals the rest on demand |
 | Tool surface (web) | `core` | agent-browser tool profile: core 29, network 9, state 27, debug 40, tabs 13, react 8, mobile 15, webmcp 4, all 156 |
 | Chrome profile / Session | – / `arya` | Reuse a Chrome login state and name the browser session |
 | Connect to my browser | – | `--cdp` port or URL to attach to a Chrome you already have open |
@@ -110,7 +111,7 @@ repair them itself at start-up - the status route reports the outcome under `mcp
 
 ## Tool presets
 
-- `lite` (default) — all 16 desktop tools listed up front.
+- `lite` (default) — all 22 desktop tools listed up front.
 - `lazy` — starts with `tools`, `snapshot`, `click`; everything else appears after `tools {operation:"load", names:[...]}`. The adapter advertises `tools.listChanged` and emits `notifications/tools/list_changed`, which DSH's MCP client re-syncs (`listChanged.onChanged → refreshTools`), so the new tools become callable immediately.
 
 ## License

package/client.js

@@ -37,7 +37,7 @@ window.__ModuleLoader__.load({
         progressPosition: '浮窗位置', progress_topRight: '右上', progress_topLeft: '左上', progress_bottomRight: '右下', progress_bottomLeft: '左下', progress_custom: '自定义',
         progressX: 'X 坐标', progressY: 'Y 坐标', progressShortcut: '浮窗解锁快捷键', progressShortcutHint: '按下后临时解锁浮窗以便拖动。留空表示不设置。',
         toolSurfaceTitle: '工具范围', toolPreset: '工具预设', toolPresetHint: 'lite 直接列出全部 16 个工具；lazy 只暴露 tools / snapshot / click，其余按需加载。修改后需重启 MCP 生效。', toolPresetLite: 'lite · 全部 16 个', toolPresetLazy: 'lazy · 按需加载',
-  behaviorTitle: '操作方式', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input, which briefly focuses the target window.',
+  behaviorTitle: '操作方式', headedDefault: '真实鼠标和键盘', headedDefaultHint: '默认关闭：输入通过后台无障碍模式完成，不会移动你的指针，也不会抢占焦点。开启后所有操作都使用真实键盘和鼠标，并会短暂聚焦目标窗口。type 工具始终使用真实按键并移动指针；set_value 在后台写入文本。',
         overlayDefault: '引擎会话叠加层', overlayDefaultHint: '启动桌面会话时启用引擎自带的光标叠加层。Arya 的虚拟光标不依赖它。',
         driverLock: '单会话驱动', driverLockHint: '同一时间只允许一个对话操作桌面；其他对话调用桌面工具会收到 BUSY 提示。关闭后最近使用的对话自动接管。',
         browserTitle: '网页通道', browserTools: '工具范围', browserToolsHint: 'agent-browser 向 MCP 暴露的工具集。修改后需重启 MCP 生效。',
@@ -89,7 +89,7 @@ window.__ModuleLoader__.load({
         progressPosition: 'Overlay position', progress_topRight: 'Top right', progress_topLeft: 'Top left', progress_bottomRight: 'Bottom right', progress_bottomLeft: 'Bottom left', progress_custom: 'Custom',
         progressX: 'X coordinate', progressY: 'Y coordinate', progressShortcut: 'Overlay unlock shortcut', progressShortcutHint: 'Press to temporarily unlock the overlay so you can drag it. Empty means none.',
         toolSurfaceTitle: 'Tool surface', toolPreset: 'Tool preset', toolPresetHint: 'lite lists all 16 tools up front; lazy exposes tools / snapshot / click and loads the rest on demand. Changes apply after the next MCP restart.', toolPresetLite: 'lite · all 16', toolPresetLazy: 'lazy · load on demand',
-  behaviorTitle: 'Control style', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input, which briefly focuses the target window.',
+  behaviorTitle: 'Control style', headedDefault: 'Real mouse and keyboard', headedDefaultHint: 'Off by default: input goes through background accessibility patterns and never moves your pointer or steals focus. Turn on to use real keyboard/mouse input for every action, which briefly focuses the target window. The type tool always uses real keystrokes and moves the pointer; set_value writes text in the background.',
         overlayDefault: 'Engine session overlay', overlayDefaultHint: "Starts the desktop session with the engine's own cursor overlay. Arya's ghost cursor does not depend on it; leave it off unless you want the engine marker too.",
         driverLock: 'One conversation drives at a time', driverLockHint: 'While one conversation is driving, desktop calls from other conversations receive a BUSY notice. Turn off to let the most recent conversation take over automatically.',
         browserTitle: 'Web lane', browserTools: 'Tool surface', browserToolsHint: 'Which agent-browser tool profiles the MCP server exposes. Changing this applies on the next MCP restart.',

package/docs/PLAN.md

@@ -69,6 +69,7 @@ Web lane tools (second row `arya_browser` → `mcp__arya_browser__*`): open, sna
 - First desktop call in a conversation → adapter runs `agent-desktop session start` (with `--cursor` when overlay enabled) → session id; every call injects `--session <id> --agent-id <conversation-id>` → refs are namespaced per conversation (no cross-conversation collisions).
 - Turn end / Esc / session end → `cursor-overlay disable` + `session end` (renderer self-reclaims ≤3000 ms; held input released via SIGTERM semantics).
 - **Headless default.** Per-conversation toggle: `arya_tools {operation:"headed", on:true}` — adapter injects `--headed` for that conversation. Skill documents when headed is required: `type`, right-click, double/triple-click, focus, cursor-moving input, notifications, shell surfaces (start-menu/tray/action-center).
+  - **Headed is a latch, so it must expire.** `runCli` appends `--headed` to *every* command, so a latch left on turns all later clicks/scrolls/launches into physical input — that is a mouse/keyboard takeover. `type` therefore passes `--headed` **per call** (headless `type` is `POLICY_DENIED` on Windows, so it is physical by nature and must not latch the conversation), the latch is released by `session_end`, and `HEADED_LEASE_MS` (5 min) self-releases it. Text entry should use `set_value`, which is headless.
 - State root: `AGENT_DESKTOP_HOME=%LOCALAPPDATA%\dsh-plugin-arya-desktop\state` (set in row env).
 
 ## 5. Takeover UI (parity checklist)
@@ -88,7 +89,7 @@ Overlay cost: ~355 ms per headless action (click 427→782 ms). Setting default:
 
 - Elevated/admin windows: UIPI boundary fails closed natively (agent-desktop) — same practical effect as wincu's admin protection; UAC secure desktop unaddressable by both.
 - Occlusion/hidden/disabled targets: agent-desktop actionability checks (stronger than wincu POINT_COVERED: identity corroboration immediately before each write; `AMBIGUOUS_TARGET` refuses identical twins instead of mis-clicking).
-- Foreground borrow-and-return: adapter restores the user's foreground window after headed actions.
+- Foreground: headed actions **steal** the foreground and `ensureForeground()` promotes with a headed `mouse-click --xy` at the target's bounds center. **Borrow-and-return is NOT implemented** — no code restores the user's foreground window, so a headed action leaves the target focused. Mitigated (not fixed) by scoping `type` to one headed call, releasing the latch on `session_end`, and the 5-minute lease. A real restore is open work.
 - Batch: native `desktop_batch` + adapter guard (1–20 steps, stop on first error/dialog/new-window, per-step delivery disposition).
 - Blocked system shortcuts (alt+f4, win+l, alt+tab, …) refused natively without `--force`; `--force` not exposed by default.
 - Delivery honesty: `delivered_verified / delivered_unverified / not_delivered` + `retry: safe|unsafe` — skill teaches: never retry unsafe; read `details.post_state`.

package/mcp/adapter-server.mjs

@@ -62,14 +62,79 @@ const SETTINGS_FILE = join(PLUGIN_DIR, 'control', 'settings.json');
 // The overlay host reads this to frame the app window that is being driven and
 // to place the ghost cursor where the harness acts — never the real pointer.
 const TARGET_FILE = join(PLUGIN_DIR, 'control', 'target.json');
+// The host takeover lane persists its paused/stopped series here. The adapter
+// watches it so the user's Esc stop also drops the physical-input latch.
+const TAKEOVER_STATE_FILE = join(PLUGIN_DIR, 'control', 'state.json');
+// The host publishes the current turn's activity lease here. The adapter reads it
+// to recompute the same series key the takeover lane blocks a stopped series on.
+const ACTIVITY_FILE = join(PLUGIN_DIR, 'control', 'activity.json');
 
 const VERSION = '0.1.0';
 const MAX_BATCH = 20;
 
+// The engine's own batch runs entries in one process, through the same policy
+// checks and dispatch as the CLI. Entry keys are the command's own field names: a
+// positional is named after itself, a flag drops `--` and writes `-` as `_`
+// (skills/agent-desktop/references/commands-system.md:170). Unknown keys are
+// rejected, so each action declares the keys it forwards.
+const NATIVE_BATCH_COMMANDS = new Map([
+  ['snapshot', 'snapshot'], ['find', 'find'], ['get', 'get'], ['is', 'is'],
+  ['click', 'click'], ['set_value', 'set-value'], ['press', 'press'],
+  ['scroll', 'scroll'], ['wait', 'wait'], ['focus', 'focus'],
+  ['double_click', 'double-click'], ['right_click', 'right-click'],
+  ['expand', 'expand'], ['collapse', 'collapse'], ['toggle', 'toggle'],
+]);
+// The engine's --headed is global and applies to every batch entry, so a step
+// that needs physical input cannot share one native batch with semantic steps:
+// those run through the adapter one at a time, each with its own scoped flag.
+const LOOP_BATCH_ACTIONS = new Set(['type', 'screenshot', 'ocr', 'focus', 'double_click', 'right_click']);
+const BATCH_ARG_KEYS = {
+  snapshot: ['app', 'window_id', 'compact', 'skeleton', 'interactive_only', 'max_depth', 'root', 'timeout_ms'],
+  find: ['app', 'window_id', 'name', 'role', 'text', 'value', 'native_id', 'description', 'first', 'last', 'nth', 'limit', 'count', 'exact', 'root', 'timeout_ms'],
+  get: ['ref_id', 'property'],
+  is: ['ref_id', 'property'],
+  click: ['ref_id', 'timeout_ms'],
+  focus: ['ref_id', 'timeout_ms'],
+  double_click: ['ref_id', 'timeout_ms'],
+  right_click: ['ref_id', 'timeout_ms'],
+  expand: ['ref_id', 'timeout_ms'],
+  collapse: ['ref_id', 'timeout_ms'],
+  toggle: ['ref_id', 'timeout_ms'],
+  set_value: ['ref_id', 'value', 'timeout_ms'],
+  press: ['combo', 'app', 'window_id', 'force'],
+  scroll: ['ref_id', 'direction', 'amount', 'timeout_ms'],
+  wait: ['ms', 'app', 'window', 'element', 'text', 'timeout'],
+};
+
+function batchEntryArgs(action, args) {
+  const entry = {};
+  for (const key of BATCH_ARG_KEYS[action] ?? []) {
+    const value = action === 'wait' && key === 'timeout' ? (args.timeout ?? args.timeout_ms) : args[key];
+    if (value === undefined || value === null) continue;
+    entry[key] = value;
+  }
+  return entry;
+}
+
+// A batch entry keeps its payload — a find step must yield refs — but a snapshot
+// tree is the largest thing the engine can return, and the batch cap allows 4 MB.
+function trimBatchData(data) {
+  if (!data || !Array.isArray(data.results)) return data;
+  return {
+    ...data,
+    results: data.results.map((entry) => (
+      entry?.command === 'snapshot' && entry.data?.tree
+        ? { ...entry, data: { ...entry.data, tree: undefined, tree_omitted: true } }
+        : entry
+    )),
+  };
+}
+
 const state = {
   sessionId: `arya-${randomBytes(4).toString('hex')}`,
   sessionStarted: false,
   headedOverride: null, // tools {operation:'headed'} per-conversation toggle
+  headedExpiresAt: 0, // headedOverride self-releases: a latch left on is a mouse/keyboard takeover
   overlayOverride: null,
   headedDefault: false,
   overlayDefault: true,
@@ -92,13 +157,26 @@ function notifyToolsChanged() {
   // swaps its registered tool generation.
   send({ jsonrpc: '2.0', method: 'notifications/tools/list_changed' });
 }
-const effectiveHeaded = () => state.headedOverride ?? state.headedDefault;
+// A headed latch left on makes EVERY later call physical (runCli appends
+// --headed to all of them), so it releases on its own. Per-call --headed is the
+// normal path; the latch is only an escape hatch for surfaces that need a
+// sequence of physical actions.
+const HEADED_LEASE_MS = 5 * 60 * 1000;
+const effectiveHeaded = () => {
+  if (state.headedOverride === true && Date.now() > state.headedExpiresAt) {
+    state.headedOverride = null;
+    state.headedExpiresAt = 0;
+  }
+  return state.headedOverride ?? state.headedDefault;
+};
 const effectiveOverlay = () => state.overlayOverride ?? state.overlayDefault;
 
 setInterval(() => {
   try {
     const settings = JSON.parse(readFileSync(SETTINGS_FILE, 'utf8'));
-    state.headedDefault = settings.headedDefault !== false;
+    // Strict: a settings file that omits the key must mean OFF. `!== false`
+    // read a missing key as ON, which is a silent mouse/keyboard takeover.
+    state.headedDefault = settings.headedDefault === true;
     state.overlayDefault = settings.overlayDefault !== false;
     const preset = settings.toolPreset === 'lazy' ? 'lazy' : 'lite';
     if (preset !== state.toolPreset) {
@@ -109,8 +187,59 @@ setInterval(() => {
   } catch {
     // Settings file not written yet (plugin not loaded): keep the defaults.
   }
+  try {
+    // Esc / stop lives in the host lane. A stopped series means the user pushed
+    // back: release the headed latch so the next series starts headless.
+    const takeover = JSON.parse(readFileSync(TAKEOVER_STATE_FILE, 'utf8'));
+    if (takeover?.paused === true && state.headedOverride === true) {
+      state.headedOverride = null;
+      state.headedExpiresAt = 0;
+    }
+  } catch {
+    // No takeover state yet.
+  }
 }, 1000).unref();
 
+/* ---------------------------------------------------------- user stop */
+
+// Esc lives in the host takeover lane: it blocks the series it was acting for,
+// keyed by [sessionId, turn] from the activity lease (mcp/takeover.mjs:111,184).
+// The adapter is a separate process, so it recomputes that key and refuses every
+// input action for a blocked series. The banner says "Esc Stop"; until now the
+// driver never honoured it. Reads stay available — the user asked to stop being
+// driven, not to be observed. A new user turn is a new key, so the block releases
+// exactly as it does in the host lane, with no settings change.
+const STOPPED_BY_USER = 'COMPUTER_USE_STOPPED_BY_USER: The user stopped this operation series with Esc or their custom shortcut. Do not retry desktop input in this turn. A new user turn, including a request to continue, can use the desktop automatically; no settings change is required.';
+const INPUT_TOOLS = new Set(['click', 'set_value', 'type', 'press', 'scroll', 'launch', 'focus', 'double_click', 'right_click', 'expand', 'collapse', 'toggle', 'batch']);
+
+function stoppedSeriesMessage() {
+  try {
+    const activity = JSON.parse(readFileSync(ACTIVITY_FILE, 'utf8'));
+    if (!(activity.active === true && Number.isFinite(activity.expiresAt) && activity.expiresAt > Date.now())) return null;
+    const saved = JSON.parse(readFileSync(TAKEOVER_STATE_FILE, 'utf8'));
+    const key = JSON.stringify([activity.sessionId, activity.turn]);
+    if (saved?.blockedSeries && Object.prototype.hasOwnProperty.call(saved.blockedSeries, key)) return saved.blockedSeries[key];
+    return saved?.paused === true && saved?.stoppedSeriesKey === key ? (saved.pauseMessage || STOPPED_BY_USER) : null;
+  } catch {
+    return null; // no lease published yet: nothing has been stopped
+  }
+}
+
+function stoppedResult(name, message) {
+  return envelopeResult({
+    version: '2.6',
+    ok: false,
+    command: name,
+    error: {
+      code: 'POLICY_DENIED',
+      message,
+      suggestion: 'Do not retry desktop input in this turn. Report that the user stopped the takeover; a new user turn can drive the desktop again.',
+      details: { kind: 'stopped_by_user' },
+      disposition: { delivery: 'not_delivered', retry: 'safe' },
+    },
+  });
+}
+
 /* ------------------------------------------------------- overlay target */
 
 // The overlay frames the app window that is being driven and puts the ghost
@@ -181,7 +310,10 @@ async function noteTarget(name, args, result) {
   if (name === 'list_windows') rememberWindows(data);
   if (name === 'snapshot') rememberWindows(data);
   if (args?.window_id) state.windowId = args.window_id;
-  if (!state.windowId) return;
+  // target.json feeds the rim and the ghost cursor only. With the overlay off,
+  // skip the CLI calls they cost (element bounds, sometimes a window list):
+  // ~85-140 ms per action spent on presentation nobody is looking at.
+  if (!effectiveOverlay() || !state.windowId) return;
   const window = await windowRectFor(state.windowId);
   const ref = args?.ref_id ?? data?.match?.ref_id ?? null;
   let pointer = ref ? await elementPoint(ref) : null;
@@ -323,8 +455,14 @@ const TOOLS = [
   tool('get', 'Read one control: property text|value|title|bounds|role|states.', { ref_id: str, property: str }),
   tool('is', 'Check one control: visible|enabled|checked|focused|expanded|selected.', { ref_id: str, property: str }),
   tool('click', 'Press a control via its semantic action (no pixels). Verify by re-snapshot.', { ref_id: str, timeout_ms: int }),
-  tool('set_value', 'Set a control value via UIA SetValue (no keystrokes). Some edits reject this — use type.', { ref_id: str, value: str, timeout_ms: int }),
-  tool('type', 'Type keystrokes into a text target. Background accessibility input by default; with headed on it briefly focuses the window and uses real keystrokes.', { ref_id: str, text: str, interval_ms: int, timeout_ms: int }),
+  tool('focus', 'Focus a control through UIA SetFocus. On Windows SetFocus takes the desktop foreground window, so the engine refuses it headless: the adapter runs it as physical input for this one call (same rule as type) without latching the conversation. For text entry prefer set_value, which focuses the field itself.', { ref_id: str, timeout_ms: int }),
+  tool('double_click', 'Double-click a control. Windows has no headless double-click, so this drives the real pointer for this one call (same rule as type). Verify by re-snapshot.', { ref_id: str, timeout_ms: int }),
+  tool('right_click', 'Open a control\u2019s context menu. Windows has no headless right-click, so this drives the real pointer for this one call (same rule as type). The menu is a separate surface: snapshot it after.', { ref_id: str, timeout_ms: int }),
+  tool('expand', 'Expand an expander, tree item, or dropdown via ExpandCollapsePattern (no pixels).', { ref_id: str, timeout_ms: int }),
+  tool('collapse', 'Collapse an expander or tree item via ExpandCollapsePattern (no pixels).', { ref_id: str, timeout_ms: int }),
+  tool('toggle', 'Flip a checkbox or toggle button via TogglePattern. Prefer it over click: it reports the resulting state, and click on a checked control can be refused.', { ref_id: str, timeout_ms: int }),
+  tool('set_value', 'Set a control value via UIA SetValue (no keystrokes, no pointer movement). This is the headless text path — use it first for any text entry. Some edits reject it — only then use type.', { ref_id: str, value: str, timeout_ms: int }),
+  tool('type', 'Type real keystrokes into a text target. On Windows keystroke synthesis is physical: this moves the pointer and briefly focuses the window. Prefer set_value (headless UIA write) for text entry.', { ref_id: str, text: str, interval_ms: int, timeout_ms: int }),
   tool('press', 'Send a key chord (e.g. ctrl+s) to an app/window. Foreground queue: for non-frontmost apps expect delivered_unverified.', { combo: str, app: str, window_id: str, force: bool }),
   tool('scroll', 'Scroll a scrollable control: direction up|down|left|right, amount in lines.', { ref_id: str, direction: str, amount: int, timeout_ms: int }),
   tool('list_windows', 'List top-level windows (ids, titles, pids, responsiveness).', { app: str, timeout_ms: int }),
@@ -332,7 +470,7 @@ const TOOLS = [
   tool('wait', 'Pause (ms) or wait for a condition: window/app/text/element before acting. Prevents acting on half-settled windows.', { ms: int, app: str, window: str, element: str, text: str, timeout_ms: int }),
   tool('screenshot', 'Capture a window/screen PNG to a file; returns the file path. Use for visual evidence only — trees are the primary signal.', { app: str, window_id: str, screen: int }),
   tool('ocr', 'Read text from a screenshot via Windows OCR (for canvas/games/no-a11y-tree UIs). Returns lines with SCREEN coordinates (not refs). query filters lines.', { app: str, window_id: str, screen: int, query: str }),
-  tool('batch', `Run 1-${MAX_BATCH} planned actions on one target; stops on first error, then observes once. Each step: {action, arguments}. Actions: snapshot,find,get,is,click,set_value,type,press,scroll,wait.`, { commands_json: str, stop_on_error: bool, timeout_ms: int }, ['commands_json']),
+  tool('batch', `Run 1-${MAX_BATCH} planned actions on one target; stops on first error, then observes once. Each step: {action, arguments}. Native single-process actions (fast path): snapshot,find,get,is,click,set_value,press,scroll,wait,focus,double_click,right_click,expand,collapse,toggle. type,screenshot,ocr run step by step.`, { commands_json: str, stop_on_error: bool, timeout_ms: int }, ['commands_json']),
   tool('tools',
     'Manage the adapter: operation list|describe (tool surface + state), load {names} (reveal tools when the lazy preset is on), headed {on} (per-conversation headed toggle — use when headless is not an option: cursor-moving input, notifications, shell surfaces), overlay {on} (presentation cursor overlay), session_end {operation} (end this conversation\'s driver session).',
     { operation: str, name: str, names: { type: 'array', items: str }, on: bool }),
@@ -360,6 +498,10 @@ function flagArgs(pairs) {
 }
 
 async function callTool(name, args = {}) {
+  if (INPUT_TOOLS.has(name)) {
+    const stopped = stoppedSeriesMessage();
+    if (stopped) return stoppedResult(name, stopped);
+  }
   switch (name) {
     case 'snapshot':
       return enqueue(async () => {
@@ -391,18 +533,38 @@ async function callTool(name, args = {}) {
         await ensureSession();
         return parseEnvelope(await runCli(['click', args.ref_id, ...flagArgs([['--timeout-ms', args.timeout_ms]])]));
       });
+    case 'expand':
+    case 'collapse':
+    case 'toggle':
+      return enqueue(async () => {
+        await ensureSession();
+        return parseEnvelope(await runCli([NATIVE_BATCH_COMMANDS.get(name), args.ref_id, ...flagArgs([['--timeout-ms', args.timeout_ms]])]));
+      });
+    case 'focus':
+    case 'double_click':
+    case 'right_click':
+      return enqueue(async () => {
+        await ensureSession();
+        // Engine fact on Windows (platform skill): right-click, double-click and
+        // focus are physical or take the desktop foreground, so they return
+        // POLICY_DENIED headless. --headed is scoped to THIS call, exactly like
+        // `type`; the conversation is never latched into physical input.
+        return parseEnvelope(await runCli([NATIVE_BATCH_COMMANDS.get(name), args.ref_id, ...flagArgs([['--timeout-ms', args.timeout_ms]])], { headed: true }));
+      });
     case 'set_value':
       return enqueue(async () => parseEnvelope(await runCli(['set-value', args.ref_id, String(args.value ?? ''), ...flagArgs([['--timeout-ms', args.timeout_ms]])])));
     case 'type':
       return enqueue(async () => {
         await ensureSession();
-        const headed = effectiveHeaded();
-        if (headed) {
-          const promote = await ensureForeground(args.ref_id);
-          if (promote) return envelopeResult(promote);
-        }
-        const env = parseEnvelope(await runCli(['type', args.ref_id, args.text ?? '', ...flagArgs([['--interval-ms', args.interval_ms], ['--timeout-ms', args.timeout_ms]])], { headed }));
-        if (!headed && env && env.ok !== true) env.hint = 'Background delivery cannot synthesize keystrokes for this target. Prefer set_value, or enable real input for this conversation: tools {operation:"headed",on:true} (moves the pointer), then type again.';
+        // Engine fact on Windows: headless `type` is POLICY_DENIED — keystroke
+        // synthesis is headed-only. `type` is therefore physical by nature, so
+        // --headed is scoped to THIS call and the conversation is never latched
+        // into it. Latching was what made every later click/scroll move the
+        // user's real pointer.
+        const promote = await ensureForeground(args.ref_id);
+        if (promote) return envelopeResult(promote);
+        const env = parseEnvelope(await runCli(['type', args.ref_id, args.text ?? '', ...flagArgs([['--interval-ms', args.interval_ms], ['--timeout-ms', args.timeout_ms]])], { headed: true }));
+        if (env && env.ok !== true) env.hint = 'Keystroke synthesis is physical: it moves the pointer and focuses the window. Prefer set_value, which writes through the UIA value pattern headless.';
         return env;
       });
     case 'press':
@@ -437,13 +599,35 @@ async function callTool(name, args = {}) {
       if (!Array.isArray(commands) || commands.length < 1 || commands.length > MAX_BATCH) {
         return { content: [{ type: 'text', text: `batch needs 1-${MAX_BATCH} steps (got ${Array.isArray(commands) ? commands.length : 'non-array'}).` }], isError: true };
       }
+      const actions = commands.map((step) => String(step?.action || ''));
+      const unknown = actions.filter((action) => !NATIVE_BATCH_COMMANDS.has(action) && !LOOP_BATCH_ACTIONS.has(action));
+      if (unknown.length > 0) {
+        return { content: [{ type: 'text', text: JSON.stringify({ ok: false, unknown, supported: [...NATIVE_BATCH_COMMANDS.keys(), ...LOOP_BATCH_ACTIONS] }) }], isError: true };
+      }
+      if (!actions.some((action) => LOOP_BATCH_ACTIONS.has(action))) {
+        // One engine process for the whole batch. Every CLI call costs ~85 ms of
+        // spawn plus UIA cold start, and the engine runs the same policy checks and
+        // dispatch per entry, so this is ~1.7x faster than the loop and keeps each
+        // entry's payload — a find step yields refs instead of "ok".
+        const entries = commands.map((step) => {
+          const action = String(step?.action || '');
+          return { command: NATIVE_BATCH_COMMANDS.get(action), args: batchEntryArgs(action, step?.arguments || {}) };
+        });
+        return enqueue(async () => {
+          await ensureSession();
+          const cli = ['batch', JSON.stringify(entries)];
+          if (args.stop_on_error !== false) cli.push('--stop-on-error');
+          if (args.timeout_ms !== undefined && args.timeout_ms !== null) cli.push('--timeout-ms', String(args.timeout_ms));
+          const envelope = parseEnvelope(await runCli(cli));
+          if (envelope.ok === true) envelope.data = trimBatchData(envelope.data);
+          return envelopeResult(envelope);
+        });
+      }
+      // type promotes the window to the foreground first, and screenshot/ocr own an
+      // adapter-side file path, so a batch containing them runs step by step.
       const results = [];
       for (const step of commands) {
         const action = String(step?.action || '');
-        if (!TOOLS_BY_NAME.has(action) || action === 'batch' || action === 'tools') {
-          results.push({ action, error: `unsupported step action: ${action}` });
-          break;
-        }
         const out = await callTool(action, step?.arguments || {});
         const envelope = out.structuredContent ?? out;
         results.push({ action, ok: envelope.ok === true, summary: summarizeEnvelope(envelope) });
@@ -460,7 +644,8 @@ async function callTool(name, args = {}) {
         }
         if (op === 'headed') {
           state.headedOverride = args.on === true;
-          return { content: [{ type: 'text', text: JSON.stringify({ ok: true, headed: effectiveHeaded(), headedDefault: state.headedDefault }) }] };
+          state.headedExpiresAt = args.on === true ? Date.now() + HEADED_LEASE_MS : 0;
+          return { content: [{ type: 'text', text: JSON.stringify({ ok: true, headed: effectiveHeaded(), headedDefault: state.headedDefault, lease_ms: args.on === true ? HEADED_LEASE_MS : 0, note: 'Headed drives the real mouse and keyboard for every action in this conversation. It releases on session_end and expires on its own. Prefer set_value for text so headed is not needed at all.' }) }] };
         }
         if (op === 'overlay') {
           state.overlayOverride = args.on === true;
@@ -506,6 +691,9 @@ async function callTool(name, args = {}) {
           await runCli(['session', 'gc']);
           state.sessionStarted = false;
           state.sessionId = `arya-${randomBytes(4).toString('hex')}`;
+          // Ending the driver session also releases the physical-input latch.
+          state.headedOverride = null;
+          state.headedExpiresAt = 0;
           return { content: [{ type: 'text', text: JSON.stringify({ ok: env.ok === true, note: 'new session starts on next action', engine: env }) }] };
         }
         return {
@@ -531,6 +719,13 @@ function summarizeEnvelope(envelope) {
     const d = envelope.data ?? {};
     const bits = [];
     if (d.ref_count !== undefined) bits.push(`refs ${d.ref_count}`);
+    // A step that locates something is useless without the ref it found, and a
+    // read step without its value is the same. Report the payload, not just "ok".
+    if (d.match?.ref_id) bits.push(`ref ${d.match.ref_id}${d.match.name ? ` (${d.match.name})` : ''}`);
+    if (Array.isArray(d.matches) && d.matches.length) bits.push(`refs ${d.matches.slice(0, 8).map((m) => m.ref_id).join(',')}`);
+    if (d.snapshot_id) bits.push(`snapshot ${d.snapshot_id}`);
+    if (d.value !== undefined) bits.push(`value ${JSON.stringify(d.value)}`);
+    if (d.result !== undefined) bits.push(`result ${JSON.stringify(d.result)}`);
     if (d.window?.id) bits.push(`window ${d.window.id}`);
     if (d.target?.process?.pid) bits.push(`pid ${d.target.process.pid}`);
     if (d.disposition?.delivery) bits.push(`delivery ${d.disposition.delivery}`);
@@ -583,11 +778,13 @@ const INSTRUCTIONS = [
   'Arya Computer Use: drive Windows apps through semantic UIA refs, not pixels.',
   'Loop: snapshot (skeleton/compact first) -> act on @refs -> re-snapshot to verify.',
   'Refs go stale after window changes: on STALE_REF/AMBIGUOUS_TARGET, re-snapshot.',
-  'type uses background accessibility input by default and never moves the pointer; tools {operation:"headed",on:true} switches this conversation to real keyboard/mouse input when a surface needs it.',
+  'set_value is the headless text path (UIA value pattern, pointer stays put). type synthesizes real keystrokes: it moves the pointer and focuses the window, scoped to that one call.',
+  'Headed mode (tools {operation:"headed",on:true}) makes EVERY action in this conversation physical input. Use it only when background input is refused; it releases on session_end and expires after 5 minutes.',
   'Chromium windows settle slowly: snapshot with timeout_ms 10000-25000.',
   'Verify stateful actions by re-read; delivered_verified = confirmed, delivered_unverified retry:unsafe = do not blind-repeat.',
   'On ACTION_FAILED post_action_verification the action landed: read data.post_state, do NOT retry.',
   'This host must run elevated (Administrator); lease_directory_untrusted means restart AryaCode as Administrator.',
+  'Esc (or the stop shortcut) stops the series: input tools then return COMPUTER_USE_STOPPED_BY_USER for this turn. Reads still work; do not retry input until a new user turn.',
   'Destructive actions (delete, send, pay, close unsaved) require user confirmation first.',
   'No accessibility tree (games/canvas): use ocr for text and headed mouse coordinates for input.',
 ].join('\n');

package/mcp/settings.mjs

@@ -3,7 +3,10 @@
 // switches for the adapter (headedDefault, overlayDefault, driverLockEnabled).
 
 export const DEFAULT_SETTINGS = Object.freeze({
-  progressOverlayEnabled: true, progressOverlayMode: 'source',
+  // 'source' paints the assistant's raw text stream on the user's desktop. The
+  // default is the condensed lane; 'source' stays available for anyone who wants
+  // the verbatim panel.
+  progressOverlayEnabled: true, progressOverlayMode: 'summary',
   progressOverlayOpacity: 60, progressOverlayWidth: 360, progressOverlayHeight: 240, progressOverlayPosition: 'top-right', progressOverlayX: 80, progressOverlayY: 80, progressOverlayLocked: true, progressLockShortcut: '', progressPreviewToken: 0,
   indicatorEnabled: true, cursorIndicatorEnabled: true, remoteIndicatorVisible: true, indicatorText: 'Arya is controlling your computer', pointerLabel: 'Arya',
   indicatorPosition: 'top', indicatorIdleMs: 800, stopShortcut: '',

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aryacode/dsh-plugin-arya-computer-use",
-  "version": "1.0.4",
+  "version": "1.0.6",
   "description": "Computer use for DSH/AryaCode: agent-desktop UIA engine, agent-browser web lane, takeover UI with a ghost cursor",
   "license": "MIT",
   "type": "module",
@@ -58,6 +58,9 @@
     "bundle": {
       "patch": "./cordis.patch.yml"
     },
+    "engines": {
+      "dsh": ">=0.2.1-rc.3 <0.3.0-0"
+    },
     "client": {
       "platform": "web",
       "inject": [

package/skills/arya-desktop/SKILL.md

@@ -18,11 +18,11 @@ Screenshots are evidence, not input.
    thin and grow: pass `timeout_ms: 10000`–`25000` and re-snapshot after the
    first contact.
 2. **Act** — one action on one ref: `click`, `set_value`, `type`, `press`,
-   `scroll`. Prefer `set_value` for edits: it writes through the accessibility
-   value pattern and never touches the pointer. `type` needs real keystrokes, so
-   in the default background mode it only works on targets that accept
-   synthetic text; when it fails, either use `set_value` or enable headed input
-   for this conversation with `tools {operation:"headed",on:true}`.
+   `scroll`. **Use `set_value` for every text entry**: it writes through the
+   accessibility value pattern, headless, and never touches the pointer. `type`
+   synthesizes real keystrokes — on Windows that is physical input, so it moves
+   the pointer and briefly focuses the window. It is the last resort, only for
+   an edit that rejects `set_value`.
 3. **Verify** — re-`snapshot` (or `get`/`is` on the ref). The envelope tells
    you what happened:
    - `delivered_verified` — confirmed by readback.
@@ -43,11 +43,14 @@ Screenshots are evidence, not input.
   `lease_directory_untrusted`, tell the user: restart AryaCode as Administrator.
 - **Headed toggle.** Default is headless (no mouse/keyboard takeover): semantic
   UIA input goes straight to the control and never moves the user's pointer.
-  When headless cannot do the job (cursor-moving input, notifications, start
-  menu, tray, surfaces that reject background input), call
-  `tools {operation:"headed",on:true}` for this conversation, then `on:false`
-  when done. Headed input briefly focuses the target window and drives the real
-  keyboard/mouse, so warn the user before switching it on.
+  Headed is an **escape hatch for surfaces that refuse background input**
+  (notifications, start menu, tray, right-click, drag, double-click). It is not
+  the answer to a text field — `set_value` does that headless. Turning it on
+  makes **every** action in this conversation physical: clicks move the real
+  pointer, keystrokes go to the foreground window. So: warn the user before
+  switching it on, call `tools {operation:"headed",on:false}` when the surface
+  is done, and end with `tools {operation:"session_end"}`, which releases it.
+  A latch left on also expires by itself after 5 minutes.
 - **Launch.** Bare names resolve only under System32 (`notepad` works,
   `chrome` does not). Use the absolute exe path. If many instances run, the
   engine refuses to guess — that is correct; ask the user.