dsh-codex-style-computer-use 1.3.0 → 1.3.1

package/README.md

@@ -1,26 +1,35 @@
-# dsh-codex-style-computer-use · Codex-style desktop control
+# Codex-style Computer Use
 
-**Give DeepSeek eyes and hands for Windows.**
+**Bring a Codex-style desktop workflow to DeepSeek Harness.**
 
-Ask your agent to work with desktop apps: see the screen, find controls, click and type. Take over with your own mouse or keyboard whenever needed; the agent waits before continuing. Approval prompts remain enabled by default.
+Describe the goal. Your agent observes the screen, locates controls, clicks and types, then checks the result. Work with native Windows apps from the conversation.
 
-Windows desktop computer use for **DeepSeek Harness**: 18 desktop tools plus an Agent activation entry that observe and drive
-native Windows applications through prebuilt C# workers included in the package. No PowerShell 7,
-SDK or install-time builds are needed.
+English | [中文](README.zh.md) · [Install](#install-and-update-in-dsh-market) · [Changelog](CHANGELOG.md)
 
-English | [中文](README.zh.md)
+![Codex-style Windows desktop control: observe, act, verify](docs/images/01-codex-style.png)
 
-Windows only (`os: win32`). Requires Node.js ≥ 22 and the .NET Framework 4.x that Windows ships with.
+- **See the target. Take the action.** Combine screenshots, UI Automation and numbered targets with mouse, keyboard, drag, scroll and window tools.
+- **Take over. Let the agent wait. Hand back and continue.** Version 1.3.0 adds a 1.5-second notice before input. Use your computer normally outside actual actuation; after handback, the agent re-observes and resumes the original task.
+- **Fewer round trips. More efficient token use.** Incremental control observations, cropped or downsampled screenshots and batches of known actions reduce repeated context and model calls.
 
-**1.3.0 desktop lifecycle update:** the deep-blue cue now matches actual actuation, with a 16-pixel pointer jitter allowance and a visible 1.5-second purple notice before input. Human takeover returns a required continuation checkpoint. Session/agent ownership and a native cross-process lease allow only one controller. Call `computer_ctrl {action:"release"}` after desktop work and verification; the conversation continues and later desktop work can open a fresh cycle in the same turn. Normal completion releases automatically without an exit question card. Explicit pauses, exits and pending questions remain authoritative.
+![Human handoff: prepare, take over, continue](docs/images/02-handoff.png)
 
-The indicators use continuous 2-pixel rails with square screen corners and an 8-pixel glow. Normal working states have no floating badges; handoff, pause and preparation use compact top hints. Content interiors remain transparent. The visual revision preserves state logic and animation timing.
+![Token efficiency: focused observations, batched actions and result readback](docs/images/03-token-efficiency.png)
 
-**1.2.3 automatic fix:** normal installation/update now installs the approval-attribution correction for computer-use calls on affected DSH hosts. No manual host patch is required for those calls. Host approvals and audit remain authoritative; follow the host's restart indication after updating an already loaded plugin. See [1.2.3](docs/RELEASE-1.2.3.md).
+These are **1.3.0 feature illustrations**. Actual token use depends on the task, model and observation strategy. Approvals remain enabled by default; explicit pause, cancellation and exit remain authoritative.
 
-**1.2.1 attention fix:** fixes a missing crypto import that prevented 1.2.0's native `probe` and `raise` requests from reaching the helper. Updating from 1.2.0 is required for this runtime fix; reinstalling 1.2.0 does not fix it. New releases are staged on npm's `next` tag before promotion to `latest`; the market's normal update follows `latest`. See [1.2.1 validation and release status](docs/RELEASE-1.2.1.md).
+**Ready to install.** 18 desktop tools, with native helpers included. No SDK or install-time compilation. Requires Windows, Node.js ≥ 22 and the .NET Framework 4.x included with Windows. An independent community plugin; Codex describes the interaction style.
 
-**1.2.2 GitHub pre-release:** includes a separately deployed DSH host approval-attribution patch and 14 regression cases. Plugin input and approval behavior is retained from 1.2.1. See [release contents and deployment scope](docs/RELEASE-1.2.2.md); GitHub publication does not update npm or the market automatically.
+<details>
+<summary>Current release and compatibility</summary>
+
+**1.3.1** is the current release on npm's `latest` channel: `dsh-codex-style-computer-use@1.3.1`. It adds worker-side OCR with per-word coordinates, change/caret-aware waits, UAC and lock-screen awareness with automatic continuation, and IME-safe clipboard text. Follow the host's restart indication when updating a loaded version. See [1.3.1](docs/RELEASE-1.3.1.md) and [GitHub Releases](https://github.com/ltmroberthk915/dsh-computer-use/releases).
+
+Version 1.3.1 reads text from the screen by name, notices the Windows secure desktop and resumes on its own, and keeps every colour on the same DPI-derived edge rail with a visible breathing floor. Version 1.3.0 releases desktop control when work ends while the chat continues.
+
+Earlier fixes: [1.2.3 approval attribution](docs/RELEASE-1.2.3.md) · [1.2.1 native attention](docs/RELEASE-1.2.1.md) · [standalone host patch](docs/RELEASE-1.2.2.md).
+
+</details>
 
 ## What it does
 
@@ -41,9 +50,9 @@ under `$DSH_HOME/data/computer-use/`.
 
 ## Install and update in dsh-market
 
-Once the catalog submission is merged and synced, search **computer-use** and select **dsh-codex-style-computer-use**, by **ltmroberthk915** (npm maintainer: **ltmroberthk**). Click **Install**. Later, use **Update** or **Update all** for versions admitted by the host's release policy. The package includes both native helpers; no PowerShell 7 or build permission is needed.
+Search **computer-use** or **Codex-style** in the market. Look for repository **ltmroberthk915/dsh-computer-use** and npm package **dsh-codex-style-computer-use** (maintainer: **ltmroberthk**), then click **Install**. Use **Update** or **Update all** for stable-channel updates. Both native helpers are included; no PowerShell 7 or build permission is needed.
 
-Before catalog sync, use **Settings → Plugins → Add plugin**. The corrected version is **`dsh-codex-style-computer-use@1.2.1`**; the unversioned name still resolves to 1.2.0 until promotion. Do not paste the GitHub repository URL: that selects the source-download path instead of the small prebuilt npm package. The default non-strict pnpm 11.7.0 configuration can install 1.2.1 immediately; an explicitly strict host policy can require waiting as described below. A newly added bundle can load live on the official Desktop host; check that its tools and skill appear. Restart from the tray when replacing an already loaded version, when the client remains stale, or when DSH reports **restart-required**.
+Alternatively, enter **`dsh-codex-style-computer-use`** in **Settings → Plugins → Add plugin** for the current stable channel, or **`dsh-codex-style-computer-use@1.3.0`** for the version illustrated here. Enter the npm package spec directly. Your host may apply a waiting period to fresh versions; follow its indication. A newly added bundle can load live on supported hosts; check that its tools and skill appear. Restart from the tray when replacing a loaded version, when the client remains stale, or when DSH reports **restart-required**.
 
 ### Releases less than 24 hours old
 

package/README.zh.md

@@ -1,27 +1,35 @@
-# dsh-codex-style-computer-use · Codex 风格桌面操控
+# Codex 风格 Computer Use
 
-**让 DeepSeek 看得见，也点得动。**
+**更接近 Codex 的电脑操控体验，带到 DeepSeek Harness。**
 
-把 Windows 应用里的重复操作交给 AI：看屏幕、找控件、点鼠标、敲键盘。AI 实际操控前提示 1.5 秒，非操控期间你可以正常使用电脑；操控中接管后，AI 等待交回并接续原任务。默认保留操作审批。
+你描述目标，AI 看屏幕、找控件、点鼠标、敲键盘，再检查结果。让原生 Windows 应用里的工作，也能在对话中完成。
 
-给 **DeepSeek Harness** 用的 Windows 桌面操控插件：18 个桌面工具加一个 Agent 激活入口，通过一个 C# worker
-观察并驱动原生 Windows 应用。发布包已包含编译好的原生组件，无需安装 PowerShell 7、SDK 或执行编译命令。
+[English](README.md) | 中文 · [安装](#在-dsh-market-安装和更新) · [版本记录](CHANGELOG.md)
 
-[English](README.md) | 中文
+![Codex 风格 Windows 桌面操控：观察、操作、验证](docs/images/01-codex-style.png)
 
-仅支持 Windows（`os: win32`）。需要 Node.js ≥ 22 与 Windows 自带的 .NET Framework 4.x。
+- **看得准，也点得动。** 截图、控件树与编号定位配合使用，支持点击、输入、拖拽、滚动和窗口操作。
+- **你接手，AI 等待，再接着做。** 1.3.0 提供 1.5 秒操作预告；非实际操控期间正常使用电脑，交回后重新观察并接续原任务。
+- **更少往返，更省 Token。** 控件增量观察、区域与降采样截图，加上连续动作批量执行，减少重复观察和模型往返。
 
-**1.3.0 桌面控制周期升级：**实际操控与深蓝色指示同步；鼠标轻微抖动有 16 像素容差；紫色操作前提示持续 1.5 秒；接管后返回必须继续任务的断点；同一时间只有一个 agent/worker 拥有控制周期。桌面工作和验证结束后调用 `computer_ctrl {action:"release"}` 即可释放，聊天继续，同一轮后来仍可重新开始。正常收尾自动释放，不再弹出结束确认选项单。人工暂停、明确退出和待回答的问题保持有效。
+![人工接管与续接：先提醒、可接管、再继续](docs/images/02-handoff.png)
 
-灯效统一为贴合屏幕直角的 2 像素光线和最多 8 像素柔光，四角没有圆弧缺口。普通工作状态不显示悬浮文字牌；让出、暂停和预告使用紧凑提示，正文区域保持透明。此次美工调整不改变状态逻辑或闪烁节奏。
+![Token 效率：聚焦观察、合并操作、回读验证](docs/images/03-token-efficiency.png)
 
-**1.2.3 自动修复：**正常安装或更新插件，即会为受影响宿主中的 computer-use 调用安装审批归因修正，无需手工修改宿主。审批决定与审计仍由宿主处理；更新已加载插件后按宿主提示重启。详见 [1.2.3](docs/RELEASE-1.2.3.md)。
+上图为 **1.3.0 功能示意**。实际 Token 用量取决于任务、模型和观察方式。默认保留操作审批；明确暂停、取消和退出始终有效。
 
-**1.2.1 提醒修复：**补齐 crypto 导入，修复 1.2.0 原生 `probe` / `raise` 请求还未发送就抛错的问题。这次是运行时代码修复，需要从 1.2.0 更新；重复安装 1.2.0 无效。新版本先发布到 npm 的 `next`，再提升为 `latest`，市场普通更新跟随 `latest`。详见 [1.2.1 验证和发布状态](docs/RELEASE-1.2.1.md)。
+**安装即用。** 18 个桌面工具，原生组件随包提供，无需 SDK 或安装时编译。支持 Windows、Node.js ≥ 22 和系统自带的 .NET Framework 4.x。独立社区插件；Codex 用于说明交互风格。
 
-宿主在禁用审批弹窗时仍提示“用户拒绝工具”的问题，详见 [DSH 审批归因补丁与回归测试](docs/APPROVAL-ATTRIBUTION.md)。
+<details>
+<summary>当前版本与兼容性</summary>
 
-**1.2.2 GitHub 预发布：**收录独立部署的宿主审批归因补丁和 14 项回归测试。插件输入与审批行为延续 1.2.1，详见 [发布内容及部署范围](docs/RELEASE-1.2.2.md)。GitHub 发布不会自动更新 npm 或市场版本。
+**1.3.1** 是 npm `latest` 频道的当前版本：`dsh-codex-style-computer-use@1.3.1`。新增 worker 侧 OCR（逐词坐标）、可定位变化与识别光标闪烁的等待、UAC/锁屏感知与自动续接、以及绕过 IME 的剪贴板文本通道。更新已加载的旧版本后，按宿主提示重启。详见 [1.3.1](docs/RELEASE-1.3.1.md) 与 [GitHub Releases](https://github.com/ltmroberthk915/dsh-computer-use/releases)。
+
+1.3.1 可直接从屏幕读字定位控件，察觉 Windows 安全桌面并在其结束后自动续接；所有颜色共用同一条按 DPI 换算的边缘光环，呼吸谷值保持可见。1.3.0 起工具用完可主动释放，聊天继续；正常收尾自动退出。
+
+历史兼容修复：[1.2.3 审批归因](docs/RELEASE-1.2.3.md) · [1.2.1 原生提醒](docs/RELEASE-1.2.1.md) · [独立宿主补丁](docs/RELEASE-1.2.2.md)。
+
+</details>
 
 ## 它能做什么
 
@@ -39,9 +47,9 @@
 
 ## 在 dsh-market 安装和更新
 
-市场收录同步后，搜索 **computer-use**，选择 **dsh-codex-style-computer-use**，作者 **ltmroberthk915**（npm 维护者 **ltmroberthk**），点击**安装**。以后通过市场的**更新**或**全部更新**安装符合宿主发布策略的版本。两份原生组件都已随包提供，无需 PowerShell 7、SDK 或构建白名单。
+在市场搜索 **computer-use** 或 **Codex-style**，认准仓库 **ltmroberthk915/dsh-computer-use**、npm 包 **dsh-codex-style-computer-use**（维护者 **ltmroberthk**），点击**安装**。以后通过市场的**更新**或**全部更新**获取正式频道版本。两份原生组件都已随包提供，无需 PowerShell 7、SDK 或构建白名单。
 
-目录尚未同步时，在 **设置 → 插件 → 添加插件** 输入修复版 **`dsh-codex-style-computer-use@1.2.1`**。正式标签提升前，省略版本号仍会装到 1.2.0。这里填 npm 包名即可；填写 GitHub 仓库网址会转为下载源码，容易受 GitHub 下载线路影响。默认非严格 pnpm 11.7.0 配置可以直接安装 1.2.1；宿主显式启用严格策略时，才可能需要等待冷却。首次新增 bundle 在官方桌面端可热加载；确认工具与 skill 已出现即可使用。替换已经加载的版本、客户端界面未刷新或界面明确提示 **restart-required / 需要重启** 时，再从托盘完整退出 DSH 并重新打开。
+也可在 **设置 → 插件 → 添加插件** 输入 **`dsh-codex-style-computer-use`**，安装当前正式频道；要体验本页展示的 1.3.0，输入 **`dsh-codex-style-computer-use@1.3.0`**。这里填 npm 包名即可。宿主可能对新版本设置等待期，以界面提示为准。首次新增 bundle 可在支持的宿主中热加载；确认工具与 skill 已出现即可使用。替换已经加载的版本、客户端界面未刷新或界面提示 **restart-required / 需要重启** 时，从托盘完整退出 DSH 并重新打开。
 
 ### 发布未满 24 小时
 

package/icon.svg

@@ -0,0 +1,29 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 32 32" fill="none">
+  <defs>
+    <linearGradient id="tile" x1="4" y1="0" x2="28" y2="32" gradientUnits="userSpaceOnUse">
+      <stop stop-color="#17243F"/>
+      <stop offset="1" stop-color="#05080F"/>
+    </linearGradient>
+    <radialGradient id="halo" cx="11" cy="27" r="20" gradientUnits="userSpaceOnUse">
+      <stop stop-color="#8B5CF6" stop-opacity=".2"/>
+      <stop offset="1" stop-color="#8B5CF6" stop-opacity="0"/>
+    </radialGradient>
+    <linearGradient id="rail" x1="5" y1="6" x2="27" y2="24" gradientUnits="userSpaceOnUse">
+      <stop stop-color="#5EEAD4"/>
+      <stop offset=".5" stop-color="#38BDF8"/>
+      <stop offset="1" stop-color="#A78BFA"/>
+    </linearGradient>
+    <radialGradient id="glow" cx="18.2" cy="19.4" r="7.6" gradientUnits="userSpaceOnUse">
+      <stop stop-color="#38BDF8" stop-opacity=".32"/>
+      <stop offset="1" stop-color="#38BDF8" stop-opacity="0"/>
+    </radialGradient>
+  </defs>
+  <rect width="32" height="32" rx="7.5" fill="url(#tile)"/>
+  <rect width="32" height="32" rx="7.5" fill="url(#halo)"/>
+  <rect x="5.6" y="5.9" width="20.8" height="16" rx="3" fill="#FFFFFF" fill-opacity=".06" stroke="url(#rail)" stroke-width="1.9"/>
+  <path d="M5.6 10.1 H26.4" stroke="url(#rail)" stroke-width="1.05" stroke-opacity=".5"/>
+  <circle cx="8.5" cy="8" r=".9" fill="#5EEAD4"/>
+  <circle cx="11.2" cy="8" r=".9" fill="#A78BFA" fill-opacity=".85"/>
+  <circle cx="18.2" cy="19.4" r="7.6" fill="url(#glow)"/>
+  <path d="M15.6 12.4 L15.6 21.2 L17.8 19 L19.3 22.4 L20.8 21.7 L19.3 18.5 L22.3 18.5 Z" fill="#F8FAFC" stroke="#070B14" stroke-width=".85" stroke-linejoin="round"/>
+</svg>

package/lib/bin/attention-worker.exe.json

@@ -3,5 +3,5 @@
   "platform": "win32",
   "architecture": "x64",
   "sourceSha256": "6e17308fc1cb0de1ece6f3164866b6de03ecdf74a42a6e78c0a8e6d02912c3bf",
-  "binarySha256": "c15feb47d7ec975ccab12fea2ee4300ea0a4699069be21d0fb9d1fedebe26158"
+  "binarySha256": "f39e79148d0f1c8fa527f55607eb4e7440e82c3ab5b33162bde773528418fe5a"
 }

package/lib/core/bin/dsh-computer-use-worker.exe.json

@@ -2,6 +2,6 @@
   "format": 1,
   "platform": "win32",
   "architecture": "x64",
-  "sourceSha256": "885c7935d5520bc6f865da92e3ab63b29c6515a853e2a3e39ceb2221606acf08",
-  "binarySha256": "c7ae5f0f77912674a8f6f310b95809eef10941cf986f8aa7691aed7a7f08bff5"
+  "sourceSha256": "8427a18e1f161959043f638cab821e701714f3d83138911dc9c4fb2cbeb152bf",
+  "binarySha256": "0e9212f628ebd73655de1235b3e7bb4ce176c4b60da16c336f2ebae7397084a2"
 }

package/lib/core/compile-worker.ps1

@@ -47,7 +47,11 @@ $refs = @(
     '/r:System.Web.Extensions.dll', '/r:System.Windows.Forms.dll',
     "/r:$gac\UIAutomationClient\v4.0_4.0.0.0__31bf3856ad364e35\UIAutomationClient.dll",
     "/r:$gac\UIAutomationTypes\v4.0_4.0.0.0__31bf3856ad364e35\UIAutomationTypes.dll",
-    "/r:$gac\WindowsBase\v4.0_4.0.0.0__31bf3856ad364e35\WindowsBase.dll"
+    "/r:$gac\WindowsBase\v4.0_4.0.0.0__31bf3856ad364e35\WindowsBase.dll",
+    # Inbox OCR (2026-11-04): union winmd ships in the package next to this script.
+    "/r:$env:windir\Microsoft.NET\Framework64\v4.0.30319\System.Runtime.dll",
+    "/r:$gac\System.Runtime.WindowsRuntime\v4.0_4.0.0.0__b77a5c561934e089\System.Runtime.WindowsRuntime.dll",
+    "/r:$PSScriptRoot\winmd\Windows.winmd"
 )
 
 New-Item -ItemType Directory -Path $outDir -Force | Out-Null

package/lib/core/index.js

@@ -22,6 +22,10 @@ const __dirname = fileURLToPath(new URL('.', import.meta.url))
 
 const DEFAULT_TIMEOUT_MS = 15_000
 const CAPTURE_TIMEOUT_MS = 30_000
+// A UIA tree walk can block the resident's synchronous COM thread. Bound the
+// caller's wait separately from recovery of a worker that never returns.
+const UIA_READS = new Set(['uia', 'uiaFromPoint', 'uiaFocused'])
+const URGENT_CONTROL = new Set(['cancelInput', 'abort', 'panic', 'exit', 'endAsk'])
 const INPUT_OPS = new Set(['click', 'shiftClick', 'move', 'drag', 'scroll', 'selectRange',
   'type', 'key', 'uiaAct', 'windowOp', 'activate', 'clipWrite', 'calibrate', 'selftest'])
 const isInputOp = (op, args = {}) => INPUT_OPS.has(op) || (op === 'focus' && args.refocus === true)
@@ -148,6 +152,7 @@ export class ComputerUse extends EventEmitter {
     // bare `Error: spawn UNKNOWN` — no path, no syscall, no Windows code — because NOTHING listened
     // for this child's 'error' event: an unhandled 'error' on an EventEmitter is thrown, so the rich
     // error object (which names the file Windows refused) was discarded on the way out.
+    const child = this.proc
     this.proc.on('error', (e) => {
       let file = '?'
       try { file = `${existsSync(this.workerExe)} exists, ${statSync(this.workerExe).size} bytes` } catch { file = 'stat failed' }
@@ -165,11 +170,11 @@ export class ComputerUse extends EventEmitter {
         const line = buf.slice(0, nl).trim()
         buf = buf.slice(nl + 1)
         if (!line) continue
-        this.onLine(line)
+        if (this.proc === child) this.onLine(line)
       }
     })
     this.proc.stderr.on('data', (d) => this.log(`[worker:stderr] ${d}`))
-    this.proc.on('exit', this.onWorkerExit)
+    this.proc.on('exit', (code) => { if (this.proc === child) this.onWorkerExit(code) })
     return new Promise((resolve, reject) => {
       const t = setTimeout(() => reject(new Error('worker ready timeout')), 10_000)
       // surface the real reason instead of letting the raw emitter error escape
@@ -185,6 +190,7 @@ export class ComputerUse extends EventEmitter {
   onWorkerExit (code) {
     this.ready = false
     this.proc = null
+    this.finishUiaRead(false)
     for (const [id, p] of this.pending) { clearTimeout(p.timer); p.cleanup?.(); p.reject(workerCallError('worker exited', 'worker-exited', p.op, id)) }
     this.pending.clear()
     this.emit('dead', code)
@@ -211,6 +217,14 @@ export class ComputerUse extends EventEmitter {
       this.syncCooperation(msg.cooperation)
       return
     }
+    // Secure-desktop transitions (UAC prompt / lock screen, human request 2026-11-04): the
+    // worker refuses to dispatch while it stands and clears it the moment the normal desktop
+    // returns. Mirror the state so callers can show it, and let the plugin notify the human.
+    if (msg.event === 'secure') {
+      this.secureDesktop = msg.why === 'active'
+      this.emit('secure', msg)
+      return
+    }
     // Durable brake events are separate from automatic physical-input handoff.
     //   panic  — ESC: the human ABORTED. The machine is stopped; wrap up briefly, then wait.
     //   resume — Ctrl+Alt+R: work may continue (a PAUSE ends; it never ends the cycle).
@@ -236,6 +250,7 @@ export class ComputerUse extends EventEmitter {
       this.emit(msg.event, msg)
       return
     }
+    if (this.uiaRead?.id === msg.id) this.finishUiaRead(true)
     const p = this.pending.get(msg.id)
     if (!p) return
     this.pending.delete(msg.id)
@@ -246,6 +261,7 @@ export class ComputerUse extends EventEmitter {
     else {
       const error = workerCallError(msg.error || 'worker error', msg.code || 'worker-refusal-or-error', p.op, msg.id,
         msg.outcome === 'not-dispatched' ? 'not-dispatched' : 'unknown')
+      if (msg.glowHealth) error.glowHealth = msg.glowHealth
       p.reject(error.code === 'HUMAN_REOBSERVE' ? humanReobserveError(error) : error)
     }
   }
@@ -266,6 +282,14 @@ export class ComputerUse extends EventEmitter {
     context?.assertOwnership?.()
     try { await this.ensureWorker() }
     catch (error) { throw workerCallError(error.message || String(error), 'worker-unavailable', op, null, 'not-dispatched') }
+    const child = this.proc
+    if (!child) throw workerCallError('worker exited before dispatch', 'worker-exited', op, null, 'not-dispatched')
+    const queueDeadline = Date.now() + timeoutMs
+    while (this.uiaRead && !URGENT_CONTROL.has(op)) {
+      await this.waitForUiaRead(op, input, signal, queueDeadline)
+      if (this.proc !== child) throw workerCallError('worker exited before dispatch', 'worker-exited', op, null, 'not-dispatched')
+      if (Date.now() >= queueDeadline) throw workerCallError('deadline expired waiting for the UIA read', 'worker-uia-busy', op, null, 'not-dispatched')
+    }
     if (signal?.aborted) throw workerCallError('owning tool cancelled', 'INPUT_CANCELLED', op, null, 'not-dispatched')
     beforeDispatch?.()
     context?.assertOwnership?.()
@@ -282,6 +306,7 @@ export class ComputerUse extends EventEmitter {
     if (input) signed.__inputSeries = this.inputContext.getStore()?.inputSeries || randomUUID()
     return new Promise((resolve, reject) => {
       let written = false
+      const read = UIA_READS.has(op) && killOnTimeout ? (this.uiaRead = { id, child, timedOut: false }) : null
       const cleanup = () => signal?.removeEventListener('abort', abort)
       const abort = () => {
         if (!this.pending.delete(id)) return
@@ -293,6 +318,21 @@ export class ComputerUse extends EventEmitter {
       const timer = setTimeout(() => {
         this.pending.delete(id)
         cleanup()
+        if (read && this.uiaRead === read) {
+          read.timedOut = true
+          this.emit('uia-read-settled', false)
+          // Drop the response by id when it eventually arrives. A later reply for
+          // this id cancels only this watchdog, never a newer worker's deadline.
+          const grace = Math.min(timeoutMs, DEFAULT_TIMEOUT_MS)
+          read.watchdog = setTimeout(() => {
+            if (this.proc !== child || this.uiaRead !== read) return
+            this.log(`[core] UIA read ${id} (${op}) still blocked after ${grace}ms recovery grace; stopping worker pid=${child.pid}`)
+            try { child.kill() } catch { /* already gone */ }
+          }, grace)
+          reject(workerCallError(`UIA observation timed out: ${op} (${timeoutMs}ms); worker still finishing this read. Narrow the tree or use a point query once it returns.`,
+            'worker-timeout', op, id, 'not-dispatched'))
+          return
+        }
         // A CALL THAT RELEASES THE BRAKE MUST NOT BE KILLED FOR BEING SLOW. `proc.kill()` deletes no
         // STOP file (there is no ProcessExit handler in worker.cs; `abort` is Environment.Exit(0) at
         // worker.cs:3045), so killing here would end a question with the brake still up — the exact
@@ -303,18 +343,51 @@ export class ComputerUse extends EventEmitter {
         // UIA/COM provider call blocks its single-threaded loop forever), so kill it and let the NEXT
         // call respawn fresh instead of queueing behind a corpse; pending calls are rejected via
         // onWorkerExit.
-        if (killOnTimeout) { try { this.proc && this.proc.kill() } catch { /* already gone */ } }
+        if (killOnTimeout) { try { if (this.proc === child) child.kill() } catch { /* already gone */ } }
         reject(workerCallError(`worker timeout: ${op} (${timeoutMs}ms)` +
           (killOnTimeout
             ? ' — worker killed, next call respawns'
             : ' — worker left alive so the brake can still be released'),
         'worker-timeout', op, id))
-      }, timeoutMs)
+      }, Math.max(0, queueDeadline - Date.now()))
       this.pending.set(id, { resolve, reject, timer, op, input, cleanup, abort })
       signal?.addEventListener('abort', abort, { once: true })
       if (signal?.aborted) { abort(); return }
       written = true
-      this.proc.stdin.write(JSON.stringify({ id, op, args: signed }) + '\r\n')
+      child.stdin.write(JSON.stringify({ id, op, args: signed }) + '\r\n')
+    })
+  }
+
+  finishUiaRead (completed) {
+    const read = this.uiaRead
+    if (!read) return
+    this.uiaRead = null
+    clearTimeout(read.watchdog)
+    this.emit('uia-read-settled', completed && !read.timedOut)
+  }
+
+  // Only the stdin reader's cancellation command is out of band. Preserve
+  // emergency/release requests, but never queue ordinary work behind a slow
+  // observation after its caller has given up. Do not replay any request.
+  waitForUiaRead (op, input, signal, deadline) {
+    const busy = () => workerCallError('native UIA observation still busy; this request was not sent',
+      'worker-uia-busy', op, null, 'not-dispatched')
+    if (this.uiaRead.timedOut || Date.now() >= deadline) return Promise.reject(busy())
+    return new Promise((resolve, reject) => {
+      const finish = (error) => {
+        clearTimeout(timer)
+        this.off('uia-read-settled', settled)
+        this.off('input-cancel', cancel)
+        signal?.removeEventListener('abort', cancel)
+        if (error) reject(error); else resolve()
+      }
+      const settled = (completed) => finish(completed ? null : busy())
+      const cancel = () => finish(workerCallError('cancelled before dispatch', 'INPUT_CANCELLED', op, null, 'not-dispatched'))
+      const timer = setTimeout(() => finish(busy()), Math.max(0, deadline - Date.now()))
+      this.once('uia-read-settled', settled)
+      if (input) this.once('input-cancel', cancel)
+      signal?.addEventListener('abort', cancel, { once: true })
+      if (signal?.aborted) cancel()
     })
   }
 
@@ -643,15 +716,21 @@ export class ComputerUse extends EventEmitter {
   async landmarks (opts = {}) {
     const step = opts.step ?? 16
     const insetPct = opts.insetPct ?? 10
+    // UIA-ONLY MODE (test report 2026-11-04): an animated neighbour (DSH Pet) keeps flipping the
+    // sampled pixel signature, so every refresh came back STALE_MARK. uiaOnly pins identity by
+    // window handle / process / geometry alone — the sampled signature is read (for the report)
+    // but never compared; dispatch-time occlusion checking still guards the actual click.
+    const uiaOnly = opts.uiaOnly === true
     const fs = await this.frameSig({ hwnd: opts.hwnd, step, insetPct })
+    if (uiaOnly) fs.sig = null
     const hwnd = fs.hwnd
-    if (!hwnd || !fs.sig) throw staleMark('window snapshot unavailable')
+    if (!hwnd || (!fs.sig && !uiaOnly)) throw staleMark('window snapshot unavailable')
     const cached = this.maps.get(String(hwnd))
-    const key = JSON.stringify([opts.depth ?? 14, opts.maxNodes ?? 1500, opts.maxMarks ?? 80, opts.minSize ?? 8, opts.role, opts.nameContains])
+    const key = JSON.stringify([opts.depth ?? 14, opts.maxNodes ?? 1500, opts.maxMarks ?? 80, opts.minSize ?? 8, opts.role, opts.nameContains, uiaOnly])
     if (cached && sameMarkFrame(cached.frame, fs) && cached.key === key && !opts.refresh) {
       this.lastMarks = cached.marks
       this.lastMarkFrame = { ...fs, step, insetPct }
-      return { hwnd, marks: cached.marks, count: cached.marks.length, reused: true, ageMs: Date.now() - cached.ts, sig: fs.sig }
+      return { hwnd, marks: cached.marks, count: cached.marks.length, reused: true, ageMs: Date.now() - cached.ts, sig: fs.sig, uiaOnly }
     }
     const uia = await this.call('uia', {
       hwnd: hwnd || undefined,
@@ -662,6 +741,7 @@ export class ComputerUse extends EventEmitter {
     })
     const marks = buildMarks(uia.flat || [], opts.maxMarks ?? 80, opts.minSize ?? 8)
     const after = await this.frameSig({ hwnd, step, insetPct })
+    if (uiaOnly) after.sig = null
     if (!sameMarkFrame(fs, after)) {
       const changed = ['hwnd', 'pid'].filter(k => fs[k] !== after[k])
       for (const k of ['x', 'y', 'width', 'height']) if (fs.region?.[k] !== after.region?.[k]) changed.push(`region.${k}`)
@@ -674,7 +754,7 @@ export class ComputerUse extends EventEmitter {
     this.lastMarkFrame = { ...after, step, insetPct }
     this.maps.set(String(hwnd), { marks, frame: after, sig: after.sig, key, ts: Date.now() })
     if (this.maps.size > 16) this.maps.delete(this.maps.keys().next().value)
-    return { hwnd, marks, count: marks.length, reused: false, sig: fs ? fs.sig : null, elements: (uia.flat || []).length }
+    return { hwnd, marks, count: marks.length, reused: false, sig: fs ? fs.sig : null, uiaOnly, elements: (uia.flat || []).length }
   }
 
   // The human's emergency brake, reachable from the model side too (ESC / Ctrl+Alt+Q are
@@ -748,6 +828,8 @@ export class ComputerUse extends EventEmitter {
   }
 
   async clipRead () { return this.call('clipRead', {}) }
+  /** Worker-side OCR: text + per-word boxes in ABSOLUTE screen pixels (2026-11-04). */
+  async ocr (region) { return this.call('ocr', region ? { ...region } : {}) }
   async clipWrite (text) { return this.actuate('clipWrite', { text }) }
 
   async activateWindow (sel) { return this.actuate('activate', sel) }
@@ -837,8 +919,13 @@ function staleMark (reason) {
 }
 
 function sameMarkFrame (a, b) {
-  return !!a?.sig && a.sig === b?.sig && a.hwnd === b.hwnd && a.pid === b.pid &&
-    ['x', 'y', 'width', 'height'].every(k => Number.isFinite(a.region?.[k]) && a.region[k] === b.region?.[k])
+  const identity = a?.hwnd === b?.hwnd && a?.pid === b?.pid &&
+    ['x', 'y', 'width', 'height'].every(k => Number.isFinite(a.region?.[k]) && a.region[k] === b.region[k])
+  if (!identity) return false
+  // A uiaOnly landmark frame (sig === null, test report 2026-11-04) deliberately carries no
+  // pixel signature: identity is geometry alone, and the click path's occlusion check remains.
+  if (a.sig === null) return true
+  return !!a?.sig && a.sig === b?.sig
 }
 
 /** Keep secrets out of the audit log. */

package/lib/core/native-runtime.js

@@ -13,7 +13,12 @@ const framework = env => path.join(env.SystemRoot || env.windir || 'C:\\Windows'
 export function workerReferences (env = process.env) {
   const gac = path.join(env.SystemRoot || env.windir || 'C:\\Windows', 'Microsoft.NET', 'assembly', 'GAC_MSIL')
   return ['System.dll', 'System.Core.dll', 'System.Drawing.dll', 'System.Web.Extensions.dll', 'System.Windows.Forms.dll',
-    ...['UIAutomationClient', 'UIAutomationTypes', 'WindowsBase'].map(name => path.join(gac, name, 'v4.0_4.0.0.0__31bf3856ad364e35', `${name}.dll`))]
+    ...['UIAutomationClient', 'UIAutomationTypes', 'WindowsBase'].map(name => path.join(gac, name, 'v4.0_4.0.0.0__31bf3856ad364e35', `${name}.dll`)),
+    // Inbox OCR (2026-11-04): the legacy csc cannot compose the split WinMetadata winmds, so the
+    // union Windows.winmd ships inside the package and the WinRT interop pieces come from the OS.
+    path.join(framework(env), 'System.Runtime.dll'),
+    path.join(gac, 'System.Runtime.WindowsRuntime', 'v4.0_4.0.0.0__b77a5c561934e089', 'System.Runtime.WindowsRuntime.dll'),
+    path.join(import.meta.dirname, 'winmd', 'Windows.winmd')]
 }
 
 export function compileNativeBinary (source, output, references, { env = process.env } = {}) {