dsh-codex-style-computer-use 1.2.3 → 1.3.0

package/README.md

@@ -12,6 +12,10 @@ English | [中文](README.zh.md)
 
 Windows only (`os: win32`). Requires Node.js ≥ 22 and the .NET Framework 4.x that Windows ships with.
 
+**1.3.0 desktop lifecycle update:** the deep-blue cue now matches actual actuation, with a 16-pixel pointer jitter allowance and a visible 1.5-second purple notice before input. Human takeover returns a required continuation checkpoint. Session/agent ownership and a native cross-process lease allow only one controller. Call `computer_ctrl {action:"release"}` after desktop work and verification; the conversation continues and later desktop work can open a fresh cycle in the same turn. Normal completion releases automatically without an exit question card. Explicit pauses, exits and pending questions remain authoritative.
+
+The indicators use continuous 2-pixel rails with square screen corners and an 8-pixel glow. Normal working states have no floating badges; handoff, pause and preparation use compact top hints. Content interiors remain transparent. The visual revision preserves state logic and animation timing.
+
 **1.2.3 automatic fix:** normal installation/update now installs the approval-attribution correction for computer-use calls on affected DSH hosts. No manual host patch is required for those calls. Host approvals and audit remain authoritative; follow the host's restart indication after updating an already loaded plugin. See [1.2.3](docs/RELEASE-1.2.3.md).
 
 **1.2.1 attention fix:** fixes a missing crypto import that prevented 1.2.0's native `probe` and `raise` requests from reaching the helper. Updating from 1.2.0 is required for this runtime fix; reinstalling 1.2.0 does not fix it. New releases are staged on npm's `next` tag before promotion to `latest`; the market's normal update follows `latest`. See [1.2.1 validation and release status](docs/RELEASE-1.2.1.md).
@@ -137,7 +141,7 @@ fields are detected through UI Automation's `IsPassword` and refused by default.
 
 ## Safety model
 
-- **Shared input** — physical typing, scrolling, mouse movement and mouse buttons immediately yield control. Continued input or a held key/button for 2 seconds enters automatic waiting. After all keys/buttons are released and 3 seconds pass without physical input, the agent takes a fresh observation and continues the authorized task. A brief touch also waits for the 3-second quiet interval. A gentle pale-yellow edge gradient shows human ownership.
+- **Shared input** — ordinary user activity is allowed while the agent is thinking, observing or preparing. During actual actuation, typing, scrolling and buttons yield immediately; pointer movement has a 16-pixel displacement tolerance. Continued input or a held key/button for 2 seconds enters automatic waiting. After release and 3 quiet seconds, the agent must inspect the current target and continue the original task, without blindly replaying partial writes. A gentle pale-yellow edge gradient shows human ownership.
 - **Manual pause** — only physical `Ctrl+Esc` raises a persistent input brake; `Ctrl+Alt+R` resumes. Ordinary Esc, typing, scrolling, pointer distance/frequency and the screen corner do not create this pause. Explicit question/diagnostic holds remain separate.
 - **Exit** — `Ctrl+Alt+Q` ends the session: the overlay goes away, its listeners are torn down, and
   further actuation is refused until a later turn opens a new one.

package/README.zh.md

@@ -2,7 +2,7 @@
 
 **让 DeepSeek 看得见，也点得动。**
 
-把 Windows 应用里的重复操作交给 AI：看屏幕、找控件、点鼠标、敲键盘。你一碰鼠标键盘，它就让出控制，等你操作完再继续；默认保留操作审批。
+把 Windows 应用里的重复操作交给 AI：看屏幕、找控件、点鼠标、敲键盘。AI 实际操控前提示 1.5 秒，非操控期间你可以正常使用电脑；操控中接管后，AI 等待交回并接续原任务。默认保留操作审批。
 
 给 **DeepSeek Harness** 用的 Windows 桌面操控插件：18 个桌面工具加一个 Agent 激活入口，通过一个 C# worker
 观察并驱动原生 Windows 应用。发布包已包含编译好的原生组件，无需安装 PowerShell 7、SDK 或执行编译命令。
@@ -11,6 +11,10 @@
 
 仅支持 Windows（`os: win32`）。需要 Node.js ≥ 22 与 Windows 自带的 .NET Framework 4.x。
 
+**1.3.0 桌面控制周期升级：**实际操控与深蓝色指示同步；鼠标轻微抖动有 16 像素容差；紫色操作前提示持续 1.5 秒；接管后返回必须继续任务的断点；同一时间只有一个 agent/worker 拥有控制周期。桌面工作和验证结束后调用 `computer_ctrl {action:"release"}` 即可释放，聊天继续，同一轮后来仍可重新开始。正常收尾自动释放，不再弹出结束确认选项单。人工暂停、明确退出和待回答的问题保持有效。
+
+灯效统一为贴合屏幕直角的 2 像素光线和最多 8 像素柔光，四角没有圆弧缺口。普通工作状态不显示悬浮文字牌；让出、暂停和预告使用紧凑提示，正文区域保持透明。此次美工调整不改变状态逻辑或闪烁节奏。
+
 **1.2.3 自动修复：**正常安装或更新插件，即会为受影响宿主中的 computer-use 调用安装审批归因修正，无需手工修改宿主。审批决定与审计仍由宿主处理；更新已加载插件后按宿主提示重启。详见 [1.2.3](docs/RELEASE-1.2.3.md)。
 
 **1.2.1 提醒修复：**补齐 crypto 导入，修复 1.2.0 原生 `probe` / `raise` 请求还未发送就抛错的问题。这次是运行时代码修复，需要从 1.2.0 更新；重复安装 1.2.0 无效。新版本先发布到 npm 的 `next`，再提升为 `latest`，市场普通更新跟随 `latest`。详见 [1.2.1 验证和发布状态](docs/RELEASE-1.2.1.md)。
@@ -131,7 +135,9 @@ ID**：派发输入前会核对窗口身份与采样像素，因此过期的 mar
 
 ## 安全模型
 
-- **人机共用输入** —— 实体键盘、滚轮、鼠标移动或按钮操作立即让出控制。持续输入或按住键/按钮 2 秒进入自动等待；全部松开并静默 3 秒后，重新观察目标并继续已授权任务。短暂触碰也等待 3 秒静默。人操作时显示温和的淡黄色边缘渐变。
+- **人机共用输入** —— 思考、观察和预告期间允许用户正常操作。实际操控时，键盘、滚轮和按钮立即触发让出；鼠标位移有 16 像素容差。持续输入或按住键/按钮 2 秒进入自动等待；全部松开并静默 3 秒后，agent 必须重新观察并接续原任务，不盲目重放部分写入。人操作时显示温和的淡黄色边缘渐变。
+- **工具用完即释放** —— `computer_ctrl {action:"release"}` 结束电脑控制但保持聊天继续；正常收尾自动释放，不弹确认选项单。暂停和待回答问题不会被自动清除。
+- **一个操作者** —— 会话／agent 所有权与跨 worker 独占锁覆盖整个控制周期，另一 agent 必须等当前操作者释放。
 - **人工暂停** —— 只有实体 `Ctrl+Esc` 触发持续输入刹车，`Ctrl+Alt+R` 恢复。普通 Esc、打字、滚动、鼠标距离/频率和屏幕左上角不再触发这种暂停；显式提问、诊断暂停保持独立。
 - **退出** —— `Ctrl+Alt+Q` 结束会话：覆盖层消失、监听全部拆除，后续注入被拒，直到之后某一轮重新开启。
 - **刹车是关于机器的事实，不属于某一个进程** —— 闩锁状态落盘，每个 worker 都会遵守，包括之后从命令行启动的。

package/lib/batch.js

@@ -1,4 +1,5 @@
 // Serial execution and observation selection; no desktop or lifecycle side effects here.
+import { handoffResult } from './continuation.js'
 const reads = new Set(['computer_state', 'computer_uia', 'computer_marks', 'computer_shot', 'computer_wait'])
 const isRead = (step) => reads.has(step?.tool) || (step?.tool === 'computer_clip' && step.args?.text === undefined)
 const isReadback = (step) => ['computer_state', 'computer_uia', 'computer_marks', 'computer_shot'].includes(step.tool) ||
@@ -10,37 +11,52 @@ export function actionRefusal (step, result) {
     return 'target window activation was not confirmed; subsequent actions were not dispatched'
   }
   if (step.tool === 'computer_ctrl' && step.args?.action === 'recover' && result?.recovered !== true) return result.reason || 'temporary pause recovery refused'
+  if (step.tool === 'computer_ctrl' && step.args?.action === 'release' && !result?.released && !result?.alreadyReleased) return result.reason || 'cycle release refused'
   return null
 }
 
-export async function runBatch ({ steps, handlers, signal, stopped, screenshot, shot = 'auto', exec }) {
+export async function runBatch ({ steps, handlers, signal, stopped, screenshot, shot = 'auto', exec, assertOwnership }) {
   if (!Array.isArray(steps) || !['auto', 'always', 'never'].includes(shot)) throw new Error('invalid batch actions or shot mode')
   const actions = []
   let failed = false
+  let continuation = null
   for (const step of steps) {
     const name = step?.tool
     if (failed || signal?.aborted) {
-      actions.push({ tool: name || '', ok: false, outcome: 'not-dispatched', error: signal?.aborted ? 'cancelled' : 'earlier action failed' })
+      actions.push({ tool: name || '', ok: false, outcome: 'not-dispatched',
+        ...(continuation && !signal?.aborted
+          ? { status: 'deferred', message: 'Resume this unfinished step after fresh target evidence; the original task must continue.' }
+          : { error: signal?.aborted ? 'cancelled' : 'earlier action failed' }) })
       failed = true
       continue
     }
     const fn = Object.hasOwn(handlers, name) && handlers[name]
     if (!fn) { actions.push({ tool: name || '', ok: false, outcome: 'not-dispatched', error: 'unknown tool' }); failed = true; continue }
     try {
+      assertOwnership?.()
       const stoppedBefore = stopped()
       const result = await fn(step.args || {}, exec)
       const refusal = actionRefusal(step, result)
       actions.push({ tool: name, ok: !refusal, result, ...(refusal ? { error: refusal } : {}) })
       failed = !!refusal || (!stoppedBefore && stopped()) ||
-        (name === 'computer_ctrl' && ['stop', 'exit'].includes(step.args?.action))
+        (name === 'computer_ctrl' && ['stop', 'exit', 'release'].includes(step.args?.action))
     } catch (error) {
-      actions.push({ tool: name, ok: false, error: String(error.message || error),
-        code: error.code || 'action-error', outcome: error.outcome || 'unknown' })
+      const checkpoint = !signal?.aborted && !stopped() ? handoffResult(error) : null
+      if (checkpoint) {
+        continuation = { ...checkpoint.continuation, resumeFromActionIndex: actions.length }
+        actions.push({ tool: name, ...checkpoint, continuation })
+      } else {
+        actions.push({ tool: name, ok: false, error: String(error.message || error),
+          code: error.code || 'action-error', outcome: error.outcome || 'unknown' })
+      }
       failed = true
     }
   }
   const needsShot = shot === 'always' || (shot === 'auto' && steps.some(s => !isRead(s)) && !isReadback(steps.at(-1) || {}))
   const result = { actions, attachedShot: null }
+  if (continuation) Object.assign(result, {
+    status: 'continuation-required', taskStatus: 'continuing', transient: true, continuation,
+  })
   if (!failed && !signal?.aborted && !stopped() && needsShot) {
     try { result.attachedShot = await screenshot() }
     catch (error) { result.observationError = String(error.message || error) }

package/lib/continuation.js

@@ -0,0 +1,14 @@
+// Temporary ownership handoff is a task checkpoint, not a failed/cancelled task.
+export function handoffResult (error) {
+  if (error?.code !== 'HUMAN_REOBSERVE' || !error.continuation?.required) return null
+  return {
+    ok: false,
+    status: 'continuation-required',
+    taskStatus: 'continuing',
+    transient: true,
+    code: error.code,
+    outcome: error.outcome || 'unknown',
+    message: error.message,
+    continuation: { ...error.continuation },
+  }
+}

package/lib/core/bin/dsh-computer-use-worker.exe.json

@@ -2,6 +2,6 @@
   "format": 1,
   "platform": "win32",
   "architecture": "x64",
-  "sourceSha256": "7e24c3fa73fed54eb8209efc253753d44664037954d936573e7c80150caf5856",
-  "binarySha256": "ee890a28d1aec984f98b48caf30acab536477ea4d9261892761ea6e8bd485c46"
+  "sourceSha256": "885c7935d5520bc6f865da92e3ab63b29c6515a853e2a3e39ceb2221606acf08",
+  "binarySha256": "c7ae5f0f77912674a8f6f310b95809eef10941cf986f8aa7691aed7a7f08bff5"
 }

package/lib/core/index.js

@@ -10,7 +10,7 @@
 // {"event":"ready"} announcement.
 
 import { spawn, execFileSync } from 'node:child_process'
-import { createHash } from 'node:crypto'
+import { createHash, randomUUID } from 'node:crypto'
 import { EventEmitter } from 'node:events'
 import { AsyncLocalStorage } from 'node:async_hooks'
 import { existsSync, readFileSync, statSync } from 'node:fs'
@@ -24,6 +24,7 @@ const DEFAULT_TIMEOUT_MS = 15_000
 const CAPTURE_TIMEOUT_MS = 30_000
 const INPUT_OPS = new Set(['click', 'shiftClick', 'move', 'drag', 'scroll', 'selectRange',
   'type', 'key', 'uiaAct', 'windowOp', 'activate', 'clipWrite', 'calibrate', 'selftest'])
+const isInputOp = (op, args = {}) => INPUT_OPS.has(op) || (op === 'focus' && args.refocus === true)
 // THE CORE'S ASK DEADLINE, AND IT IS A BARE LITERAL ON PURPOSE (merge-design.md §3, §3.2).
 //
 // It bounds ONE thing: a wedged worker inside `beginAsk`'s raise sequence (a hung UIA provider), or
@@ -48,6 +49,28 @@ export function workerCallError (message, code, op, requestId, outcome = 'unknow
     { code, op, requestId, outcome })
 }
 
+/** A handoff interrupts an input operation, never the authorized task. */
+export function humanContinuation (error = {}) {
+  return {
+    required: true,
+    taskStatus: 'continuing',
+    handback: 'received',
+    nextAction: 'observe-current-target-then-continue-original-task',
+    replayInterruptedInput: false,
+    interruptedOperation: error.op || null,
+    requestId: error.requestId ?? null,
+    outcome: error.outcome === 'not-dispatched' ? 'not-dispatched' : 'unknown',
+    instruction: 'Temporary input interruption only. Control has been handed back. You MUST immediately obtain fresh target evidence, verify any partial effect, and continue the original authorized task from its unfinished step until complete. Do not abandon the task, issue a final wrap-up, or ask for permission to continue because of this handoff. Never blindly replay an interrupted write. Explicit user cancellation, manual pause, or exit still takes precedence.',
+  }
+}
+
+function humanReobserveError (error = {}) {
+  const continuation = humanContinuation(error)
+  return Object.assign(workerCallError(continuation.instruction, 'HUMAN_REOBSERVE',
+    continuation.interruptedOperation, continuation.requestId, continuation.outcome),
+  { transient: true, taskStatus: 'continuing', continuation })
+}
+
 /** Interactive roles that receive Set-of-Marks labels. */
 const SOM_ROLES = new Set([
   'Button', 'MenuItem', 'ListItem', 'TreeItem', 'DataItem', 'TabItem',
@@ -220,8 +243,11 @@ export class ComputerUse extends EventEmitter {
     p.cleanup?.()
     if (msg.data?.cooperation) this.syncCooperation(msg.data.cooperation)
     if (msg.ok) p.resolve(msg.data)
-    else p.reject(workerCallError(msg.error || 'worker error', msg.code || 'worker-refusal-or-error', p.op, msg.id,
-      msg.outcome === 'not-dispatched' ? 'not-dispatched' : 'unknown'))
+    else {
+      const error = workerCallError(msg.error || 'worker error', msg.code || 'worker-refusal-or-error', p.op, msg.id,
+        msg.outcome === 'not-dispatched' ? 'not-dispatched' : 'unknown')
+      p.reject(error.code === 'HUMAN_REOBSERVE' ? humanReobserveError(error) : error)
+    }
   }
 
   /**
@@ -231,12 +257,18 @@ export class ComputerUse extends EventEmitter {
    * @param {boolean} [killOnTimeout] kill a wedged worker when that cap expires (default true)
    * @returns {Promise<any>} worker `data` payload
    */
-  async call (op, args = {}, timeoutMs = DEFAULT_TIMEOUT_MS, killOnTimeout = true) {
-    const signal = INPUT_OPS.has(op) ? this.inputContext.getStore()?.signal : undefined
+  async call (op, args = {}, timeoutMs = DEFAULT_TIMEOUT_MS, killOnTimeout = true, beforeDispatch) {
+    const input = isInputOp(op, args)
+    const context = input ? this.inputContext.getStore() : undefined
+    const signal = context?.signal
     if (signal?.aborted) throw workerCallError('owning tool cancelled', 'INPUT_CANCELLED', op, null, 'not-dispatched')
+    beforeDispatch?.()
+    context?.assertOwnership?.()
     try { await this.ensureWorker() }
     catch (error) { throw workerCallError(error.message || String(error), 'worker-unavailable', op, null, 'not-dispatched') }
     if (signal?.aborted) throw workerCallError('owning tool cancelled', 'INPUT_CANCELLED', op, null, 'not-dispatched')
+    beforeDispatch?.()
+    context?.assertOwnership?.()
     const id = ++this.seq
     // `__cycle` is this layer's SIGNATURE on a call made on the agent's behalf, and the worker
     // opens (or keeps lit) a computer-use CYCLE only for signed calls. A one-shot
@@ -245,6 +277,9 @@ export class ComputerUse extends EventEmitter {
     // LAST so an op's own args can never forge it.
     const signed = { ...args, __cycle: true }
     signed.__kickoff = this.kickoffRequired
+    // One warning per external call/batch, with a fresh identity for each context.
+    // Never let a caller-supplied argument reuse another tool's announcement.
+    if (input) signed.__inputSeries = this.inputContext.getStore()?.inputSeries || randomUUID()
     return new Promise((resolve, reject) => {
       let written = false
       const cleanup = () => signal?.removeEventListener('abort', abort)
@@ -275,7 +310,7 @@ export class ComputerUse extends EventEmitter {
             : ' — worker left alive so the brake can still be released'),
         'worker-timeout', op, id))
       }, timeoutMs)
-      this.pending.set(id, { resolve, reject, timer, op, cleanup, abort })
+      this.pending.set(id, { resolve, reject, timer, op, input, cleanup, abort })
       signal?.addEventListener('abort', abort, { once: true })
       if (signal?.aborted) { abort(); return }
       written = true
@@ -293,12 +328,17 @@ export class ComputerUse extends EventEmitter {
   // ---------------- screen observation ----------------
   syncCooperation (state) {
     if (!state || Number(state.revision) < Number(this.cooperation.revision)) return
+    if (Number(state.revision) === Number(this.cooperation.revision) && Number.isFinite(state.inputEpoch) &&
+        state.inputEpoch < (this.cooperation.inputEpoch || 0)) return
+    const inputChanged = Number.isFinite(state.inputEpoch) && state.inputEpoch !== (this.cooperation.inputEpoch || 0)
     this.cooperation = state
-    if (state.active) { this.maps.clear(); this.lastMarks = []; this.lastMarkFrame = null }
+    if (state.active || inputChanged) { this.maps.clear(); this.lastMarks = []; this.lastMarkFrame = null }
     this.emit('handoff', state)
   }
 
-  withInputContext (context, fn) { return this.inputContext.run({ signal: context?.signal }, fn) }
+  withInputContext (context, fn) {
+    return this.inputContext.run({ signal: context?.signal, assertOwnership: context?.assertOwnership, inputSeries: randomUUID() }, fn)
+  }
 
   waitForHuman () {
     const signal = this.inputContext.getStore()?.signal
@@ -324,13 +364,13 @@ export class ComputerUse extends EventEmitter {
   async waitInputReady () {
     if (!this.cooperation.active) return
     await this.waitForHuman()
-    throw workerCallError('Human input ended and the three-second quiet interval completed. Observe the current target, then continue this task automatically. Previous coordinates/focus may have changed.',
-      'HUMAN_REOBSERVE', null, null, 'not-dispatched')
+    this.inputContext.getStore()?.assertOwnership?.()
+    throw humanReobserveError({ outcome: 'not-dispatched' })
   }
 
   cancelInputs () {
     this.emit('input-cancel')
-    for (const pending of [...this.pending.values()]) if (INPUT_OPS.has(pending.op)) pending.abort?.()
+    for (const pending of [...this.pending.values()]) if (pending.input || INPUT_OPS.has(pending.op)) pending.abort?.()
   }
 
   /** Raw capture; returns { image: Buffer, mime, region, cursor, settled, settleMs, ts, scaledTo }. */
@@ -463,7 +503,7 @@ export class ComputerUse extends EventEmitter {
    * emergency brake. Called when the agent's turn ends, so a lingering blue border can never
    * claim the machine is still under control after control actually stopped.
    */
-  async calm () { return this.call('calm', {}) }
+  async calm ({ beforeDispatch } = {}) { return this.call('calm', {}, DEFAULT_TIMEOUT_MS, true, beforeDispatch) }
 
   /// **Ask the human one question, machine paused — the ONE question path.**
   ///
@@ -584,7 +624,9 @@ export class ComputerUse extends EventEmitter {
    * the intended target, whether it had to be re-asserted, and an integrity-level warning when
    * Windows (UIPI) would silently discard our input. Text-only — no screenshot needed.
    */
-  async focus (opts = {}) { return this.call('focus', opts) }
+  async focus (opts = {}) {
+    return opts.refocus === true ? this.actuate('focus', opts) : this.call('focus', opts)
+  }
 
   /** Sampled fingerprint plus window identity/geometry; a change invalidates cached marks. */
   async frameSig (opts = {}) { return this.call('frameSig', opts) }
@@ -763,10 +805,23 @@ export class ComputerUse extends EventEmitter {
       if (err.code === 'HUMAN_YIELD') {
         // The hook's event can arrive just after its operation refusal. One cheap local
         // status read closes that race; waiting spends no model requests and never replays input.
-        await this.call('ping', {})
-        await this.waitForHuman()
-        err.code = 'HUMAN_REOBSERVE'
-        err.message += ' Human input is now idle for at least three seconds. Inspect the current target and continue automatically; do not repeat any partial write without a readback.'
+        try {
+          await this.call('ping', {})
+          await this.waitForHuman()
+          this.inputContext.getStore()?.assertOwnership?.()
+          err = humanReobserveError({ ...err, op })
+        } catch (waitError) {
+          // Cancelling the local wait cannot erase input already accepted before handoff.
+          err = workerCallError(waitError.message || String(waitError), waitError.code || 'handoff-wait-failed',
+            op, err.requestId ?? null, err.outcome === 'not-dispatched' ? 'not-dispatched' : 'unknown')
+        }
+      }
+      if (err.code === 'HUMAN_REOBSERVE') {
+        try { this.inputContext.getStore()?.assertOwnership?.() }
+        catch (ownerError) {
+          err = workerCallError(ownerError.message, ownerError.code || 'COMPUTER_OWNER_CHANGED', op,
+            err.requestId ?? null, err.outcome)
+        }
       }
       rec.error = String(err.message || err)
       throw err