@momasiku/dsh-pilot 0.5.16 → 0.6.0

package/README.md

@@ -7,7 +7,7 @@
 **Give DeepSeek Harness hands and eyes on your desktop.**
 
 [![awesome · DSH plugin](https://awesome-dsh-plugin.com/badge.svg)](https://awesome-dsh-plugin.com/p/momasiku/dsh-pilot/)
-[![npm](https://img.shields.io/npm/v/@momasiku/dsh-pilot?label=npm)](https://www.npmjs.com/package/@momasiku/dsh-pilot)
+[![npm](https://img.shields.io/npm/v/@momasiku%2Fdsh-pilot?label=npm)](https://www.npmjs.com/package/@momasiku/dsh-pilot)
 [![compat](https://github.com/momasiku/dsh-pilot/actions/workflows/compat.yml/badge.svg)](https://github.com/momasiku/dsh-pilot/actions/workflows/compat.yml)
 [![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
@@ -23,18 +23,32 @@ touch it: you describe what is on screen, you take the screenshots, you do the
 clicking, and you report what happened. **dsh-pilot closes that loop.** The model
 looks at the real pixels and then operates them:
 
+### Demos
+
+<!-- Served from raw.githubusercontent.com, the same host every asset here uses. If
+     that host is slow where you read this, jsDelivr mirrors the same files:
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-cad.gif
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-excel.gif
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-blender.gif -->
+
+![dsh-pilot drawing a mechanical plate in AutoCAD: the model types REC and the corner coordinates, the inner outline, five circles, the centre lines and two linear dimensions into the command line, then zooms to extents](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-cad.gif)
+
+**A CAD drawing, drawn by the model.** It drives AutoCAD's own command line — `REC` → `0,0` → `240,150`, the inner outline, five holes, centre lines and two dimensions — and installs nothing inside AutoCAD.
+
+![dsh-pilot renaming a column in LibreOffice Calc: focus the workbook, move the pointer to C1, click, paste "Q3 收入", press Enter, Ctrl+S — all in one desktop_sequence call](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-excel.gif)
+
+**A spreadsheet edit, in one call.** `desktop_sequence` focuses the workbook, moves the pointer to C1, clicks, pastes `Q3 收入`, presses Enter and saves — one round trip for the whole chain.
+
+![dsh-pilot modelling in Blender: the viewport is cleared, a cube is added, and subdivision turns it into a sphere, then the model orbits to check it](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-blender.gif)
+
+**A 3D model, built from nothing.** It clears the viewport, adds a cube, then `Ctrl+1` and `Ctrl+3` subdivision turn that cube into a sphere. Real Blender 4.2 on a machine with **no GPU** — driven through Mesa's software OpenGL, which the model set up itself.
+
 | tool | what it does |
 | --- | --- |
 | **`screen_view`** | captures the live desktop and returns the PNG as a **real image block** the model can see — not OCR, not a text summary |
 | **`desktop_control`** | moves the pointer, clicks, drags, scrolls, types, sends keys, focuses and lists windows — and can capture the result in the same call |
 | **`desktop_sequence`** | runs a **whole batch** of those actions in one call and looks once at the end |
 
-```text
-You : 把浏览器里那个报表的第三列改名成 "Q3 收入"，然后保存
-DSH : desktop_sequence([focus "报表.xlsx", click 812,430, type "Q3 收入", key enter, key ctrl+s])
-      → 1 round trip, 1 screenshot, per-step timings, done
-```
-
 ## Why this exists
 
 A model that cannot see the screen has to be told what is on it. A model that cannot
@@ -234,6 +248,51 @@ and what `dsh plugin add` checks before it installs anything. Requirements:
 
 ### Recent changes
 
+**0.6.0** — a takeover line, a lit pointer, and titles that stop hiding characters.
+
+- **The screen says who is driving.** While an action runs, one line of large
+  translucent type names the operator and the way out — `DSH 正在操作你的电脑 · 按 Esc
+  取消` — and a halo follows the cursor. No border: the first version drew one, and at
+  2560x1600 it was either invisible against a blue wallpaper or heavy enough to feel
+  like a frame. Sizes come from the virtual desktop, so a 4K wall and a 1080p laptop
+  get the same visual weight.
+- **Esc cancels the batch, not the step.** The overlay stamps the cancel token the
+  action loop already polled; that token now belongs to the whole `desktop_sequence`,
+  so a `wait` in the middle is interrupted too, and the result says why:
+  `the person pressed Esc; the batch stopped where it was`. Measured: a 12 s batch with
+  Esc at 4.5 s returned after 7.1 s.
+- **Titles are compared after stripping zero-width characters.** Edge's own title has
+  a U+200B inside `Microsoft Edge`; `focus "Microsoft Edge"` failed with "no visible
+  top-level window matches" on a window the person could read perfectly.
+- **`focus` accepts an `hwnd` on its own**, as the description always promised.
+- **Every helper script must keep its UTF-8 BOM** — `script-smoke` fails if one loses
+  it, because PowerShell 5.1 reads a BOM-less file as ANSI and the Chinese caption
+  renders as mojibake. This caught three other scripts that had already lost it.
+- The pointer halo is drawn, never written: no `SetCursorPos` and no input hook, which
+  is why the ring never makes a person's own mouse feel laggy.
+
+- **A takeover line and a lit pointer.** While an action runs, one line of large
+  translucent type names the operator and the way out — `DSH 正在操作你的电脑 · 按 Esc
+  取消` — and a halo follows the cursor. No border around the screen: the first version
+  drew one, and at 2560x1600 it was either invisible against a blue wallpaper or heavy
+  enough to feel like a frame. The type and the halo say the same thing without taking
+  the desktop hostage. Sizes come from the virtual desktop, so a 4K wall and a 1080p
+  laptop get the same visual weight.
+- **Esc cancels the batch, not the step.** The overlay stamps the same cancel token the
+  action loop already polled; that token now belongs to the whole `desktop_sequence`,
+  so a `wait` in the middle is interrupted too, and the result says why:
+  `the person pressed Esc; the batch stopped where it was`. Measured: a 12 s batch with
+  Esc at 4.5 s returned after 7.1 s.
+- **The pointer is lit, not taken over.** A halo tracks the cursor while the ring is up,
+  so it is obvious the movement is not yours. It is drawing, not the pointer: Windows
+  paints the real cursor above every window, so the halo sits under it and the pointer
+  reads as larger and lit without being replaced — and without the cursor being written,
+  which is what makes other agents' takeovers feel laggy. The halo lives in its own small
+  window rather than as a child of the ring: a child Grid with a runtime margin never
+  painted on this layered surface, while a window that is simply moved does.
+- Fully permissioned sessions are not asked again — the ring is the statement, not a
+  prompt. `takeover: false` turns it off.
+
 **0.5.14** — the coordinate contract stops guessing how big the image you see is.
 
 - **The request pipeline resizes again after capture**, and this plugin never saw
@@ -508,18 +567,32 @@ DSH 原本只会"说"：它能推理、能写代码、能规划，但**看不见
 
 **dsh-pilot 把这个环闭上**：模型直接看真实像素，然后自己动手。
 
+### 实例演示
+
+<!-- 图片走 raw.githubusercontent.com（本仓库所有资源都用这个域名）。若在你那边较慢，
+     jsDelivr 上有同样的镜像：
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-cad.gif
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-excel.gif
+     https://cdn.jsdelivr.net/gh/momasiku/dsh-pilot@main/assets/dsh-pilot-blender.gif -->
+
+![dsh-pilot 在 AutoCAD 里画一块机械板：模型把 REC、角点坐标、内轮廓、五个圆、中心线和两道线性标注依次输入命令行，最后缩放到范围](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-cad.gif)
+
+**模型自己画出来的 CAD 图。** 用的就是 AutoCAD 自己的命令行——`REC` → `0,0` → `240,150`、内轮廓、五个圆孔、中心线、两道尺寸，**AutoCAD 里什么都没装**。
+
+![dsh-pilot 在 LibreOffice Calc 里改一列表头：聚焦工作簿、鼠标移到 C1、点击、粘贴「Q3 收入」、回车、Ctrl+S —— 全部在一次 desktop_sequence 调用里完成](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-excel.gif)
+
+**一次调用改完一张表。** `desktop_sequence` 聚焦工作簿、把鼠标移到 C1、点击、粘贴「Q3 收入」、回车、保存——整串动作只花一次往返。
+
+![dsh-pilot 在 Blender 里建模：先清空视口，加一个立方体，再用细分曲面把它磨成一个球，最后轨道旋转检查](https://raw.githubusercontent.com/momasiku/dsh-pilot/main/assets/dsh-pilot-blender.gif)
+
+**从零建一个 3D 模型。** 清空视口 → 加一个立方体 → `Ctrl+1`、`Ctrl+3` 两级细分，方盒子被磨成一个球。真机上的 Blender 4.2，跑在**一台没有显卡的机器**上——靠 Mesa 的软件 OpenGL，而那套环境也是模型自己配好的。
+
 | 工具 | 作用 |
 | --- | --- |
 | **`screen_view`** | 抓取当前屏幕，把 PNG 作为**真正的图像块**交给模型（不是 OCR、不是文字描述），连光标位置、前台窗口、各显示器范围一起给出 |
 | **`desktop_control`** | 移动鼠标、单击/双击/右键、拖拽、滚轮、打字、按键、聚焦与列出窗口；可以在同一次调用里顺手截一张，让效果立刻可见 |
 | **`desktop_sequence`** | **一次调用跑完一整串动作**，最后只看一眼结果 |
 
-```text
-你 ：把浏览器里那张报表的第三列改名成 "Q3 收入"，然后保存
-DSH：desktop_sequence([focus "报表.xlsx", click 812,430, type "Q3 收入", key enter, key ctrl+s])
-     → 1 次往返、1 张截图、每步耗时，完成
-```
-
 ## 它到底解决了什么
 
 桌面自动化的瓶颈**从来不是鼠标点得快不快**，而是 **"模型和电脑要来回说多少次话"**：

package/lib/index.js

@@ -45,7 +45,9 @@ const Config = z.object({
 	/** Draw coordinate rulers on every frame, so readings do not depend on counting pixels. */
 	rulers: z.boolean().default(true),
 	/** Also capture a frame when a tool result carries no image (still materializes the file). */
-	alwaysSaveFile: z.boolean().default(true)
+	alwaysSaveFile: z.boolean().default(true),
+	/** Ring the desktop and offer Esc while an action runs, so a person can see who is driving and stop it. */
+	takeover: z.boolean().default(true)
 });
 
 /** Services this plugin registers against. */
@@ -109,12 +111,12 @@ function nextToken() {
  * @param timeoutMs - hard kill deadline in milliseconds.
  * @returns the parsed JSON result object.
  */
-function runHelper(script, params, signal, timeoutMs) {
+function runHelper(script, params, signal, timeoutMs, cancelPathOverride) {
 	if (!existsSync(script)) throw new Error(`pilot: helper script is missing at ${script}`);
 	const dir = scratchDir();
 	const token = nextToken();
 	const paramsPath = join(dir, `params-${token}.json`);
-	const cancelPath = join(dir, `cancel-${token}.txt`);
+	const cancelPath = cancelPathOverride ?? join(dir, `cancel-${token}.txt`);
 	writeFileSync(paramsPath, JSON.stringify({ ...params, cancelFile: cancelPath }), 'utf8');
 
 	return new Promise((resolvePromise, rejectPromise) => {
@@ -485,27 +487,116 @@ function warmWorker(config) {
  * @param options - signal and timeout.
  * @returns the parsed result.
  */
-async function runAction(params, config, options) {
-	if (config.useWorker === false || !actionWorker.usable()) {
-		return runHelper(actionScript(), params, options.signal, options.timeoutMs);
-	}
-	const cancelPath = join(scratchDir(), `cancel-${nextToken()}.txt`);
+async function runAction(params, config, options, cancelPathOverride) {
+	const ownsRing = cancelPathOverride === undefined;
+	const cancelPath = cancelPathOverride ?? join(scratchDir(), `cancel-${nextToken()}.txt`);
+	const startedAt = Date.now();
+	const takeDir = ownsRing ? beginTakeover(cancelPath, config) : null;
 	try {
-		return await actionWorker.request({ ...params, cancelFile: cancelPath }, options);
-	} catch (error) {
-		if (error?.transport === true) {
-			return runHelper(actionScript(), params, options.signal, options.timeoutMs);
+		if (config.useWorker === false || !actionWorker.usable()) {
+			return await runHelper(actionScript(), params, options.signal, options.timeoutMs, cancelPath);
 		}
-		throw error;
-	} finally {
 		try {
-			rmSync(cancelPath, { force: true });
-		} catch {
-			// A leftover cancel file is harmless: the token is unique.
+			return await actionWorker.request({ ...params, cancelFile: cancelPath }, options);
+		} catch (error) {
+			if (error?.transport === true) {
+				return await runHelper(actionScript(), params, options.signal, options.timeoutMs, cancelPath);
+			}
+			// An abort and an Esc both land here as "cancelled"; only one of them was
+			// a person deciding, and the model should be told which.
+			if (error?.cancelled === true && tookEsc(takeDir, startedAt)) {
+				throw Object.assign(new Error('pilot: the person pressed Esc and cancelled this action'), { cancelled: true });
+			}
+			throw error;
+		}
+	} finally {
+		endTakeover(takeDir);
+		if (ownsRing) {
+			try {
+				rmSync(cancelPath, { force: true });
+			} catch {
+				// A leftover cancel file is harmless: the token is unique.
+			}
 		}
 	}
 }
 
+/** The one overlay process, started the first time a takeover is worth showing. */
+let overlayProcess = null;
+
+/** Absolute path of the takeover overlay helper. */
+function overlayScript() {
+	return join(packageRoot(), 'lib', 'scripts', '_dsh-overlay.ps1');
+}
+
+/** Directory the overlay polls for its instructions. */
+function takeoverDir() {
+	const dir = join(scratchDir(), 'takeover');
+	mkdirSync(dir, { recursive: true });
+	return dir;
+}
+
+/**
+ * Raise the takeover ring for one action and hand it the token to stamp.
+ *
+ * The ring is decoration plus one hotkey: it never writes the cursor and never
+ * installs an input hook, because either would add exactly the input lag that
+ * makes a person's own mouse feel broken while an agent drives. Esc writes the
+ * same "stop" the host writes on abort, so the loop stops between two steps.
+ *
+ * @param cancelPath - the cancel token this action polls.
+ * @param config - validated plugin configuration.
+ * @returns the directory to clear afterwards, or null when the ring is off.
+ */
+function beginTakeover(cancelPath, config) {
+	if (config.takeover === false) return null;
+	let dir;
+	try {
+		dir = takeoverDir();
+		writeFileSync(join(dir, 'show.txt'), cancelPath, 'utf8');
+	} catch {
+		return null;
+	}
+	if (overlayProcess !== null && overlayProcess.exitCode === null) return dir;
+	try {
+		overlayProcess = spawn(
+			'powershell.exe',
+			['-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass', '-File', overlayScript(), '-StateDir', dir],
+			{ stdio: 'ignore', windowsHide: true }
+		);
+		overlayProcess.on('exit', () => {
+			overlayProcess = null;
+		});
+		overlayProcess.on('error', () => {
+			overlayProcess = null;
+		});
+		overlayProcess.unref();
+	} catch {
+		overlayProcess = null;
+	}
+	return dir;
+}
+
+/** Drop the ring: the host owns visibility, the overlay only polls the file. */
+function endTakeover(dir) {
+	if (dir === null) return;
+	try {
+		rmSync(join(dir, 'show.txt'), { force: true });
+	} catch {
+		// Leaving it would keep the ring up; the next action rewrites it anyway.
+	}
+}
+
+/** True when the person pressed Esc during this action. */
+function tookEsc(dir, sinceMs) {
+	if (dir === null) return false;
+	try {
+		return statSync(join(dir, 'esc.txt')).mtimeMs >= sinceMs;
+	} catch {
+		return false;
+	}
+}
+
 /**
  * Run one capture: through the warm worker when it is available, otherwise
  * through the one-shot probe.
@@ -1138,7 +1229,9 @@ function normalizeSequence(args, config) {
 		}
 		if (action === 'type' && typeof normalized.text !== 'string') throw new Error(`step ${position} (type) needs text`);
 		if (action === 'key' && typeof normalized.key !== 'string') throw new Error(`step ${position} (key) needs key`);
-		if (action === 'focus' && typeof normalized.title !== 'string') throw new Error(`step ${position} (focus) needs title`);
+		if (action === 'focus' && typeof normalized.title !== 'string' && typeof normalized.hwnd !== 'string') {
+			throw new Error(`step ${position} (focus) needs a title, or an hwnd`);
+		}
 		if (action === 'wait' && !Number.isInteger(normalized.ms)) throw new Error(`step ${position} (wait) needs integer ms`);
 		if (normalized.risk !== undefined && !RISKS.includes(normalized.risk)) {
 			throw new Error(`step ${position} risk must be one of ${RISKS.join('/')}`);
@@ -1178,27 +1271,58 @@ function sequenceRisk(steps, declared, confirm) {
 	return effective;
 }
 
-/** A wait that still honours an interrupted turn. */
-function abortableWait(ms, signal) {
+/** True when the cancel token was stamped. */
+function stopRequested(cancelPath) {
+	if (typeof cancelPath !== 'string' || cancelPath.length === 0) return false;
+	try {
+		return readFileSync(cancelPath, 'utf8').trim() === 'stop';
+	} catch {
+		return false;
+	}
+}
+
+/**
+ * A wait that still honours an interrupted turn AND the person pressing Esc.
+ * A plain timer would sit through a 900 ms settle after the takeover was already
+ * called off, which is the one moment the ring most obviously lies.
+ */
+function abortableWait(ms, signal, cancelPath) {
 	return new Promise((resolvePromise, rejectPromise) => {
-		const timer = setTimeout(resolvePromise, Math.max(0, ms));
+		let timer;
+		const deadline = Date.now() + Math.max(0, ms);
+		const tick = () => {
+			if (signal?.aborted) {
+				rejectPromise(new Error('pilot: the batch was cancelled during a wait'));
+				return;
+			}
+			if (stopRequested(cancelPath)) {
+				rejectPromise(Object.assign(new Error('pilot: the person pressed Esc and cancelled this batch'), { cancelled: true }));
+				return;
+			}
+			const left = deadline - Date.now();
+			if (left <= 0) {
+				resolvePromise();
+				return;
+			}
+			timer = setTimeout(tick, Math.min(100, left));
+		};
 		signal?.addEventListener('abort', () => {
 			clearTimeout(timer);
-			rejectPromise(new Error('pilot: the batch was cancelled during a wait'));
 		}, { once: true });
+		tick();
 	});
 }
 
 /** Run one step: `wait` locally, everything else through the action helper. */
-async function runSequenceStep(step, config, options) {
+async function runSequenceStep(step, config, options, cancelPath) {
 	const started = Date.now();
 	if (step.action === 'wait') {
-		await abortableWait(step.ms, options.signal);
+		await abortableWait(step.ms, options.signal, cancelPath);
 		return { action: 'wait', ok: true, ms: Date.now() - started };
 	}
 	const params = { action: step.action, settleMs: Number.isSafeInteger(step.settleMs) ? step.settleMs : config.stepSettleMs };
 	for (const field of STEP_FIELDS) if (step[field] !== undefined) params[field] = step[field];
-	const acted = await runAction(params, config, options);
+	const acted = await runAction(params, config, options, cancelPath);
 	const entry = { action: step.action, ok: true, ms: Date.now() - started };
 	if (acted.cursor !== undefined) entry.cursor = acted.cursor;
 	if (acted.foregroundTitle !== undefined) entry.foregroundTitle = acted.foregroundTitle;
@@ -1584,24 +1708,47 @@ function apply(ctx, config) {
 			const stopOnError = args.stopOnError !== false;
 			const results = [];
 			let failed = 0;
-
-			for (const [offset, step] of steps.entries()) {
-				if (exec?.signal?.aborted) throw new Error('desktop_sequence: the batch was cancelled');
-				const stepStarted = Date.now();
+			// One token and one ring cover the whole batch, so Esc stops the batch rather
+			// than the single step that happened to be running when the key landed.
+			const batchCancel = join(scratchDir(), `cancel-${nextToken()}.txt`);
+			const batchStarted = Date.now();
+			const batchDir = beginTakeover(batchCancel, config);
+			let cancelledByEsc = false;
+			try {
+				for (const [offset, step] of steps.entries()) {
+					if (exec?.signal?.aborted) throw new Error('desktop_sequence: the batch was cancelled');
+					if (stopRequested(batchCancel)) {
+						cancelledByEsc = true;
+						break;
+					}
+					const stepStarted = Date.now();
+					try {
+						const entry = await runSequenceStep(step, config, options, batchCancel);
+						results.push({ index: offset + 1, ...entry, ...(step.risk === undefined ? {} : { risk: step.risk }) });
+					} catch (error) {
+						failed += 1;
+						results.push({
+							index: offset + 1,
+							action: step.action,
+							ok: false,
+							ms: Date.now() - stepStarted,
+							note: error instanceof Error ? error.message : String(error),
+							...(step.risk === undefined ? {} : { risk: step.risk })
+						});
+						if (error?.cancelled === true) {
+							cancelledByEsc = true;
+							break;
+						}
+						if (stopOnError) break;
+					}
+				}
+			} finally {
+				cancelledByEsc = cancelledByEsc || tookEsc(batchDir, batchStarted);
+				endTakeover(batchDir);
 				try {
-					const entry = await runSequenceStep(step, config, options);
-					results.push({ index: offset + 1, ...entry, ...(step.risk === undefined ? {} : { risk: step.risk }) });
-				} catch (error) {
-					failed += 1;
-					results.push({
-						index: offset + 1,
-						action: step.action,
-						ok: false,
-						ms: Date.now() - stepStarted,
-						note: error instanceof Error ? error.message : String(error),
-						...(step.risk === undefined ? {} : { risk: step.risk })
-					});
-					if (stopOnError) break;
+					rmSync(batchCancel, { force: true });
+				} catch {
+					// The token is unique, so a leftover file cannot cancel anything else.
 				}
 			}
 
@@ -1615,6 +1762,7 @@ function apply(ctx, config) {
 				steps: results,
 				actedAt: new Date().toISOString()
 			};
+			if (cancelledByEsc) value.notes = ['the person pressed Esc; the batch stopped where it was'];
 			const lastCursor = [...results].reverse().find((entry) => entry.cursor !== undefined);
 			if (lastCursor !== undefined) value.cursor = lastCursor.cursor;
 			const lastForeground = [...results].reverse().find((entry) => entry.foregroundTitle !== undefined);