@momasiku/dsh-pilot 0.5.12 → 0.5.16

package/README.md

@@ -2,14 +2,16 @@
 
 # dsh-pilot
 
+## 让 DSH 看见并高效地操作你的电脑
+
 **Give DeepSeek Harness hands and eyes on your desktop.**
 
+[![awesome · DSH plugin](https://awesome-dsh-plugin.com/badge.svg)](https://awesome-dsh-plugin.com/p/momasiku/dsh-pilot/)
 [![npm](https://img.shields.io/npm/v/@momasiku/dsh-pilot?label=npm)](https://www.npmjs.com/package/@momasiku/dsh-pilot)
 [![compat](https://github.com/momasiku/dsh-pilot/actions/workflows/compat.yml/badge.svg)](https://github.com/momasiku/dsh-pilot/actions/workflows/compat.yml)
 [![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 ![platform](https://img.shields.io/badge/platform-Windows-0078d4.svg)
-![DSH](https://img.shields.io/badge/DSH-0.1.5--rc.2%20%E2%86%92%200.2.x-5b50ed.svg)
 
 [English](#why-this-exists) · [中文说明](#中文说明)
 
@@ -232,6 +234,30 @@ and what `dsh plugin add` checks before it installs anything. Requirements:
 
 ### Recent changes
 
+**0.5.14** — the coordinate contract stops guessing how big the image you see is.
+
+- **The request pipeline resizes again after capture**, and this plugin never saw
+  that step: on a 2560x1600 display the frame reached the model as 1610x1006 while
+  the contract said "delivered at capture size … do not rescale", so every reading
+  taken off it was 1.59x off. The contract now **predicts** the projected size from
+  the image budget the deployment publishes, and prints the size and the multiplier:
+  `2560x1600 capture -> 1609x1006 preview, multiply every reading by 1.5911`. A frame
+  whose area fits the budget arrives untouched, so framing the window or a region
+  gives a true 1:1 frame — 1365x960 does, 1024x768 does, 4K and 5K say 2.2642x and
+  3.0189x instead.
+
+**0.5.12** — a cold tree, and an empty list that explained nothing.
+
+- **Chromium builds its accessibility tree lazily**, and the first caller pays for it:
+  measured on the harness window right after a restart, 3 elements at 1.2 s and 77
+  elements at 0.3 s on the next call. A caller that trusts the first answer concludes
+  the window has no tree and falls back to pixels for nothing. `controls` now retries
+  once, 300 ms later, when the worker has not yet seen a real tree and the answer is
+  suspiciously small.
+- **A filtered listing that legitimately matches nothing** now says so, instead of
+  returning an empty list that reads as "this window has no tree":
+  `(no operable controls here; the window has 2 elements - pass meaningfulOnly:false to see them)`.
+
 **0.5.11** — the control tree, and two fixes that make a batch honest.
 
 - **`controls` reads the window's UI Automation tree**, and `target` clicks a listed
@@ -616,6 +642,15 @@ dsh plugin --profile <你的profile> add "file:E:\path\to\dsh-pilot"
 
 ### 最近改动
 
+**0.5.14** —— 坐标契约不再猜「你看到的那张图有多大」。
+
+- **请求管线会在抓取之后再缩一次**，而插件看不到这一步：2560×1600 的屏幕上，帧送到模型时是 1610×1006，契约却写着「交付即抓取尺寸……不要缩放」，于是每个读数都偏 1.59 倍。现在契约用部署公布的图片预算**预测**投影尺寸，并把尺寸和乘数一起写出来：`2560x1600 抓取 → 1609x1006 预览，把每个读数乘 1.5911`。面积落在预算内的帧会原样送达，所以抓窗口或抓区域能得到真正的 1:1 —— 1365×960 可以，1024×768 可以；4K 和 5K 则分别写 2.2642 倍和 3.0189 倍。
+
+**0.5.12** —— 冷树，以及一个什么都没说的空列表。
+
+- **Chromium 的无障碍树是懒加载的**，而且由第一个来问的人付账：实测重启后在 1.2 秒时只有 3 个元素，下一次调用 0.3 秒时是 77 个。信任第一个答案的人会认定「这窗口没有树」，白白退回像素。现在 `controls` 在「worker 还没见过完整树、且结果异常少」时会等 300 ms 重试一次。
+- **过滤后确实什么都没匹配到时，现在会明说**，而不是返回一个读起来像「这里没有树」的空列表：`(no operable controls here; the window has 2 elements - pass meaningfulOnly:false to see them)`。
+
 **0.5.11** —— 控件树，外加两个让批量操作「不说谎」的修复。
 
 - **`controls` 读窗口的 UI Automation 树**，`target` 点列表里那个控件的精确矩形，于是有名字的控件不必再去像素里找。它不截图，所以也不付 settle。在 673 个元素的窗口上实测：过滤后 **0.65 秒、约 850 token**（77 个可操作元素），不过滤约 9k。暖 worker 记着它发出去的那份列表，所以某一行换了身份会报错，而不是点到搬进来的那个。

package/lib/index.js

@@ -654,6 +654,11 @@ async function capture(ctx, exec, config, args) {
 			...(ref.name === undefined ? {} : { name: ref.name }),
 			...(ref.originalDimensions === undefined ? {} : { originalDimensions: { ...ref.originalDimensions } })
 		};
+		// Predict where the request pipeline will put this frame, from the same budget
+		// it uses. A prediction, always paired with the deferral to the preview size the
+		// model is actually shown.
+		const projected = projectedRequestSize(Number(result.image.width), Number(result.image.height), limits);
+		if (projected !== undefined) result.requestPreview = projected;
 	} catch (error) {
 		result.imageError = `the frame could not be attached: ${error instanceof Error ? error.message : String(error)}`;
 	}
@@ -736,26 +741,76 @@ function captureContent(value) {
  * @param value - a capture, or a tool result carrying one under `capture`.
  * @returns the contract text for that frame.
  */
+/**
+ * Where the request pipeline will put this frame.
+ *
+ * The store hands back a normalized image, and then the request pipeline shrinks it
+ * again to fit the model's image budget - after this plugin has let go of it, which
+ * is why a frame that really was 1:1 at capture can still reach the model smaller.
+ * The geometry is a pure function of the budget the deployment publishes, so the
+ * pipeline can be predicted instead of described. Mirrors requestImageDimensions()
+ * in @deepseek-ai/dsh-attachment, and kept local so a drift between the two can only
+ * cost accuracy: the contract always defers to the preview size the message states.
+ *
+ * @param width - normalized image width.
+ * @param height - normalized image height.
+ * @param limits - the deployment's ImageAttachmentLimits.
+ * @returns predicted request dimensions plus the budget they came from, or undefined.
+ */
+function projectedRequestSize(width, height, limits) {
+	const maxPixels = Number(limits?.maxImagePixels);
+	const maxDimension = Number(limits?.maxImageDimension);
+	if (!Number.isFinite(width) || !Number.isFinite(height) || width <= 0 || height <= 0) return undefined;
+	let w = width;
+	let h = height;
+	if (Number.isFinite(maxDimension) && maxDimension > 0 && Math.max(w, h) > maxDimension) {
+		if (w >= h) { h = Math.max(1, Math.round(maxDimension * h / w)); w = maxDimension; }
+		else { w = Math.max(1, Math.round(maxDimension * w / h)); h = maxDimension; }
+	}
+	if (Number.isFinite(maxPixels) && maxPixels > 0 && w * h > maxPixels) {
+		const s = Math.min(1, Math.sqrt(maxPixels / (w * h)));
+		if (w >= h) {
+			w = Math.max(1, Math.floor(w * s));
+			h = Math.max(1, Math.round(w * height / width));
+			while (w * h > maxPixels && w > 1) { w -= 1; h = Math.max(1, Math.round(w * height / width)); }
+		} else {
+			h = Math.max(1, Math.floor(h * s));
+			w = Math.max(1, Math.round(h * width / height));
+			while (w * h > maxPixels && h > 1) { h -= 1; w = Math.max(1, Math.round(h * width / height)); }
+		}
+	}
+	const out = { width: w, height: h };
+	if (Number.isFinite(maxPixels) && maxPixels > 0) out.maxPixels = maxPixels;
+	return out;
+}
 function coordinateContract(value) {
 	const capture = value?.capture ?? value;
+	const width = Number(capture?.imageWidth);
+	const height = Number(capture?.imageHeight);
+	const sizeKnown = Number.isFinite(width) && width > 0 && Number.isFinite(height) && height > 0;
+	const frame = sizeKnown ? `${width}x${height}` : 'the captured frame';
 	const scale = deliveredScale(capture);
-	if (scaleIsOne(scale)) {
-		return [
-			'Image pixel (0,0) is the top-left of this frame. Both halves of this device are per-monitor DPI aware and this frame is delivered at capture size, so one image pixel is one real screen pixel:',
-			'to act on a feature at image (x,y), call desktop_control with exactly x and y. Do not rescale for display scaling.'
-		].join(' ');
-	}
-	if (axesAgree(scale)) {
-		const ratio = (scale.x + scale.y) / 2;
-		return [
-			`This frame was delivered at ${printed(ratio)}x the capture, so image pixels are NOT screen pixels:`,
-			`to act on a feature at image (x,y), multiply both readings by ${printed(1 / ratio)} and pass those physical screen pixels to desktop_control.`
-		].join(' ');
+	const storeShrank = !scaleIsOne(scale);
+	// No prediction. The request pipeline resizes a frame to fit the model's image
+	// budget, after this plugin has let go of it, and the budget it uses is not the
+	// deployment's admission limit the plugin can read: measured 64,000,000 in
+	// imageLimits.maxImagePixels against an actual projection of ~1,640,000. A
+	// predicted size built from the wrong number reads as a promise, so the contract
+	// stops predicting and points at the one size that is not a guess - the preview
+	// size the message itself states.
+	const lines = [
+		`Image pixel (0,0) is the top-left of this frame, which was captured at ${frame} physical screen pixels; both halves of this device are per-monitor DPI aware.`
+	];
+	if (storeShrank) {
+		lines.push(axesAgree(scale)
+			? `The attachment store already delivered it at ${printed((scale.x + scale.y) / 2)}x that.`
+			: `The attachment store already delivered it at ${printed(scale.x)}x horizontally and ${printed(scale.y)}x vertically.`);
 	}
-	return [
-		`This frame was delivered at ${printed(scale.x)}x horizontally and ${printed(scale.y)}x vertically of the capture, so image pixels are NOT screen pixels:`,
-		`to act on a feature at image (x,y), multiply the readings by ${printed(1 / scale.x)} and ${printed(1 / scale.y)} (x*${printed(1 / scale.x)}, y*${printed(1 / scale.y)}) and pass those physical screen pixels to desktop_control.`
-	].join(' ');
+	lines.push('The image you are looking at may be smaller again: the request pipeline resizes a frame to fit the model image budget, and this plugin cannot see that step, so do not trust the capture size as the size on screen. This same message states what it settled on as "request preview WxH" - that number is authoritative.');
+	lines.push(sizeKnown
+		? `If the preview is not ${frame}, multiply every reading by ${width} / preview_width (and ${height} / preview_height) before passing it to desktop_control; if it is, call desktop_control with exactly x and y. Ruler labels, when drawn, are already screen coordinates.`
+		: 'If the preview is smaller than the capture, multiply every reading by capture_width / preview_width before passing it to desktop_control.');
+	return lines.join(' ');
 }
 
 /**
@@ -941,6 +996,17 @@ function windowInfoOf(window) {
 	};
 }
 
+/** Where the request pipeline is predicted to put a frame, and the budget used. */
+const REQUEST_PREVIEW_SCHEMA = {
+	type: 'object',
+	additionalProperties: false,
+	properties: {
+		width: { type: 'integer', required: true },
+		height: { type: 'integer', required: true },
+		maxPixels: { type: 'number' }
+	}
+};
+
 const CAPTURE_SCHEMA = {
 	type: 'object',
 	additionalProperties: false,
@@ -969,6 +1035,7 @@ const CAPTURE_SCHEMA = {
 		frameHash: { type: 'string' },
 		unchanged: { type: 'boolean' },
 		rulers: { type: 'boolean' },
+		requestPreview: REQUEST_PREVIEW_SCHEMA,
 		capturedAt: { type: 'string' }
 	}
 };
@@ -1378,6 +1445,10 @@ function apply(ctx, config) {
 			// Copy the declared fields only: the helper's own bookkeeping keys
 			// must not reach the closed output schema.
 			const value = { action: args.action };
+			// A one-element list can still arrive unrolled from the helper; the output
+			// schema wants an array, and a string here is a rejected result, not a bug the
+			// caller can see.
+			if (typeof acted.controls === 'string') acted.controls = [acted.controls];
 			for (const field of ['cursor', 'foregroundTitle', 'foregroundClass', 'foregroundProcess', 'notes', 'windowCount', 'actedAt', 'controls', 'controlCount', 'controlsTruncated', 'controlsWindow']) {
 				if (acted[field] !== undefined) value[field] = acted[field];
 			}

package/lib/scripts/_dsh-uia.ps1

@@ -124,7 +124,11 @@ function Format-DshUiaNodes {
         $lines.Add(('{0} [{1}] ''{2}'' @ {3}{4}' -f $n.index, $n.role, $name, $rect, $tail))
     }
     if ($Truncated) { $lines.Add(('... truncated at the node cap ({0} elements in total)' -f $Total)) }
-    return $lines.ToArray()
+    # The unary comma stops PowerShell unrolling a one-element array on return:
+    # without it a window that exposes a single control comes back as a bare
+    # string, and the result then fails its own schema ("controls must be an
+    # array"). Measured on a LibreOffice window that publishes exactly one.
+    return ,($lines.ToArray())
 }
 
 <# Resolve an index back to an element by re-reading the same listing. #>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@momasiku/dsh-pilot",
-  "version": "0.5.12",
+  "version": "0.5.16",
   "description": "Desktop vision and control for DeepSeek Harness - the whole Windows machine, any application, not browser automation: screen_view captures the live screen as an image the model can see, desktop_control drives the mouse, keyboard, and windows against the exact same pixel space; desktop_sequence runs a whole batch of actions in one call and looks once at the end.",
   "license": "MIT",
   "author": "momasiku",