dsh-click 0.3.25 → 0.3.26
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README-es.md +1 -1
- package/README-hi.md +1 -1
- package/README-pt.md +1 -1
- package/README-zh.md +1 -1
- package/README.md +1 -1
- package/cordis.patch.yml +1 -1
- package/lib/index.js +68 -51
- package/lib/types/actions.d.ts +1 -1
- package/lib/types/actions.d.ts.map +1 -1
- package/lib/types/config.d.ts +7 -7
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/observe.d.ts +10 -3
- package/lib/types/observe.d.ts.map +1 -1
- package/lib/types/version.d.ts +1 -1
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.3.26] - 2026-10-27
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
|
|
7
|
+
- **Element-targeted actions now verify the target instead of the whole window (issue #29).** `screen_read` stores a per-element hash; `click`/`type`/`scroll` re-read the tree and allow the action when the addressed element is unchanged, even if an unrelated clock, cursor, spinner, or memory label moved the whole-tree hash. `staleCheckTree` gains `target` and becomes the default; `strict` preserves the old whole-window refusal, and `off` remains the explicit escape hatch. Screenshot-only observations and `key` fall back to the strict whole-tree check. Added unchanged-target and changed-target regression tests.
|
|
8
|
+
|
|
3
9
|
## [0.3.25] - 2026-10-26
|
|
4
10
|
|
|
5
11
|
### Added
|
package/README-es.md
CHANGED
|
@@ -183,7 +183,7 @@ Todos los ajustes son campos `Config` de Schemastery (modificables desde cordis.
|
|
|
183
183
|
| `maxHelperOutputBytes` | `25165824` | Límite de una respuesta del helper en bytes (1024..67108864) |
|
|
184
184
|
| `maxScreenshotSide` | `2560` | Lado más largo de la captura en píxeles (320..7680); las mayores se reducen |
|
|
185
185
|
| `staleCheckPixels` | `true` | Comparar un hash de píxeles fresco antes de cada acción y rechazar si cambió |
|
|
186
|
-
| `staleCheckTree` | `
|
|
186
|
+
| `staleCheckTree` | `target` | Frescura del árbol: `target` verifica el elemento destino (por defecto), `strict` rechaza cualquier nodo cambiado, `off` omite el hash del árbol |
|
|
187
187
|
| `maxObservationAgeMs` | `30000` | Antigüedad máxima en ms de una observación que una acción puede citar (1000..600000) |
|
|
188
188
|
| `maxCachedObservations` | `8` | Límite LRU de observaciones en caché (1..64) |
|
|
189
189
|
| `maxElements` | `500` | Límite de elementos de accesibilidad por `screen_read` (1..2000) |
|
package/README-hi.md
CHANGED
|
@@ -183,7 +183,7 @@ Error: dsh-click backend unavailable on linux: Linux backend is reserved but not
|
|
|
183
183
|
| `maxHelperOutputBytes` | `25165824` | एक हेल्पर प्रतिक्रिया की बाइट सीमा (1024..67108864) |
|
|
184
184
|
| `maxScreenshotSide` | `2560` | कैप्चर की सबसे लंबी भुजा पिक्सेल में (320..7680); बड़ी कैप्चर छोटी की जाती हैं |
|
|
185
185
|
| `staleCheckPixels` | `true` | हर क्रिया से पहले ताज़ा पिक्सेल हैश मिलाएँ और बदलाव पर अस्वीकार करें |
|
|
186
|
-
| `staleCheckTree` | `
|
|
186
|
+
| `staleCheckTree` | `target` | ट्री ताज़गी: `target` लक्ष्य तत्व जाँचता है (डिफ़ॉल्ट), `strict` किसी भी बदले नोड पर अस्वीकार, `off` ट्री हैश छोड़ता है |
|
|
187
187
|
| `maxObservationAgeMs` | `30000` | किसी अवलोकन की अधिकतम आयु ms में जिसे कोई क्रिया उद्धृत कर सकती है (1000..600000) |
|
|
188
188
|
| `maxCachedObservations` | `8` | कैश किए गए अवलोकनों की LRU सीमा (1..64) |
|
|
189
189
|
| `maxElements` | `500` | प्रति `screen_read` एक्सेसिबिलिटी एलिमेंट की सीमा (1..2000) |
|
package/README-pt.md
CHANGED
|
@@ -183,7 +183,7 @@ Todos os ajustes são campos `Config` do Schemastery (alteráveis pelo cordis.ym
|
|
|
183
183
|
| `maxHelperOutputBytes` | `25165824` | Limite de uma resposta do helper em bytes (1024..67108864) |
|
|
184
184
|
| `maxScreenshotSide` | `2560` | Maior lado da captura em pixels (320..7680); capturas maiores são reduzidas |
|
|
185
185
|
| `staleCheckPixels` | `true` | Comparar um hash de pixels novo antes de cada ação e recusar se mudou |
|
|
186
|
-
| `staleCheckTree` | `
|
|
186
|
+
| `staleCheckTree` | `target` | Frescor da árvore: `target` verifica o elemento alvo (padrão), `strict` recusa qualquer nó alterado, `off` ignora o hash da árvore |
|
|
187
187
|
| `maxObservationAgeMs` | `30000` | Idade máxima em ms de uma observação que uma ação pode citar (1000..600000) |
|
|
188
188
|
| `maxCachedObservations` | `8` | Limite LRU de observações em cache (1..64) |
|
|
189
189
|
| `maxElements` | `500` | Limite de elementos de acessibilidade por `screen_read` (1..2000) |
|
package/README-zh.md
CHANGED
|
@@ -183,7 +183,7 @@ Error: dsh-click backend unavailable on linux: Linux backend is reserved but not
|
|
|
183
183
|
| `maxHelperOutputBytes` | `25165824` | 单次 helper 响应的字节上限(1024..67108864) |
|
|
184
184
|
| `maxScreenshotSide` | `2560` | 截图最长边像素(320..7680);超出即缩放 |
|
|
185
185
|
| `staleCheckPixels` | `true` | 每次操作前对比新像素哈希,变化即拒绝 |
|
|
186
|
-
| `staleCheckTree` | `
|
|
186
|
+
| `staleCheckTree` | `target` | 控件树新鲜度:`target` 校验目标元素(默认),`strict` 任一节点变化即拒绝,`off` 跳过整树哈希(身份/年龄/像素检查仍生效) |
|
|
187
187
|
| `maxObservationAgeMs` | `30000` | 操作可引用观察的最大时限(毫秒,1000..600000) |
|
|
188
188
|
| `maxCachedObservations` | `8` | 观察缓存 LRU 上限(1..64) |
|
|
189
189
|
| `maxElements` | `500` | 每次 `screen_read` 的无障碍元素上限(1..2000) |
|
package/README.md
CHANGED
|
@@ -183,7 +183,7 @@ All tunables are Schemastery `Config` fields (changeable from cordis.yml). An id
|
|
|
183
183
|
| `maxHelperOutputBytes` | `25165824` | Cap on one helper response in bytes (1024..67108864) |
|
|
184
184
|
| `maxScreenshotSide` | `2560` | Longest screenshot side in pixels (320..7680); larger captures are downscaled |
|
|
185
185
|
| `staleCheckPixels` | `true` | Compare a fresh pixel hash before every action and refuse on change |
|
|
186
|
-
| `staleCheckTree` | `
|
|
186
|
+
| `staleCheckTree` | `target` | Accessibility-tree freshness: `target` verifies the addressed element (default), `strict` refuses any changed node, `off` skips the tree hash (identity/age/pixel checks still apply) |
|
|
187
187
|
| `maxObservationAgeMs` | `30000` | Maximum age in ms of an observation an action may cite (1000..600000) |
|
|
188
188
|
| `maxCachedObservations` | `8` | LRU cap on cached observations (1..64) |
|
|
189
189
|
| `maxElements` | `500` | Cap on accessibility elements per `screen_read` (1..2000) |
|
package/cordis.patch.yml
CHANGED
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
# 'strict' refuses any changed tree node; 'off' skips the whole-tree
|
|
38
38
|
# hash for windows with a ticking clock/cursor (identity/age/pixel
|
|
39
39
|
# checks still apply).
|
|
40
|
-
staleCheckTree:
|
|
40
|
+
staleCheckTree: target
|
|
41
41
|
# Maximum age in ms of an observation that an action may still base
|
|
42
42
|
# on (1000..600000). Older observations are refused and must be
|
|
43
43
|
# re-observed.
|
package/lib/index.js
CHANGED
|
@@ -7,7 +7,7 @@ import { fileURLToPath } from "node:url";
|
|
|
7
7
|
import { defineTool } from "@deepseek-ai/dsh-tools";
|
|
8
8
|
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
9
9
|
import { tmpdir } from "node:os";
|
|
10
|
-
|
|
10
|
+
// src/events.ts
|
|
11
11
|
/** The observation audit event type. */
|
|
12
12
|
const OBSERVED_EVENT = "dsh-click/observed";
|
|
13
13
|
/** The action audit event type. */
|
|
@@ -36,8 +36,7 @@ function appendAuditEvent(session, type, data) {
|
|
|
36
36
|
else session.append(type, data);
|
|
37
37
|
return "appended";
|
|
38
38
|
}
|
|
39
|
-
|
|
40
|
-
//#region src/sanitize.ts
|
|
39
|
+
// src/sanitize.ts
|
|
41
40
|
/**
|
|
42
41
|
* Pure display/audit redaction for everything dsh-click shows to the model or
|
|
43
42
|
* writes to a session log: window titles, executable paths, and free text that
|
|
@@ -104,8 +103,7 @@ function redactSensitive(text) {
|
|
|
104
103
|
function sanitizeVisible(text, maxLength) {
|
|
105
104
|
return sanitizeText(redactSensitive(text), maxLength);
|
|
106
105
|
}
|
|
107
|
-
|
|
108
|
-
//#region src/platform/types.ts
|
|
106
|
+
// src/platform/types.ts
|
|
109
107
|
/** Error with a stable machine-routing code, model-readable message included. */
|
|
110
108
|
var DshClickError = class extends Error {
|
|
111
109
|
code;
|
|
@@ -146,8 +144,7 @@ function expectProcessFacts(value, op) {
|
|
|
146
144
|
executablePath
|
|
147
145
|
};
|
|
148
146
|
}
|
|
149
|
-
|
|
150
|
-
//#region src/actions.ts
|
|
147
|
+
// src/actions.ts
|
|
151
148
|
/** Approval failure detail, model-readable. */
|
|
152
149
|
const APPROVAL_DETAIL = {
|
|
153
150
|
rejected: "rejected by the approval answerer",
|
|
@@ -192,12 +189,21 @@ var ActionExecutor = class {
|
|
|
192
189
|
* @param signal - cancellation.
|
|
193
190
|
* @returns the cited record and the fresh snapshot.
|
|
194
191
|
*/
|
|
195
|
-
async requireFreshWindow(observationId, windowId, signal) {
|
|
192
|
+
async requireFreshWindow(observationId, windowId, signal, targetRef) {
|
|
196
193
|
const record = this.deps.observations.get(observationId);
|
|
197
194
|
if (record === void 0) throw new DshClickError(`unknown observation "${observationId}" — run screen_read or screen_shot again and cite the returned observationId`, "UNKNOWN_OBSERVATION");
|
|
198
195
|
if (record.windowId !== windowId) throw new DshClickError(`observation ${observationId} belongs to window ${record.windowId}, not ${windowId}`, "UNKNOWN_OBSERVATION");
|
|
199
|
-
|
|
200
|
-
|
|
196
|
+
let fresh = await this.deps.backend.snapshot(windowId, signal);
|
|
197
|
+
let targetElements;
|
|
198
|
+
if (targetRef !== void 0 && record.elementHashes !== void 0 && this.config.staleCheckTree === "target") {
|
|
199
|
+
const tree = await this.deps.backend.tree({ windowId }, this.config.maxElements, this.config.maxTreeDepth, false, signal);
|
|
200
|
+
fresh = tree.snapshot;
|
|
201
|
+
targetElements = tree.elements;
|
|
202
|
+
}
|
|
203
|
+
const verdict = this.deps.observations.verify(record, fresh, Date.now(), targetRef === void 0 ? void 0 : {
|
|
204
|
+
ref: targetRef,
|
|
205
|
+
elements: targetElements ?? []
|
|
206
|
+
});
|
|
201
207
|
if (!verdict.ok) throw new DshClickError(`${verdict.detail} — run screen_read or screen_shot again before acting`, verdict.code);
|
|
202
208
|
return {
|
|
203
209
|
record,
|
|
@@ -253,12 +259,12 @@ var ActionExecutor = class {
|
|
|
253
259
|
* @param run - the backend call (receives the resolved focus flag).
|
|
254
260
|
* @returns the validated backend outcome.
|
|
255
261
|
*/
|
|
256
|
-
async perform(toolName, exec, observationId, windowId, run) {
|
|
262
|
+
async perform(toolName, exec, observationId, windowId, run, targetRef) {
|
|
257
263
|
let approved = "none";
|
|
258
264
|
let observationIdAudited;
|
|
259
265
|
let windowIdAudited;
|
|
260
266
|
try {
|
|
261
|
-
const { record } = await this.requireFreshWindow(observationId, windowId, exec.signal);
|
|
267
|
+
const { record } = await this.requireFreshWindow(observationId, windowId, exec.signal, targetRef);
|
|
262
268
|
observationIdAudited = observationId;
|
|
263
269
|
windowIdAudited = windowId;
|
|
264
270
|
approved = await this.gate(exec, toolName, {
|
|
@@ -334,8 +340,7 @@ var ActionExecutor = class {
|
|
|
334
340
|
}
|
|
335
341
|
}
|
|
336
342
|
};
|
|
337
|
-
|
|
338
|
-
//#region src/config.ts
|
|
343
|
+
// src/config.ts
|
|
339
344
|
/**
|
|
340
345
|
* Plugin configuration and its explicit resolve step. `resolveConfig` re-judges
|
|
341
346
|
* every default and bound so programmatic construction that bypasses
|
|
@@ -375,7 +380,11 @@ const Config = z.object({
|
|
|
375
380
|
maxHelperOutputBytes: z.number().min(1024).max(MAX_HELPER_OUTPUT_BYTES).default(DEFAULT_MAX_HELPER_OUTPUT_BYTES),
|
|
376
381
|
maxScreenshotSide: z.number().min(320).max(MAX_SCREENSHOT_SIDE).default(DEFAULT_MAX_SCREENSHOT_SIDE),
|
|
377
382
|
staleCheckPixels: z.boolean().default(true),
|
|
378
|
-
staleCheckTree: z.union([
|
|
383
|
+
staleCheckTree: z.union([
|
|
384
|
+
"strict",
|
|
385
|
+
"target",
|
|
386
|
+
"off"
|
|
387
|
+
]).default("target"),
|
|
379
388
|
maxObservationAgeMs: z.number().min(MIN_OBSERVATION_AGE_MS).max(MAX_OBSERVATION_AGE_MS).default(DEFAULT_MAX_OBSERVATION_AGE_MS),
|
|
380
389
|
maxCachedObservations: z.number().min(1).max(64).default(8),
|
|
381
390
|
maxElements: z.number().min(1).max(MAX_ELEMENTS).default(500),
|
|
@@ -424,8 +433,8 @@ function resolveConfig(config) {
|
|
|
424
433
|
if (!Number.isInteger(maxScreenshotSide) || maxScreenshotSide < 320 || maxScreenshotSide > 7680) invalid("maxScreenshotSide", `must be an integer between 320 and ${MAX_SCREENSHOT_SIDE}`);
|
|
425
434
|
const staleCheckPixels = config?.staleCheckPixels ?? true;
|
|
426
435
|
if (typeof staleCheckPixels !== "boolean") invalid("staleCheckPixels", "must be a boolean");
|
|
427
|
-
const staleCheckTree = config?.staleCheckTree ?? "
|
|
428
|
-
if (staleCheckTree !== "strict" && staleCheckTree !== "off") invalid("staleCheckTree", "must be \"strict\" or \"off\"");
|
|
436
|
+
const staleCheckTree = config?.staleCheckTree ?? "target";
|
|
437
|
+
if (staleCheckTree !== "strict" && staleCheckTree !== "target" && staleCheckTree !== "off") invalid("staleCheckTree", "must be \"strict\", \"target\", or \"off\"");
|
|
429
438
|
const maxObservationAgeMs = config?.maxObservationAgeMs ?? 3e4;
|
|
430
439
|
if (!Number.isFinite(maxObservationAgeMs) || maxObservationAgeMs < 1e3 || maxObservationAgeMs > 6e5) invalid("maxObservationAgeMs", `must be a finite number between ${MIN_OBSERVATION_AGE_MS} and ${MAX_OBSERVATION_AGE_MS}`);
|
|
431
440
|
const maxCachedObservations = config?.maxCachedObservations ?? 8;
|
|
@@ -468,8 +477,7 @@ function resolveConfig(config) {
|
|
|
468
477
|
})
|
|
469
478
|
});
|
|
470
479
|
}
|
|
471
|
-
|
|
472
|
-
//#region src/observe.ts
|
|
480
|
+
// src/observe.ts
|
|
473
481
|
/**
|
|
474
482
|
* Observation cache and the staleness boundary. Every observation stores the
|
|
475
483
|
* window's identity facts plus content hashes; before any action the executor
|
|
@@ -498,6 +506,18 @@ function observationIdOf(snapshot) {
|
|
|
498
506
|
]);
|
|
499
507
|
return createHash("sha256").update(material).digest("hex").slice(0, 32);
|
|
500
508
|
}
|
|
509
|
+
/** Hash one captured element's identity-relevant fields. */
|
|
510
|
+
function elementHashOf(element) {
|
|
511
|
+
return createHash("sha256").update(JSON.stringify([
|
|
512
|
+
element.elementId,
|
|
513
|
+
element.controlType,
|
|
514
|
+
element.name,
|
|
515
|
+
element.automationId,
|
|
516
|
+
element.rect,
|
|
517
|
+
element.enabled,
|
|
518
|
+
element.patterns
|
|
519
|
+
])).digest("hex");
|
|
520
|
+
}
|
|
501
521
|
/** Bounded LRU cache of observations, plus the staleness verdict logic. */
|
|
502
522
|
var ObservationStore = class {
|
|
503
523
|
config;
|
|
@@ -512,7 +532,7 @@ var ObservationStore = class {
|
|
|
512
532
|
* @param snapshot - the snapshot facts it came from.
|
|
513
533
|
* @returns the stored record (id = hash of the state).
|
|
514
534
|
*/
|
|
515
|
-
record(snapshot) {
|
|
535
|
+
record(snapshot, elements) {
|
|
516
536
|
const record = {
|
|
517
537
|
id: observationIdOf(snapshot),
|
|
518
538
|
windowId: snapshot.windowId,
|
|
@@ -523,6 +543,7 @@ var ObservationStore = class {
|
|
|
523
543
|
rect: { ...snapshot.rect },
|
|
524
544
|
treeHash: snapshot.treeHash,
|
|
525
545
|
shotHash: snapshot.shotHash,
|
|
546
|
+
...elements === void 0 ? {} : { elementHashes: Object.fromEntries(elements.map((element) => [element.elementId, elementHashOf(element)])) },
|
|
526
547
|
elementCount: snapshot.elementCount,
|
|
527
548
|
observedAt: Date.now()
|
|
528
549
|
};
|
|
@@ -556,7 +577,7 @@ var ObservationStore = class {
|
|
|
556
577
|
* @param now - wall-clock now (injectable for tests).
|
|
557
578
|
* @returns the verdict.
|
|
558
579
|
*/
|
|
559
|
-
verify(record, fresh, now = Date.now()) {
|
|
580
|
+
verify(record, fresh, now = Date.now(), target) {
|
|
560
581
|
if (now - record.observedAt > this.config.maxObservationAgeMs) return {
|
|
561
582
|
ok: false,
|
|
562
583
|
code: "EXPIRED",
|
|
@@ -567,12 +588,18 @@ var ObservationStore = class {
|
|
|
567
588
|
code: "STALE_IDENTITY",
|
|
568
589
|
detail: `window ${record.windowId} no longer matches observation ${record.id} (title/class/process or rect changed)`
|
|
569
590
|
};
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
591
|
+
let scopedTargetUnchanged = false;
|
|
592
|
+
if (this.config.staleCheckTree !== "off" && fresh.treeHash !== record.treeHash) {
|
|
593
|
+
const expected = target === void 0 || record.elementHashes === void 0 ? void 0 : record.elementHashes[target.ref];
|
|
594
|
+
const current = target?.elements.find((element) => element.elementId === target.ref);
|
|
595
|
+
if (this.config.staleCheckTree === "target" && expected !== void 0 && current !== void 0 && elementHashOf(current) === expected) scopedTargetUnchanged = true;
|
|
596
|
+
else return {
|
|
597
|
+
ok: false,
|
|
598
|
+
code: "STALE_TREE",
|
|
599
|
+
detail: target === void 0 ? `the accessibility tree of window ${record.windowId} changed since observation ${record.id}` : `target element ${target.ref} changed or disappeared since observation ${record.id}`
|
|
600
|
+
};
|
|
601
|
+
}
|
|
602
|
+
if (!scopedTargetUnchanged && this.config.staleCheckPixels && fresh.shotHash !== record.shotHash) return {
|
|
576
603
|
ok: false,
|
|
577
604
|
code: "STALE_PIXELS",
|
|
578
605
|
detail: `the pixels of window ${record.windowId} changed since observation ${record.id}`
|
|
@@ -580,8 +607,7 @@ var ObservationStore = class {
|
|
|
580
607
|
return { ok: true };
|
|
581
608
|
}
|
|
582
609
|
};
|
|
583
|
-
|
|
584
|
-
//#region src/version.ts
|
|
610
|
+
// src/version.ts
|
|
585
611
|
/**
|
|
586
612
|
* The plugin version, hardcoded on purpose: it rides the helper-process
|
|
587
613
|
* protocol handshake, so it must track `package.json` or released helpers
|
|
@@ -591,11 +617,10 @@ var ObservationStore = class {
|
|
|
591
617
|
* @module dsh-click/version
|
|
592
618
|
*/
|
|
593
619
|
/** Plugin version advertised in the helper-protocol handshake. */
|
|
594
|
-
const VERSION = "0.3.
|
|
620
|
+
const VERSION = "0.3.26";
|
|
595
621
|
/** The helper protocol revision; bump only on incompatible wire changes. */
|
|
596
622
|
const HELPER_PROTOCOL_VERSION = 1;
|
|
597
|
-
|
|
598
|
-
//#region src/platform/runner.ts
|
|
623
|
+
// src/platform/runner.ts
|
|
599
624
|
/**
|
|
600
625
|
* The Windows backend: every operation spawns the native PowerShell helper
|
|
601
626
|
* (`native/win32/dsh-click-helper.ps1`) through the harness's `ctx.subprocess`
|
|
@@ -879,8 +904,7 @@ var HelperBackend = class {
|
|
|
879
904
|
};
|
|
880
905
|
}
|
|
881
906
|
};
|
|
882
|
-
|
|
883
|
-
//#region src/platform/selection.ts
|
|
907
|
+
// src/platform/selection.ts
|
|
884
908
|
/** A backend that cannot act: every operation fails closed with the given reason. */
|
|
885
909
|
var UnavailableBackend = class {
|
|
886
910
|
platform;
|
|
@@ -951,8 +975,7 @@ function createBackend(ctx, config) {
|
|
|
951
975
|
}
|
|
952
976
|
return new UnavailableBackend(process.platform, RESERVED_BACKENDS[process.platform] ?? "unsupported platform");
|
|
953
977
|
}
|
|
954
|
-
|
|
955
|
-
//#region src/vision.ts
|
|
978
|
+
// src/vision.ts
|
|
956
979
|
/**
|
|
957
980
|
* Resolve whether the calling agent's current model declares image input.
|
|
958
981
|
*
|
|
@@ -974,8 +997,7 @@ async function sessionAcceptsImages(agent, ctx, signal) {
|
|
|
974
997
|
return false;
|
|
975
998
|
}
|
|
976
999
|
}
|
|
977
|
-
|
|
978
|
-
//#region src/vision/ocr.ts
|
|
1000
|
+
// src/vision/ocr.ts
|
|
979
1001
|
/**
|
|
980
1002
|
* Optical character recognition seam for windows without a UIA tree. OCR is
|
|
981
1003
|
* optional and probed at mount: the `tesseract` provider is used only when the
|
|
@@ -1120,8 +1142,7 @@ async function probeTesseract(subprocess, command, language) {
|
|
|
1120
1142
|
return new UnavailableOcrProvider("tesseract", `tesseract (${command}) is not installed`);
|
|
1121
1143
|
}
|
|
1122
1144
|
}
|
|
1123
|
-
|
|
1124
|
-
//#region src/vision/grounding.ts
|
|
1145
|
+
// src/vision/grounding.ts
|
|
1125
1146
|
/**
|
|
1126
1147
|
* Visual grounding seam: turning a screenshot plus a natural-language prompt
|
|
1127
1148
|
* into labeled bounding boxes. Kept as an independent provider so a vision
|
|
@@ -1149,8 +1170,7 @@ var UnavailableGroundingProvider = class {
|
|
|
1149
1170
|
return Promise.reject(new DshClickError(`visual grounding unavailable: ${this.reason}`, "GROUNDING_UNAVAILABLE"));
|
|
1150
1171
|
}
|
|
1151
1172
|
};
|
|
1152
|
-
|
|
1153
|
-
//#region src/vision/index.ts
|
|
1173
|
+
// src/vision/index.ts
|
|
1154
1174
|
/**
|
|
1155
1175
|
* Map one screenshot box (possibly downscaled by `maxScreenshotSide`) to its
|
|
1156
1176
|
* screen-space box, centering on the window's screen rect origin.
|
|
@@ -1177,8 +1197,7 @@ function centerOf(box) {
|
|
|
1177
1197
|
y: Math.round(box.y + box.height / 2)
|
|
1178
1198
|
};
|
|
1179
1199
|
}
|
|
1180
|
-
|
|
1181
|
-
//#region src/tools.ts
|
|
1200
|
+
// src/tools.ts
|
|
1182
1201
|
/** Sanitize one snapshot into the model-visible window value. */
|
|
1183
1202
|
function observedWindow(snapshot, maxTextLength) {
|
|
1184
1203
|
return {
|
|
@@ -1533,7 +1552,7 @@ function screenReadTool(services) {
|
|
|
1533
1552
|
const parsed = args;
|
|
1534
1553
|
const tree = await backend.tree(parsed.target ?? {}, config.maxElements, config.maxTreeDepth, parsed.includePixels ?? true, exec.signal);
|
|
1535
1554
|
const sanitized = observedWindow(tree.snapshot, config.maxTextLength);
|
|
1536
|
-
const record = observations.record(tree.snapshot);
|
|
1555
|
+
const record = observations.record(tree.snapshot, tree.elements);
|
|
1537
1556
|
auditObservation(exec, {
|
|
1538
1557
|
observationId: record.id,
|
|
1539
1558
|
windowId: tree.snapshot.windowId,
|
|
@@ -1858,7 +1877,7 @@ function clickTool(services) {
|
|
|
1858
1877
|
y: target.y
|
|
1859
1878
|
},
|
|
1860
1879
|
button: parsed.button ?? "left"
|
|
1861
|
-
}, focusFallback, exec.signal));
|
|
1880
|
+
}, focusFallback, exec.signal), byElement ? target.elementId : void 0);
|
|
1862
1881
|
return {
|
|
1863
1882
|
ok: true,
|
|
1864
1883
|
windowId: outcome.windowId,
|
|
@@ -1917,7 +1936,7 @@ function typeTool(services) {
|
|
|
1917
1936
|
elementId: parsed.elementId,
|
|
1918
1937
|
text: parsed.text,
|
|
1919
1938
|
rollback: config.rollbackEnabled
|
|
1920
|
-
}, focusFallback, exec.signal));
|
|
1939
|
+
}, focusFallback, exec.signal), parsed.elementId);
|
|
1921
1940
|
return {
|
|
1922
1941
|
ok: true,
|
|
1923
1942
|
windowId: outcome.windowId,
|
|
@@ -1986,7 +2005,7 @@ function scrollTool(services) {
|
|
|
1986
2005
|
...parsed.elementId !== void 0 ? { elementId: parsed.elementId } : {},
|
|
1987
2006
|
direction: parsed.direction,
|
|
1988
2007
|
amount
|
|
1989
|
-
}, focusFallback, exec.signal));
|
|
2008
|
+
}, focusFallback, exec.signal), parsed.elementId);
|
|
1990
2009
|
return {
|
|
1991
2010
|
ok: true,
|
|
1992
2011
|
windowId: outcome.windowId,
|
|
@@ -2214,8 +2233,7 @@ function allTools(services) {
|
|
|
2214
2233
|
appLaunchTool(services)
|
|
2215
2234
|
];
|
|
2216
2235
|
}
|
|
2217
|
-
|
|
2218
|
-
//#region src/index.ts
|
|
2236
|
+
// src/index.ts
|
|
2219
2237
|
const name = "dsh-click";
|
|
2220
2238
|
/** Hard services: the tool registry every contribution lands in, plus the subprocess runtime the Windows helper spawns through. */
|
|
2221
2239
|
const inject = ["tools", "subprocess"];
|
|
@@ -2250,5 +2268,4 @@ async function apply(ctx, config) {
|
|
|
2250
2268
|
};
|
|
2251
2269
|
for (const tool of allTools(services)) ctx.effect(() => ctx.tools.register(tool), `dsh-click: ${tool.name} tool`);
|
|
2252
2270
|
}
|
|
2253
|
-
//#endregion
|
|
2254
2271
|
export { ACTION_EVENT, ActionExecutor, Config, DshClickError, HELPER_PROTOCOL_VERSION, HelperBackend, OBSERVED_EVENT, ObservationStore, TesseractOcrProvider, UnavailableBackend, UnavailableGroundingProvider, UnavailableOcrProvider, VERSION, allTools, apply, centerOf, createBackend, inject, name, observationIdOf, parseTesseractTsv, probeTesseract, redactSensitive, resolveConfig, sanitizePath, sanitizeText, sanitizeVisible, screenBoxFor, tesseractArgv };
|
package/lib/types/actions.d.ts
CHANGED
|
@@ -80,7 +80,7 @@ export declare class ActionExecutor {
|
|
|
80
80
|
* @param run - the backend call (receives the resolved focus flag).
|
|
81
81
|
* @returns the validated backend outcome.
|
|
82
82
|
*/
|
|
83
|
-
perform(toolName: string, exec: ToolRunContext, observationId: string, windowId: number, run: (focusFallback: boolean) => Promise<ActionOutcome
|
|
83
|
+
perform(toolName: string, exec: ToolRunContext, observationId: string, windowId: number, run: (focusFallback: boolean) => Promise<ActionOutcome>, targetRef?: string): Promise<ActionOutcome>;
|
|
84
84
|
/**
|
|
85
85
|
* Gate and launch one application. No window observation exists yet, so the
|
|
86
86
|
* gate runs against the requested name/path alone and the launch outcome's
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"actions.d.ts","sourceRoot":"","sources":["../../src/actions.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAClD,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAA;AAE5D,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAA0B,MAAM,cAAc,CAAA;AAEvE,OAAO,EAAiB,KAAK,aAAa,EAAE,KAAK,cAAc,EAAE,KAAK,aAAa,EAAuB,MAAM,qBAAqB,CAAA;AAErI,yDAAyD;AACzD,MAAM,MAAM,YAAY,GAAG,UAAU,GAAG,WAAW,GAAG,MAAM,CAAA;AAS5D,iEAAiE;AACjE,MAAM,WAAW,kBAAkB;IACjC,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;CACxC;AAED,yEAAyE;AACzE,qBAAa,cAAc;IAIb,OAAO,CAAC,QAAQ,CAAC,IAAI;IAHjC,+DAA+D;IAC/D,OAAO,CAAC,eAAe,CAAQ;IAC/B,2CAA2C;gBACd,IAAI,EAAE,kBAAkB;IAErD,4BAA4B;IAC5B,OAAO,KAAK,GAAG,GAEd;IAED,2BAA2B;IAC3B,OAAO,KAAK,MAAM,GAEjB;IAED,gFAAgF;IAChF,OAAO,CAAC,KAAK;IAkBb;;;;;;;;OAQG;YACW,kBAAkB;
|
|
1
|
+
{"version":3,"file":"actions.d.ts","sourceRoot":"","sources":["../../src/actions.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAEH,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAA;AAClD,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAA;AAE5D,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAEjD,OAAO,EAAE,gBAAgB,EAA0B,MAAM,cAAc,CAAA;AAEvE,OAAO,EAAiB,KAAK,aAAa,EAAE,KAAK,cAAc,EAAE,KAAK,aAAa,EAAuB,MAAM,qBAAqB,CAAA;AAErI,yDAAyD;AACzD,MAAM,MAAM,YAAY,GAAG,UAAU,GAAG,WAAW,GAAG,MAAM,CAAA;AAS5D,iEAAiE;AACjE,MAAM,WAAW,kBAAkB;IACjC,4BAA4B;IAC5B,QAAQ,CAAC,GAAG,EAAE,OAAO,CAAA;IACrB,8BAA8B;IAC9B,QAAQ,CAAC,MAAM,EAAE,cAAc,CAAA;IAC/B,kCAAkC;IAClC,QAAQ,CAAC,OAAO,EAAE,cAAc,CAAA;IAChC,6BAA6B;IAC7B,QAAQ,CAAC,YAAY,EAAE,gBAAgB,CAAA;CACxC;AAED,yEAAyE;AACzE,qBAAa,cAAc;IAIb,OAAO,CAAC,QAAQ,CAAC,IAAI;IAHjC,+DAA+D;IAC/D,OAAO,CAAC,eAAe,CAAQ;IAC/B,2CAA2C;gBACd,IAAI,EAAE,kBAAkB;IAErD,4BAA4B;IAC5B,OAAO,KAAK,GAAG,GAEd;IAED,2BAA2B;IAC3B,OAAO,KAAK,MAAM,GAEjB;IAED,gFAAgF;IAChF,OAAO,CAAC,KAAK;IAkBb;;;;;;;;OAQG;YACW,kBAAkB;IAiChC;;;;;;;;;;;OAWG;YACW,IAAI;IAuClB;;;;;OAKG;IACH,OAAO,CAAC,sBAAsB;IAS9B;;;;;;;;;;OAUG;IACG,OAAO,CACX,QAAQ,EAAE,MAAM,EAChB,IAAI,EAAE,cAAc,EACpB,aAAa,EAAE,MAAM,EACrB,QAAQ,EAAE,MAAM,EAChB,GAAG,EAAE,CAAC,aAAa,EAAE,OAAO,KAAK,OAAO,CAAC,aAAa,CAAC,EACvD,SAAS,CAAC,EAAE,MAAM,GACjB,OAAO,CAAC,aAAa,CAAC;IAwCzB;;;;;;;;;OASG;IACG,MAAM,CAAC,IAAI,EAAE,cAAc,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,aAAa,CAAC;CAoBlG"}
|
package/lib/types/config.d.ts
CHANGED
|
@@ -86,13 +86,13 @@ export interface Config {
|
|
|
86
86
|
/** Compare a fresh pixel hash before every action (default true; the stale-state boundary). */
|
|
87
87
|
staleCheckPixels?: boolean;
|
|
88
88
|
/**
|
|
89
|
-
*
|
|
90
|
-
*
|
|
91
|
-
*
|
|
92
|
-
*
|
|
93
|
-
*
|
|
89
|
+
* Accessibility-tree staleness policy. `strict` refuses an action when any node in the
|
|
90
|
+
* window changed; `target` (default) lets `click`/`type`/`scroll` proceed when the
|
|
91
|
+
* addressed element itself is unchanged, even if an unrelated clock/cursor moved the
|
|
92
|
+
* whole-tree hash; `off` skips the tree hash entirely. Identity, observation-age, and
|
|
93
|
+
* (unless disabled) pixel checks still apply.
|
|
94
94
|
*/
|
|
95
|
-
staleCheckTree?: 'strict' | 'off';
|
|
95
|
+
staleCheckTree?: 'strict' | 'target' | 'off';
|
|
96
96
|
/** Maximum age in ms of an observation that an action may still base on (default 30000). */
|
|
97
97
|
maxObservationAgeMs?: number;
|
|
98
98
|
/** Cap on cached observations (default 8). */
|
|
@@ -129,7 +129,7 @@ export interface ResolvedConfig {
|
|
|
129
129
|
/** Whether the pixel-level stale check runs. */
|
|
130
130
|
staleCheckPixels: boolean;
|
|
131
131
|
/** Whole-window tree-staleness policy. */
|
|
132
|
-
staleCheckTree: 'strict' | 'off';
|
|
132
|
+
staleCheckTree: 'strict' | 'target' | 'off';
|
|
133
133
|
/** Maximum observation age in milliseconds. */
|
|
134
134
|
maxObservationAgeMs: number;
|
|
135
135
|
/** Cap on cached observations. */
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/config.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,CAAC,MAAM,0BAA0B,CAAA;AAExC,+EAA+E;AAC/E,MAAM,MAAM,SAAS,GAAG,MAAM,GAAG,MAAM,CAAA;AAEvC,mFAAmF;AACnF,MAAM,MAAM,aAAa,GAAG,OAAO,GAAG,OAAO,CAAA;AAE7C,uDAAuD;AACvD,eAAO,MAAM,yBAAyB,QAAS,CAAA;AAE/C,qFAAqF;AACrF,eAAO,MAAM,qBAAqB,SAAU,CAAA;AAE5C,sFAAsF;AACtF,eAAO,MAAM,+BAA+B,QAAmB,CAAA;AAE/D,sCAAsC;AACtC,eAAO,MAAM,uBAAuB,QAAmB,CAAA;AAEvD,2FAA2F;AAC3F,eAAO,MAAM,2BAA2B,OAAO,CAAA;AAE/C,sFAAsF;AACtF,eAAO,MAAM,mBAAmB,MAAM,CAAA;AACtC,eAAO,MAAM,mBAAmB,OAAO,CAAA;AAEvC,8EAA8E;AAC9E,eAAO,MAAM,8BAA8B,QAAS,CAAA;AAEpD,sCAAsC;AACtC,eAAO,MAAM,sBAAsB,OAAQ,CAAA;AAC3C,eAAO,MAAM,sBAAsB,SAAU,CAAA;AAE7C,4EAA4E;AAC5E,eAAO,MAAM,+BAA+B,IAAI,CAAA;AAEhD,sCAAsC;AACtC,eAAO,MAAM,uBAAuB,KAAK,CAAA;AAEzC,8EAA8E;AAC9E,eAAO,MAAM,oBAAoB,MAAM,CAAA;AAEvC,kCAAkC;AAClC,eAAO,MAAM,YAAY,OAAQ,CAAA;AAEjC,4DAA4D;AAC5D,eAAO,MAAM,sBAAsB,KAAK,CAAA;AAExC,iCAAiC;AACjC,eAAO,MAAM,cAAc,KAAK,CAAA;AAEhC,qEAAqE;AACrE,eAAO,MAAM,uBAAuB,MAAM,CAAA;AAE1C,qDAAqD;AACrD,eAAO,MAAM,eAAe,QAAS,CAAA;AAErC,8DAA8D;AAC9D,eAAO,MAAM,mBAAmB,cAAc,CAAA;AAE9C,sDAAsD;AACtD,eAAO,MAAM,oBAAoB,QAAQ,CAAA;AAEzC,iEAAiE;AACjE,MAAM,WAAW,SAAS;IACxB,kFAAkF;IAClF,OAAO,CAAC,EAAE,OAAO,CAAA;IACjB,uDAAuD;IACvD,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;CAClB;AAED,6DAA6D;AAC7D,MAAM,WAAW,MAAM;IACrB,oGAAoG;IACpG,eAAe,CAAC,EAAE,OAAO,CAAA;IACzB,sGAAsG;IACtG,kBAAkB,CAAC,EAAE,MAAM,EAAE,CAAA;IAC7B;;;;;;;OAOG;IACH,kBAAkB,CAAC,EAAE,OAAO,CAAA;IAC5B,8GAA8G;IAC9G,aAAa,CAAC,EAAE,aAAa,CAAA;IAC7B,iGAAiG;IACjG,SAAS,CAAC,EAAE,SAAS,CAAA;IACrB,+DAA+D;IAC/D,eAAe,CAAC,EAAE,MAAM,CAAA;IACxB,8DAA8D;IAC9D,oBAAoB,CAAC,EAAE,MAAM,CAAA;IAC7B,sEAAsE;IACtE,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAC1B,+FAA+F;IAC/F,gBAAgB,CAAC,EAAE,OAAO,CAAA;IAC1B;;;;;;OAMG;IACH,cAAc,CAAC,EAAE,QAAQ,GAAG,KAAK,CAAA;
|
|
1
|
+
{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/config.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,CAAC,MAAM,0BAA0B,CAAA;AAExC,+EAA+E;AAC/E,MAAM,MAAM,SAAS,GAAG,MAAM,GAAG,MAAM,CAAA;AAEvC,mFAAmF;AACnF,MAAM,MAAM,aAAa,GAAG,OAAO,GAAG,OAAO,CAAA;AAE7C,uDAAuD;AACvD,eAAO,MAAM,yBAAyB,QAAS,CAAA;AAE/C,qFAAqF;AACrF,eAAO,MAAM,qBAAqB,SAAU,CAAA;AAE5C,sFAAsF;AACtF,eAAO,MAAM,+BAA+B,QAAmB,CAAA;AAE/D,sCAAsC;AACtC,eAAO,MAAM,uBAAuB,QAAmB,CAAA;AAEvD,2FAA2F;AAC3F,eAAO,MAAM,2BAA2B,OAAO,CAAA;AAE/C,sFAAsF;AACtF,eAAO,MAAM,mBAAmB,MAAM,CAAA;AACtC,eAAO,MAAM,mBAAmB,OAAO,CAAA;AAEvC,8EAA8E;AAC9E,eAAO,MAAM,8BAA8B,QAAS,CAAA;AAEpD,sCAAsC;AACtC,eAAO,MAAM,sBAAsB,OAAQ,CAAA;AAC3C,eAAO,MAAM,sBAAsB,SAAU,CAAA;AAE7C,4EAA4E;AAC5E,eAAO,MAAM,+BAA+B,IAAI,CAAA;AAEhD,sCAAsC;AACtC,eAAO,MAAM,uBAAuB,KAAK,CAAA;AAEzC,8EAA8E;AAC9E,eAAO,MAAM,oBAAoB,MAAM,CAAA;AAEvC,kCAAkC;AAClC,eAAO,MAAM,YAAY,OAAQ,CAAA;AAEjC,4DAA4D;AAC5D,eAAO,MAAM,sBAAsB,KAAK,CAAA;AAExC,iCAAiC;AACjC,eAAO,MAAM,cAAc,KAAK,CAAA;AAEhC,qEAAqE;AACrE,eAAO,MAAM,uBAAuB,MAAM,CAAA;AAE1C,qDAAqD;AACrD,eAAO,MAAM,eAAe,QAAS,CAAA;AAErC,8DAA8D;AAC9D,eAAO,MAAM,mBAAmB,cAAc,CAAA;AAE9C,sDAAsD;AACtD,eAAO,MAAM,oBAAoB,QAAQ,CAAA;AAEzC,iEAAiE;AACjE,MAAM,WAAW,SAAS;IACxB,kFAAkF;IAClF,OAAO,CAAC,EAAE,OAAO,CAAA;IACjB,uDAAuD;IACvD,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB,8CAA8C;IAC9C,QAAQ,CAAC,EAAE,MAAM,CAAA;CAClB;AAED,6DAA6D;AAC7D,MAAM,WAAW,MAAM;IACrB,oGAAoG;IACpG,eAAe,CAAC,EAAE,OAAO,CAAA;IACzB,sGAAsG;IACtG,kBAAkB,CAAC,EAAE,MAAM,EAAE,CAAA;IAC7B;;;;;;;OAOG;IACH,kBAAkB,CAAC,EAAE,OAAO,CAAA;IAC5B,8GAA8G;IAC9G,aAAa,CAAC,EAAE,aAAa,CAAA;IAC7B,iGAAiG;IACjG,SAAS,CAAC,EAAE,SAAS,CAAA;IACrB,+DAA+D;IAC/D,eAAe,CAAC,EAAE,MAAM,CAAA;IACxB,8DAA8D;IAC9D,oBAAoB,CAAC,EAAE,MAAM,CAAA;IAC7B,sEAAsE;IACtE,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAC1B,+FAA+F;IAC/F,gBAAgB,CAAC,EAAE,OAAO,CAAA;IAC1B;;;;;;OAMG;IACH,cAAc,CAAC,EAAE,QAAQ,GAAG,QAAQ,GAAG,KAAK,CAAA;IAC5C,4FAA4F;IAC5F,mBAAmB,CAAC,EAAE,MAAM,CAAA;IAC5B,8CAA8C;IAC9C,qBAAqB,CAAC,EAAE,MAAM,CAAA;IAC9B,oEAAoE;IACpE,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,iEAAiE;IACjE,YAAY,CAAC,EAAE,MAAM,CAAA;IACrB,2EAA2E;IAC3E,aAAa,CAAC,EAAE,MAAM,CAAA;IACtB,yEAAyE;IACzE,eAAe,CAAC,EAAE,OAAO,CAAA;IACzB,iEAAiE;IACjE,GAAG,CAAC,EAAE,SAAS,CAAA;CAChB;AAED,4DAA4D;AAC5D,MAAM,WAAW,cAAc;IAC7B,iDAAiD;IACjD,eAAe,EAAE,OAAO,CAAA;IACxB,qEAAqE;IACrE,mBAAmB,EAAE,aAAa,CAAC,MAAM,CAAC,CAAA;IAC1C,oEAAoE;IACpE,kBAAkB,EAAE,OAAO,CAAA;IAC3B,2CAA2C;IAC3C,aAAa,EAAE,aAAa,CAAA;IAC5B,8BAA8B;IAC9B,SAAS,EAAE,SAAS,CAAA;IACpB,+CAA+C;IAC/C,eAAe,EAAE,MAAM,CAAA;IACvB,2CAA2C;IAC3C,oBAAoB,EAAE,MAAM,CAAA;IAC5B,yCAAyC;IACzC,iBAAiB,EAAE,MAAM,CAAA;IACzB,gDAAgD;IAChD,gBAAgB,EAAE,OAAO,CAAA;IACzB,0CAA0C;IAC1C,cAAc,EAAE,QAAQ,GAAG,QAAQ,GAAG,KAAK,CAAA;IAC3C,+CAA+C;IAC/C,mBAAmB,EAAE,MAAM,CAAA;IAC3B,kCAAkC;IAClC,qBAAqB,EAAE,MAAM,CAAA;IAC7B,uCAAuC;IACvC,WAAW,EAAE,MAAM,CAAA;IACnB,+BAA+B;IAC/B,YAAY,EAAE,MAAM,CAAA;IACpB,+CAA+C;IAC/C,aAAa,EAAE,MAAM,CAAA;IACrB,oEAAoE;IACpE,eAAe,EAAE,OAAO,CAAA;IACxB,2CAA2C;IAC3C,GAAG,EAAE;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAA;KAAE,CAAA;CAC7D;AAED,6DAA6D;AAC7D,eAAO,MAAM,MAAM,EAAE,CAAC,CAAC,MAAM,CAsB3B,CAAA;AAgBF;;;;;GAKG;AACH,wBAAgB,aAAa,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,GAAG,cAAc,CAyFxE"}
|
package/lib/types/observe.d.ts
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
* @module dsh-click/observe
|
|
9
9
|
*/
|
|
10
10
|
import type { ResolvedConfig } from './config.js';
|
|
11
|
-
import type { Rect, WindowSnapshot } from './platform/types.js';
|
|
11
|
+
import type { ElementInfo, Rect, WindowSnapshot } from './platform/types.js';
|
|
12
12
|
/** One cached observation, immutable after construction. */
|
|
13
13
|
export interface ObservationRecord {
|
|
14
14
|
/** Observation id (content hash). */
|
|
@@ -21,6 +21,8 @@ export interface ObservationRecord {
|
|
|
21
21
|
rect: Rect;
|
|
22
22
|
treeHash: string;
|
|
23
23
|
shotHash: string;
|
|
24
|
+
/** Per-element hashes captured from a tree read (absent for screenshot-only observations). */
|
|
25
|
+
elementHashes?: Readonly<Record<string, string>>;
|
|
24
26
|
elementCount: number;
|
|
25
27
|
/** Wall-clock capture time in milliseconds. */
|
|
26
28
|
observedAt: number;
|
|
@@ -40,6 +42,8 @@ export type FreshnessVerdict = {
|
|
|
40
42
|
* @returns a hex observation id.
|
|
41
43
|
*/
|
|
42
44
|
export declare function observationIdOf(snapshot: WindowSnapshot): string;
|
|
45
|
+
/** Hash one captured element's identity-relevant fields. */
|
|
46
|
+
export declare function elementHashOf(element: ElementInfo): string;
|
|
43
47
|
/** Bounded LRU cache of observations, plus the staleness verdict logic. */
|
|
44
48
|
export declare class ObservationStore {
|
|
45
49
|
private readonly config;
|
|
@@ -52,7 +56,7 @@ export declare class ObservationStore {
|
|
|
52
56
|
* @param snapshot - the snapshot facts it came from.
|
|
53
57
|
* @returns the stored record (id = hash of the state).
|
|
54
58
|
*/
|
|
55
|
-
record(snapshot: WindowSnapshot): ObservationRecord;
|
|
59
|
+
record(snapshot: WindowSnapshot, elements?: readonly ElementInfo[]): ObservationRecord;
|
|
56
60
|
/**
|
|
57
61
|
* Look one observation up by id.
|
|
58
62
|
*
|
|
@@ -72,6 +76,9 @@ export declare class ObservationStore {
|
|
|
72
76
|
* @param now - wall-clock now (injectable for tests).
|
|
73
77
|
* @returns the verdict.
|
|
74
78
|
*/
|
|
75
|
-
verify(record: ObservationRecord, fresh: WindowSnapshot, now?: number
|
|
79
|
+
verify(record: ObservationRecord, fresh: WindowSnapshot, now?: number, target?: {
|
|
80
|
+
readonly ref: string;
|
|
81
|
+
readonly elements: readonly ElementInfo[];
|
|
82
|
+
}): FreshnessVerdict;
|
|
76
83
|
}
|
|
77
84
|
//# sourceMappingURL=observe.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"observe.d.ts","sourceRoot":"","sources":["../../src/observe.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAGH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AACjD,OAAO,KAAK,EAAE,IAAI,EAAE,cAAc,EAAE,MAAM,qBAAqB,CAAA;
|
|
1
|
+
{"version":3,"file":"observe.d.ts","sourceRoot":"","sources":["../../src/observe.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAGH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AACjD,OAAO,KAAK,EAAE,WAAW,EAAE,IAAI,EAAE,cAAc,EAAE,MAAM,qBAAqB,CAAA;AAE5E,4DAA4D;AAC5D,MAAM,WAAW,iBAAiB;IAChC,qCAAqC;IACrC,EAAE,EAAE,MAAM,CAAA;IACV,QAAQ,EAAE,MAAM,CAAA;IAChB,SAAS,EAAE,MAAM,CAAA;IACjB,cAAc,EAAE,MAAM,GAAG,IAAI,CAAA;IAC7B,KAAK,EAAE,MAAM,CAAA;IACb,SAAS,EAAE,MAAM,CAAA;IACjB,IAAI,EAAE,IAAI,CAAA;IACV,QAAQ,EAAE,MAAM,CAAA;IAChB,QAAQ,EAAE,MAAM,CAAA;IAChB,8FAA8F;IAC9F,aAAa,CAAC,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC,CAAA;IAChD,YAAY,EAAE,MAAM,CAAA;IACpB,+CAA+C;IAC/C,UAAU,EAAE,MAAM,CAAA;CACnB;AAED,wFAAwF;AACxF,MAAM,MAAM,gBAAgB,GACxB;IAAE,EAAE,EAAE,IAAI,CAAA;CAAE,GACZ;IAAE,EAAE,EAAE,KAAK,CAAC;IAAC,IAAI,EAAE,qBAAqB,GAAG,SAAS,GAAG,gBAAgB,GAAG,YAAY,GAAG,cAAc,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAAA;AAE7H;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,QAAQ,EAAE,cAAc,GAAG,MAAM,CAYhE;AAED,4DAA4D;AAC5D,wBAAgB,aAAa,CAAC,OAAO,EAAE,WAAW,GAAG,MAAM,CAU1D;AAED,2EAA2E;AAC3E,qBAAa,gBAAgB;IAIf,OAAO,CAAC,QAAQ,CAAC,MAAM;IAHnC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAuC;IAE/D,wEAAwE;gBAC3C,MAAM,EAAE,cAAc;IAEnD;;;;;OAKG;IACH,MAAM,CAAC,QAAQ,EAAE,cAAc,EAAE,QAAQ,CAAC,EAAE,SAAS,WAAW,EAAE,GAAG,iBAAiB;IA2BtF;;;;;OAKG;IACH,GAAG,CAAC,EAAE,EAAE,MAAM,GAAG,iBAAiB,GAAG,SAAS;IAI9C;;;;;;;;;;;OAWG;IACH,MAAM,CACJ,MAAM,EAAE,iBAAiB,EACzB,KAAK,EAAE,cAAc,EACrB,GAAG,GAAE,MAAmB,EACxB,MAAM,CAAC,EAAE;QAAE,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,QAAQ,EAAE,SAAS,WAAW,EAAE,CAAA;KAAE,GAC3E,gBAAgB;CAoDpB"}
|
package/lib/types/version.d.ts
CHANGED
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
* @module dsh-click/version
|
|
8
8
|
*/
|
|
9
9
|
/** Plugin version advertised in the helper-protocol handshake. */
|
|
10
|
-
export declare const VERSION = "0.3.
|
|
10
|
+
export declare const VERSION = "0.3.26";
|
|
11
11
|
/** The helper protocol revision; bump only on incompatible wire changes. */
|
|
12
12
|
export declare const HELPER_PROTOCOL_VERSION = 1;
|
|
13
13
|
//# sourceMappingURL=version.d.ts.map
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-click",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.26",
|
|
4
4
|
"description": "Cross-platform native desktop control tools for DeepSeek Harness (Windows first): screen_shot, screen_read (accessibility tree + pixel description for text-only models), click/type/scroll/key, and app_list/app_launch — every mutating action gated by approval, never stealing foreground focus, with stale-observation rejection and process-identity checks",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -137,7 +137,7 @@
|
|
|
137
137
|
"oxlint": "0.18.1",
|
|
138
138
|
"vitest": "^3.2.0",
|
|
139
139
|
"@deepseek-ai/dsh-working-directory": "0.2.1-rc.2",
|
|
140
|
-
"tsdown": "^0.23.
|
|
140
|
+
"tsdown": "^0.23.2",
|
|
141
141
|
"typescript": "^5.9.0"
|
|
142
142
|
},
|
|
143
143
|
"scripts": {
|