nexus-windows-codex 3.0.0 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -0
- package/README.md +12 -7
- package/lib/garde-fous.mjs +15 -0
- package/lib/mcp.mjs +62 -8
- package/lib/native.mjs +16 -10
- package/lib/session.mjs +18 -12
- package/native/bin/NexusCodex.UIA.exe +2 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [3.1.0] - 2026-10-24 — Tk in the background, common image rule, zoom
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- Tk in the background, window covered: a click at a point is sent with the mouse button marked pressed for Tk's own
|
|
14
|
+
thread only (press and release owned by a helper thread); lists scroll through their native scroll bar, read back
|
|
15
|
+
(recipe shared with NexusWindows for Claude).
|
|
16
|
+
- `look(region=[x1, y1, x2, y2])`: an area of the last full native image, read at full resolution and enlarged up to
|
|
17
|
+
2x, within the same caps; the zoomed image has its own frame, so its points work with its own `screenshotId`, while
|
|
18
|
+
the move guard keeps checking the whole window frame. An area outside the window is refused.
|
|
19
|
+
- No duplicate image: same WebP bytes (deterministic encoder, so same pixels), size and frame as the last image sent for
|
|
20
|
+
that window or zoom: not attached again; the reply names the id that takes points; `mode="image"` always attaches it.
|
|
21
|
+
The zoom reference only follows images the model actually received.
|
|
22
|
+
- `launch` of a Chromium/Electron executable by its path: started with the switches that keep it drawn while covered.
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
- Native images follow the common Nexus image rule: at most 1568 px and 1.15 MP (area-average reduction), lossless
|
|
26
|
+
WebP from a built-in VP8L encoder (no dependency; checked pixel-exact against libwebp), near-lossless above 200 KB,
|
|
27
|
+
never over 500 KB. A point on a reduced image is scaled back to the exact screen pixel, the same one the guards check.
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
- Outcome classification: once any input may have left, a failure is an unknown outcome (never replayed, no fallback,
|
|
31
|
+
accord consumed); a refusal raised before any input (applicability checks, the foreground token wait, the user's
|
|
32
|
+
pause, the emergency stop, a COM refusal before any pattern call, Chromium before its first activation message) is
|
|
33
|
+
"nothing sent", so the planned fallback stays available.
|
|
34
|
+
- The emergency stop is checked between Tk scroll lines and is never masked by the end-of-scroll cleanup.
|
|
35
|
+
|
|
36
|
+
### Security
|
|
37
|
+
- Reviewed by the Codex agent over many rounds (partial input, held press ownership, frames of zoomed images, outcome
|
|
38
|
+
classification before and after input).
|
|
39
|
+
|
|
40
|
+
## [3.0.1] - 2026-10-23 — Family naming, release with provenance
|
|
41
|
+
|
|
42
|
+
### Changed
|
|
43
|
+
- Same version as NexusWindows for Claude (now published as `nexus-windows-claude`) and NexusWindows for
|
|
44
|
+
Antigravity; the README family table points to the new Claude package.
|
|
45
|
+
- Published to npm from GitHub Actions through trusted publishing, with provenance (3.0.0 was published by hand).
|
|
46
|
+
|
|
10
47
|
## [3.0.0] - 2026-10-23 — Standalone MCP server, no Sky
|
|
11
48
|
|
|
12
49
|
The three NexusWindows tools (Claude, Codex, Antigravity) move to 3.0.0 together: one-command install from a package
|
package/README.md
CHANGED
|
@@ -22,7 +22,7 @@ capture of covered windows, Win32 messages and UI Automation patterns without fo
|
|
|
22
22
|
keyboard as the last resort, under a shared foreground token. Every action is judged from the application's real
|
|
23
23
|
state, and a batch needs an explicit postcondition per step. It also ships as a Codex skill that layers the same session over Codex's Computer Use API (`@oai/sky`).
|
|
24
24
|
|
|
25
|
-
Version 3.0.
|
|
25
|
+
Version 3.0.1 · Windows 10/11 · Node 22+ · .NET 8 · MIT licence
|
|
26
26
|
|
|
27
27
|
Part of the Nexus family: [NexusWindows](https://julienpiron.fr/nexuswindows/)
|
|
28
28
|
([Claude](https://github.com/pironjulien/NexusWindows-Claude) ·
|
|
@@ -51,8 +51,8 @@ promise: the user keeps their mouse, keyboard and foreground window, and every a
|
|
|
51
51
|
|
|
52
52
|
| | Claude | Codex | Antigravity |
|
|
53
53
|
|---|---|---|---|
|
|
54
|
-
| Install | `uvx --python 3.14 nexus-windows` | `npx -y nexus-windows-codex` | `uvx --python 3.14 nexus-windows-antigravity` |
|
|
55
|
-
| Package | [PyPI](https://pypi.org/project/nexus-windows/) | [npm](https://www.npmjs.com/package/nexus-windows-codex) | [PyPI](https://pypi.org/project/nexus-windows-antigravity/) |
|
|
54
|
+
| Install | `uvx --python 3.14 nexus-windows-claude` | `npx -y nexus-windows-codex` | `uvx --python 3.14 nexus-windows-antigravity` |
|
|
55
|
+
| Package | [PyPI](https://pypi.org/project/nexus-windows-claude/) | [npm](https://www.npmjs.com/package/nexus-windows-codex) | [PyPI](https://pypi.org/project/nexus-windows-antigravity/) |
|
|
56
56
|
| Tools | `windows` `look` `act` `launch` `window`, plus `notifications` and `logs` | `windows` `look` `act` `launch` `window` | `windows` `look` `act` `launch` `window` |
|
|
57
57
|
| Repository | [NexusWindows-Claude](https://github.com/pironjulien/NexusWindows-Claude) | [NexusWindows-Codex](https://github.com/pironjulien/NexusWindows-Codex) | [NexusWindows-Antigravity](https://github.com/pironjulien/NexusWindows-Antigravity) |
|
|
58
58
|
|
|
@@ -101,7 +101,7 @@ the JavaScript module cache does not reload.
|
|
|
101
101
|
| Tool | Purpose |
|
|
102
102
|
|---|---|
|
|
103
103
|
| `windows(filter)` | Application windows (id, app, title, pid), like the taskbar; password managers never listed |
|
|
104
|
-
| `look(window, mode)` | One window by id or a unique part of its title or app: UI Automation tree, plus a capture that works on covered windows with `mode="image"` or `"both"`; selects the window for `act` |
|
|
104
|
+
| `look(window, mode, region)` | One window by id or a unique part of its title or app: UI Automation tree, plus a capture that works on covered windows with `mode="image"` or `"both"`; `region` zooms on an area of the last image; selects the window for `act` |
|
|
105
105
|
| `act(steps)` | One step, or up to 32 each with its own `expect`, on the window of the last `look`; stops at the first step not verified |
|
|
106
106
|
| `launch(app)` | Start an application (name, path, Store app id), a folder, a document or a Settings page (`ms-settings:`) and give the foreground back |
|
|
107
107
|
| `window(action, window)` | `action="close"`: one close message, read back; a "save?" prompt is reported, never answered |
|
|
@@ -195,9 +195,14 @@ its option element.
|
|
|
195
195
|
`observe({mode:'image', capture:'native'})` captures the bound window with `PrintWindow(PW_RENDERFULLCONTENT)`
|
|
196
196
|
without bringing it forward, even when covered: per-monitor DPI aware, cropped to the DWM frame, DPI-unaware windows
|
|
197
197
|
drawn at their logical size and scaled. In the Codex skill, a blank, minimized or failed capture falls back to Sky's
|
|
198
|
-
screenshot; the MCP server has no such fallback and reports the failure. Native images
|
|
199
|
-
at most
|
|
200
|
-
|
|
198
|
+
screenshot; the MCP server has no such fallback and reports the failure. Native images follow the common Nexus image
|
|
199
|
+
rule: at most 1568 px and 1.15 MP, lossless WebP from a built-in encoder (near-lossless above 200 KB, never over
|
|
200
|
+
500 KB), local files (32 at most, removed after 10 minutes at the next capture, at server start and at a clean stop):
|
|
201
|
+
evidence for the model, and the coordinate frame of a point action that names the image (`screenshotId`), scaled back
|
|
202
|
+
to the screen pixel. `look(region=[x1, y1, x2, y2])` reads an area of the last full image at full resolution, enlarged
|
|
203
|
+
up to 2x for small text; its own `screenshotId` takes points. An image identical to the last one sent for that window
|
|
204
|
+
(same pixels and frame) is not attached again; the reply names the id that takes points, and `mode="image"` always
|
|
205
|
+
attaches it.
|
|
201
206
|
|
|
202
207
|
### Verified batches (`actBatch`)
|
|
203
208
|
|
package/lib/garde-fous.mjs
CHANGED
|
@@ -205,10 +205,25 @@ export function elementsAt(elements, x, y) {
|
|
|
205
205
|
return w > 0 && h > 0 && x >= ex && y >= ey && x < ex + w && y < ey + h;
|
|
206
206
|
});
|
|
207
207
|
}
|
|
208
|
+
// Point d'une capture native → pixel écran ENTIER, celui que l'entrée vise (session) et que la confirmation contrôle
|
|
209
|
+
// (screenPoints) : une seule conversion, arrondie au centre de pixel et bornée au cadre (relecture de Codex du 24/10).
|
|
210
|
+
export function nativeScreen(shot, point) {
|
|
211
|
+
const f = shot.frame;
|
|
212
|
+
const sx = shot.width > 0 ? f.width / shot.width : 1;
|
|
213
|
+
const sy = shot.height > 0 ? f.height / shot.height : 1;
|
|
214
|
+
return {
|
|
215
|
+
x: Math.max(f.left, Math.min(f.left + f.width - 1, Math.round(f.left + (point.x + 0.5) * sx - 0.5))),
|
|
216
|
+
y: Math.max(f.top, Math.min(f.top + f.height - 1, Math.round(f.top + (point.y + 0.5) * sy - 0.5))),
|
|
217
|
+
};
|
|
218
|
+
}
|
|
208
219
|
// Points écran possibles d'un point de capture Sky : origine de la capture + point, ou point ramené à la taille de la
|
|
209
220
|
// fenêtre (capture mise à l'échelle). Les deux sont contrôlés : le doute ne laisse rien passer.
|
|
210
221
|
export function screenPoints(point, screenshot, root) {
|
|
211
222
|
const out = [];
|
|
223
|
+
if (screenshot?.native && screenshot.frame) { // capture native éventuellement réduite : le pixel même de l'entrée
|
|
224
|
+
const at = nativeScreen(screenshot, point);
|
|
225
|
+
out.push([at.x, at.y]);
|
|
226
|
+
}
|
|
212
227
|
if (Number.isFinite(screenshot?.originX) && Number.isFinite(screenshot?.originY)) out.push([screenshot.originX + point.x, screenshot.originY + point.y]);
|
|
213
228
|
const [rx, ry, rw, rh] = root?.bounds ?? [];
|
|
214
229
|
if (rw > 0 && rh > 0 && screenshot?.width > 0 && screenshot?.height > 0) {
|
package/lib/mcp.mjs
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
// sans Sky, exposée en outils. Aucune dépendance npm : le protocole utile (initialize, ping, tools/list, tools/call)
|
|
3
3
|
// tient ici, et la chaîne d'approvisionnement reste vide comme pour le reste de l'outil.
|
|
4
4
|
import { readFile } from 'node:fs/promises';
|
|
5
|
+
import { createHash } from 'node:crypto';
|
|
5
6
|
import { summarize } from './session.mjs';
|
|
6
7
|
|
|
7
8
|
export const PROTOCOL_VERSIONS = ['2025-11-25', '2025-06-18', '2025-03-26', '2024-11-05'];
|
|
@@ -42,6 +43,8 @@ const ACTION = {
|
|
|
42
43
|
},
|
|
43
44
|
};
|
|
44
45
|
const OBSERVE_MODE = { type: 'string', enum: ['text', 'image', 'both'], description: 'text (UI Automation tree, default), image (native capture), both' };
|
|
46
|
+
const REGION = { type: 'array', items: { type: 'number' }, minItems: 4, maxItems: 4,
|
|
47
|
+
description: '[x1, y1, x2, y2] in pixels of the last native image of this window: that area enlarged up to 2x (small text); its own screenshotId takes points' };
|
|
45
48
|
const WINDOW = { type: ['integer', 'string'], description: 'Window id from windows, or a unique part of its title or app; omitted: the window of the last look' };
|
|
46
49
|
const READ = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
|
|
47
50
|
const WRITE = { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: true };
|
|
@@ -54,9 +57,10 @@ export const TOOLS = [
|
|
|
54
57
|
inputSchema: { type: 'object', additionalProperties: false, properties: { filter: { type: 'string' } } } },
|
|
55
58
|
{ name: 'look', title: 'Look at a window', annotations: { ...READ, title: 'Look at a window' },
|
|
56
59
|
description: 'See a window in the background, without bringing it forward: UI Automation tree (roles, names, ids, values, states) and, ' +
|
|
57
|
-
'with mode image or both, a native capture that works even when the window is covered
|
|
60
|
+
'with mode image or both, a native capture that works even when the window is covered (WebP, at most 1568 px; ' +
|
|
61
|
+
'unchanged since the last image sent: not attached again, mode image always attaches it). Selects that window for act; ' +
|
|
58
62
|
'an observation serves one act and expires after 60 s.',
|
|
59
|
-
inputSchema: { type: 'object', additionalProperties: false, properties: { window: WINDOW, mode: OBSERVE_MODE } } },
|
|
63
|
+
inputSchema: { type: 'object', additionalProperties: false, properties: { window: WINDOW, mode: OBSERVE_MODE, region: REGION } } },
|
|
60
64
|
{ name: 'act', title: 'Act in the window', annotations: { ...WRITE, title: 'Act in the window' },
|
|
61
65
|
description: 'Steps on the window of the last look, in the background first (Win32 messages, UI Automation, DevTools for Chromium), ' +
|
|
62
66
|
'the real mouse/keyboard only as a fallback under the shared foreground token. One step, or up to 32 each with its own expect; ' +
|
|
@@ -91,17 +95,49 @@ function errorStatus(error) {
|
|
|
91
95
|
}
|
|
92
96
|
|
|
93
97
|
// Observation lisible par le modèle : arbre et état, plus l'image native jointe (fichier local, jamais copiée ailleurs).
|
|
94
|
-
|
|
98
|
+
// Pas d'image en double (règle commune des produits Nexus, shared/images.md) : mêmes octets (encodeur WebP déterministe :
|
|
99
|
+
// mêmes pixels), mêmes dimensions et même cadre écran que la dernière image envoyée pour cette fenêtre (ou ce zoom) →
|
|
100
|
+
// non jointe, la réponse dit quel identifiant prend les points. `always` (mode image) la joint toujours. `received` :
|
|
101
|
+
// appelé pour chaque image que le modèle a réellement (jointe, ou identique à la dernière envoyée), jamais pour une
|
|
102
|
+
// image illisible (purgée) qui ne part pas.
|
|
103
|
+
async function observationContent(observation, sent = null, always = false, received = null) {
|
|
95
104
|
const summary = summarize(observation);
|
|
96
105
|
const w = observation.window ?? {};
|
|
97
106
|
const content = [text(`window ${w.id} "${w.title ?? ''}" (${w.app ?? '?'}) · ${observation.elements?.length ?? 0} elements · ${observation.source}`), text(summary)];
|
|
98
107
|
for (const image of observation.images ?? []) {
|
|
99
|
-
|
|
100
|
-
|
|
108
|
+
const mimeType = image.path.endsWith('.webp') ? 'image/webp' : 'image/png';
|
|
109
|
+
let data;
|
|
110
|
+
try { data = await readFile(image.path); }
|
|
111
|
+
catch { continue; /* capture purgée entre-temps : l'arbre reste */ }
|
|
112
|
+
const key = `${w.id}|${image.zoom ? 'zoom' : 'full'}`;
|
|
113
|
+
const print = createHash('sha256').update(JSON.stringify([image.width, image.height, image.frame])).update(data).digest('hex');
|
|
114
|
+
const last = sent?.get(key);
|
|
115
|
+
if (!always && last?.print === print) {
|
|
116
|
+
content.push(text(`Image ${image.id} unchanged since ${last.id} (same pixels and frame): not attached again; points take screenshotId ${image.id} (mode image attaches it).`));
|
|
117
|
+
received?.(image, w.id);
|
|
118
|
+
continue;
|
|
119
|
+
}
|
|
120
|
+
sent?.set(key, { print, id: image.id });
|
|
121
|
+
content.push({ type: 'image', data: data.toString('base64'), mimeType });
|
|
122
|
+
received?.(image, w.id);
|
|
101
123
|
}
|
|
102
124
|
return content;
|
|
103
125
|
}
|
|
104
126
|
|
|
127
|
+
// Zone [x1, y1, x2, y2] (pixels inclus) d'une image native → zone écran [left, top, width, height] : bords des pixels de
|
|
128
|
+
// l'image ramenés au cadre (même échelle que les points), bornés au cadre.
|
|
129
|
+
export function regionToScreen(image, region) {
|
|
130
|
+
if (!image?.frame || !(image.width > 0) || !(image.height > 0) || !Array.isArray(region) || region.length !== 4 || !region.every(Number.isFinite)) {
|
|
131
|
+
throw new Error('region needs [x1, y1, x2, y2] and a native image of this window (look with mode image or both first)');
|
|
132
|
+
}
|
|
133
|
+
const f = image.frame, sx = f.width / image.width, sy = f.height / image.height;
|
|
134
|
+
const [x1, y1, x2, y2] = region;
|
|
135
|
+
const left = Math.max(0, Math.floor(Math.min(x1, x2) * sx)), top = Math.max(0, Math.floor(Math.min(y1, y2) * sy));
|
|
136
|
+
const right = Math.min(f.width, Math.ceil((Math.max(x1, x2) + 1) * sx)), bottom = Math.min(f.height, Math.ceil((Math.max(y1, y2) + 1) * sy));
|
|
137
|
+
if (right - left < 2 || bottom - top < 2) throw new Error('region is empty or outside the image');
|
|
138
|
+
return [f.left + left, f.top + top, right - left, bottom - top];
|
|
139
|
+
}
|
|
140
|
+
|
|
105
141
|
function stepLine(step) {
|
|
106
142
|
const how = [step.technique, step.foregroundTaken ? 'foreground taken' : null].filter(Boolean).join(', ');
|
|
107
143
|
return `${mark(step.status)} ${step.kind} — ${step.status}${how ? ` (${how})` : ''}${step.error ? `: ${step.error}` : ''}`;
|
|
@@ -109,6 +145,16 @@ function stepLine(step) {
|
|
|
109
145
|
|
|
110
146
|
export class WindowsTools {
|
|
111
147
|
#session; #observation = null; #bound = null;
|
|
148
|
+
#sent = new Map(); // fenêtre|full ou zoom → dernière image envoyée (empreinte, id)
|
|
149
|
+
#images = new Map(); // fenêtre observée → dernière image native complète envoyée (repère des zones de zoom)
|
|
150
|
+
#observed = new Map(); // fenêtre liée → fenêtre réellement observée (dialogue suivi)
|
|
151
|
+
// Toute observation renvoyée (look, act avec refresh) : fenêtre réellement observée ; son image complète devient le
|
|
152
|
+
// repère des régions seulement si le modèle l'a reçue (#received, relecture de Codex du 24/10).
|
|
153
|
+
#remember(observation) {
|
|
154
|
+
const id = observation?.window?.id;
|
|
155
|
+
if (id !== undefined && this.#bound?.id !== undefined) this.#observed.set(this.#bound.id, id);
|
|
156
|
+
}
|
|
157
|
+
#received = (image, windowId) => { if (!image.zoom && windowId !== undefined) this.#images.set(windowId, image); };
|
|
112
158
|
constructor(session) { this.#session = session; }
|
|
113
159
|
async #choose(window) {
|
|
114
160
|
const listed = await this.#session.windows();
|
|
@@ -149,8 +195,13 @@ export class WindowsTools {
|
|
|
149
195
|
}
|
|
150
196
|
case 'look': {
|
|
151
197
|
await this.#bind(args.window);
|
|
152
|
-
|
|
153
|
-
|
|
198
|
+
// Repère : dernière image complète envoyée de la fenêtre que cette liaison observe (dialogue suivi compris).
|
|
199
|
+
const shown = this.#observed.get(this.#bound?.id) ?? this.#bound?.id;
|
|
200
|
+
const zoom = args.region === undefined ? null : regionToScreen(this.#images.get(shown), args.region);
|
|
201
|
+
const mode = zoom && (args.mode ?? 'text') === 'text' ? 'both' : args.mode ?? 'text';
|
|
202
|
+
this.#observation = await s.observe({ mode, backend: 'native', capture: 'native', zoom });
|
|
203
|
+
this.#remember(this.#observation);
|
|
204
|
+
return observationContent(this.#observation, this.#sent, mode === 'image', this.#received);
|
|
154
205
|
}
|
|
155
206
|
case 'act': {
|
|
156
207
|
if (!this.#observation) throw new Error('Look at the window first (look)');
|
|
@@ -172,7 +223,10 @@ export class WindowsTools {
|
|
|
172
223
|
const lines = report.steps.map(stepLine).join('\n');
|
|
173
224
|
const content = [text(lines + (this.#observation ? '' : '\nNo fresh observation: look again before the next act.'))];
|
|
174
225
|
// Observation rafraîchie : arbre et, demandée par refresh image|both, l'image native jointe comme pour look.
|
|
175
|
-
if (report.observation)
|
|
226
|
+
if (report.observation) {
|
|
227
|
+
this.#remember(report.observation);
|
|
228
|
+
content.push(...(await observationContent(report.observation, this.#sent, false, this.#received)).slice(1));
|
|
229
|
+
}
|
|
176
230
|
return content;
|
|
177
231
|
}
|
|
178
232
|
case 'launch': {
|
package/lib/native.mjs
CHANGED
|
@@ -5,24 +5,30 @@ import { policyPath } from './garde-fous.mjs';
|
|
|
5
5
|
|
|
6
6
|
// Arrêt d'urgence rapporté par le helper : avant toute entrée (refus, rien d'envoyé), ou APRÈS une entrée partielle
|
|
7
7
|
// (« Emergency stop after input: k of n characters sent ») : issue inconnue, accord consommé, jamais rejoué.
|
|
8
|
-
export const stopRefusal = (message, error) => {
|
|
8
|
+
export const stopRefusal = (message, error, inputStarted = false) => {
|
|
9
9
|
if (error !== 'EmergencyStopException') return null;
|
|
10
10
|
const partial = /^Emergency stop after input: (\d+) of (\d+) characters sent/.exec(message ?? '');
|
|
11
|
-
|
|
12
|
-
|
|
11
|
+
const after = Boolean(partial) || inputStarted === true; // helper : entrée déjà partie (inputStarted)
|
|
12
|
+
return Object.assign(new Error(message), { name: 'EmergencyStopError', emergencyStop: true, deferred: !after,
|
|
13
|
+
...(after ? { afterInput: true } : {}),
|
|
14
|
+
...(partial ? { typed: { characters: Number(partial[1]), total: Number(partial[2]) } } : {}) });
|
|
13
15
|
};
|
|
14
16
|
|
|
15
17
|
// Erreurs du helper. « notApplicable » : aucune entrée envoyée, la technique suivante est permise. « refused » : aucune
|
|
16
18
|
// entrée envoyée et aucun repli (mot de passe, application exclue, identité changée, fenêtre fermée, cible désactivée).
|
|
17
19
|
// « julienActive » : Julien a repris clavier ou souris pendant l'attente du jeton, rien n'a été envoyé. Toute autre
|
|
18
20
|
// erreur d'action peut suivre une entrée : résultat inconnu, jamais rejoué.
|
|
19
|
-
|
|
21
|
+
// inputStarted (helper) : une entrée est déjà partie ; aucune de ces natures « rien n'a été envoyé » ne s'applique plus,
|
|
22
|
+
// l'issue est inconnue et terminale (relecture de Codex du 24/10 : sinon un repli physique rejouait le geste).
|
|
23
|
+
function nativeError({ message, error, inputStarted }) {
|
|
24
|
+
const after = inputStarted === true;
|
|
20
25
|
return Object.assign(new Error(message || 'Native UIA failed'), {
|
|
21
26
|
nativeError: error ?? null,
|
|
22
|
-
deferred: error === 'TimeoutException' || error === 'JulienActiveException',
|
|
23
|
-
julienActive: error === 'JulienActiveException',
|
|
24
|
-
notApplicable: error === 'BackgroundNotApplicableException' || error === 'CaptureUnavailableException',
|
|
25
|
-
refused: error === 'ActionRefusedException',
|
|
27
|
+
deferred: !after && (error === 'TimeoutException' || error === 'JulienActiveException'),
|
|
28
|
+
julienActive: !after && error === 'JulienActiveException',
|
|
29
|
+
notApplicable: !after && (error === 'BackgroundNotApplicableException' || error === 'CaptureUnavailableException'),
|
|
30
|
+
refused: !after && error === 'ActionRefusedException',
|
|
31
|
+
...(after ? { afterInput: true } : {}),
|
|
26
32
|
});
|
|
27
33
|
}
|
|
28
34
|
|
|
@@ -103,7 +109,7 @@ export class NativeUIA {
|
|
|
103
109
|
this.#idle = setTimeout(() => this.close(), this.idleTimeoutMs);
|
|
104
110
|
this.#idle.unref();
|
|
105
111
|
if (response.result.ok) pending.resolve(response.result);
|
|
106
|
-
else pending.reject(stopRefusal(response.result.message, response.result.error) ?? nativeError(response.result));
|
|
112
|
+
else pending.reject(stopRefusal(response.result.message, response.result.error, response.result.inputStarted) ?? nativeError(response.result));
|
|
107
113
|
} catch (error) { fail(error); return; }
|
|
108
114
|
}
|
|
109
115
|
});
|
|
@@ -132,7 +138,7 @@ export class NativeUIA {
|
|
|
132
138
|
}
|
|
133
139
|
inspect(window) { return this.call(window, 'inspect'); }
|
|
134
140
|
// Lectures sans entrée (permises pendant l'arrêt d'urgence).
|
|
135
|
-
capture(window, directory) { return this.call(window, 'capture', { directory }); }
|
|
141
|
+
capture(window, directory, zoom) { return this.call(window, 'capture', { directory, ...(zoom ? { zoom } : {}) }); }
|
|
136
142
|
related(window) { return this.call(window, 'related'); }
|
|
137
143
|
alive(window) { return this.call(window, 'alive'); }
|
|
138
144
|
// Page des Paramètres (ms-settings: seul), ouverte sous le jeton et premier plan rendu ; rend la fenêtre notée.
|
package/lib/session.mjs
CHANGED
|
@@ -105,7 +105,10 @@ const actionFields = {
|
|
|
105
105
|
function nativeShot(observation, id) {
|
|
106
106
|
const image = (observation.images ?? []).find(i => i.id === id);
|
|
107
107
|
if (!image?.frame) return undefined;
|
|
108
|
-
|
|
108
|
+
// frame : zone que couvre l'image (points) ; windowFrame : cadre entier de la fenêtre (garde de mouvement du helper),
|
|
109
|
+
// distincts pour une image zoomée (relecture de Codex du 24/10).
|
|
110
|
+
return { id, width: image.width, height: image.height, originX: image.frame.left, originY: image.frame.top, frame: image.frame,
|
|
111
|
+
windowFrame: image.windowFrame ?? image.frame, native: true };
|
|
109
112
|
}
|
|
110
113
|
function validateAction(spec) {
|
|
111
114
|
if (!spec || typeof spec !== 'object' || Array.isArray(spec) || !Object.hasOwn(actionFields, spec.kind)) throw new Error('Unsupported action');
|
|
@@ -525,7 +528,7 @@ export class WindowsSession {
|
|
|
525
528
|
}
|
|
526
529
|
async #pruneCaptures(directory) {
|
|
527
530
|
let files;
|
|
528
|
-
try { files = (await readdir(directory)).filter(f => f.endsWith('.png')); } catch { return; }
|
|
531
|
+
try { files = (await readdir(directory)).filter(f => f.endsWith('.png') || f.endsWith('.webp')); } catch { return; }
|
|
529
532
|
const dated = [];
|
|
530
533
|
for (const name of files) {
|
|
531
534
|
try { dated.push({ file: path.join(directory, name), at: (await stat(path.join(directory, name))).mtimeMs }); } catch { /* déjà retiré */ }
|
|
@@ -536,7 +539,7 @@ export class WindowsSession {
|
|
|
536
539
|
if (i >= CAPTURE_KEEP || now - entry.at > CAPTURE_RETENTION_MS) await unlink(entry.file).catch(() => {});
|
|
537
540
|
}
|
|
538
541
|
}
|
|
539
|
-
async #capture(mode, backend = 'auto', { capture = 'sky', windowChange = null } = {}) {
|
|
542
|
+
async #capture(mode, backend = 'auto', { capture = 'sky', windowChange = null, zoom = null } = {}) {
|
|
540
543
|
this.#invalidate();
|
|
541
544
|
if (!this.#window) throw new Error('Bind a returned window before observing');
|
|
542
545
|
if (!['text', 'image', 'both'].includes(mode)) throw new Error('mode must be text, image or both');
|
|
@@ -559,11 +562,12 @@ export class WindowsSession {
|
|
|
559
562
|
else {
|
|
560
563
|
const directory = path.join(this.#root, 'captures');
|
|
561
564
|
try {
|
|
562
|
-
const shot = await this.#native.capture(current, directory);
|
|
565
|
+
const shot = await this.#native.capture(current, directory, zoom);
|
|
563
566
|
if (shot.blank) imageFallback = 'PrintWindow returned a uniform (blank) image';
|
|
564
567
|
// id : repère de coordonnées natif (banc commun du 21/10, plan de Codex étape 5), distinct des captures Sky ;
|
|
565
|
-
// un point de cette image vise l'écran à frame.left + x
|
|
566
|
-
|
|
568
|
+
// un point de cette image vise l'écran à frame.left + x * frame.width / width (image réduite au plus à
|
|
569
|
+
// 1568 px et 1,15 Mpx, règle commune des produits Nexus : nativeScreen).
|
|
570
|
+
else images = [{ id: `native-${++this.#nativeShots}`, path: shot.path, width: shot.width, height: shot.height, method: shot.method, scale: shot.scale, ms: shot.ms, frame: shot.frame, windowFrame: shot.windowFrame ?? shot.frame, ...(zoom ? { zoom: true } : {}) }];
|
|
567
571
|
} catch (error) { imageFallback = error.message; }
|
|
568
572
|
await this.#pruneCaptures(directory);
|
|
569
573
|
}
|
|
@@ -604,10 +608,11 @@ export class WindowsSession {
|
|
|
604
608
|
this.#observation = observation;
|
|
605
609
|
return observation;
|
|
606
610
|
}
|
|
607
|
-
|
|
611
|
+
// zoom : zone écran [left, top, width, height] d'une capture native (lookZoom), agrandie jusqu'à ×2.
|
|
612
|
+
async observe({ mode = 'text', backend = 'auto', capture = 'sky', zoom = null } = {}) {
|
|
608
613
|
return this.#exclusive(async () => {
|
|
609
614
|
const windowChange = await this.#follow(false);
|
|
610
|
-
return this.#capture(mode, backend, { capture, windowChange });
|
|
615
|
+
return this.#capture(mode, backend, { capture, windowChange, zoom });
|
|
611
616
|
});
|
|
612
617
|
}
|
|
613
618
|
#take(observation) {
|
|
@@ -745,8 +750,9 @@ export class WindowsSession {
|
|
|
745
750
|
if (leased) return null;
|
|
746
751
|
// Point d'une capture native : clic (bouton, double) ou molette postés au contrôle sous le point ; le helper relit
|
|
747
752
|
// le cadre de la fenêtre et refuse s'il a bougé depuis la capture.
|
|
748
|
-
const f = screenshot.frame;
|
|
749
|
-
const
|
|
753
|
+
const f = screenshot.windowFrame ?? screenshot.frame;
|
|
754
|
+
const at = gardeFous.nativeScreen(screenshot, spec.point);
|
|
755
|
+
const args = { mode: 'background', x: at.x, y: at.y, frame: [f.left, f.top, f.width, f.height] };
|
|
750
756
|
if (spec.kind === 'click') Object.assign(args, { action: 'click', button: spec.button ?? 'left', count: spec.count ?? 1 });
|
|
751
757
|
else {
|
|
752
758
|
const notches = v => (v ? Math.sign(v) * Math.max(1, Math.round(Math.abs(v) / 120)) : 0);
|
|
@@ -993,8 +999,8 @@ export class WindowsSession {
|
|
|
993
999
|
// cible d'observation native ; null : pas de repère natif (Sky en secours).
|
|
994
1000
|
#nativePoint(spec, basis, selected, screenshot, key = 'point') {
|
|
995
1001
|
if (screenshot?.native && spec[key]) {
|
|
996
|
-
const f = screenshot.frame;
|
|
997
|
-
return {
|
|
1002
|
+
const f = screenshot.windowFrame ?? screenshot.frame;
|
|
1003
|
+
return { ...gardeFous.nativeScreen(screenshot, spec[key]), frame: [f.left, f.top, f.width, f.height] };
|
|
998
1004
|
}
|
|
999
1005
|
if (basis.source === 'native-uia' && selected?.bounds?.length === 4 && selected.bounds[2] > 0 && selected.bounds[3] > 0) {
|
|
1000
1006
|
const [x, y, w, h] = selected.bounds;
|
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
[diffend] Oversized file quarantined before diffing.
|
|
2
2
|
name: package/native/bin/NexusCodex.UIA.exe
|
|
3
|
-
size:
|
|
4
|
-
sha256:
|
|
3
|
+
size: 35239104 bytes
|
|
4
|
+
sha256: 5173778c736d462f737f3fc1142d09d1b160ae7cb01976203da67ab46c21954f
|
package/package.json
CHANGED