nexus-windows-codex 3.0.1 → 3.1.1

package/CHANGELOG.md

@@ -7,6 +7,60 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [3.1.1] - 2026-10-24 — Numbers on the image
+
+### Added
+- `look(marks="auto"|"all")` (mode both, native tree): the tree is read just before the capture, once, under the same
+  lock; each kept element gets its exact tree line number drawn on the image (digits only, built-in 5×7 font ×2, thin
+  outline), after scaling and before encoding, so at a constant size in the final image. `auto`: unnamed controls with
+  a rectangle; `all`: every element with a rectangle. Default `none`.
+- Guards: the window frame is read before and after the tree and after PrintWindow; if it moved or was resized, no
+  number and the reason is given. Elements of another window (popup menu absent from PrintWindow) are never labelled;
+  a blank capture is judged before any drawing; a label that does not fit whole is omitted, never cut; labels without
+  room, past 120, or past 400 eligible elements are counted in the report.
+
+### Fixed
+- A zoom region computed on one window's image, after a dialog has been followed, is refused before any capture
+  (before, the parent's area was read in the dialog).
+- A refused or failed `look` no longer leaves the previous observation usable by `act`.
+- Live test of the foreground fallback: it had never run (always skipped while the user was active) and failed as
+  soon as it ran, 3.0.1 included, because it required `focused` after the foreground was given back. It is now judged
+  by the fixture's focus counter, the technique and the give-back, with no postcondition.
+
+### Security
+- Reviewed by the Codex agent over three rounds (cut label reported as drawn, window resized during PrintWindow, zoom
+  region applied to a followed dialog, stale observation after a refused look, silent 400 cap).
+
+## [3.1.0] - 2026-10-24 — Tk in the background, common image rule, zoom
+
+### Added
+- Tk in the background, window covered: a click at a point is sent with the mouse button marked pressed for Tk's own
+  thread only (press and release owned by a helper thread); lists scroll through their native scroll bar, read back
+  (recipe shared with NexusWindows for Claude).
+- `look(region=[x1, y1, x2, y2])`: an area of the last full native image, read at full resolution and enlarged up to
+  2x, within the same caps; the zoomed image has its own frame, so its points work with its own `screenshotId`, while
+  the move guard keeps checking the whole window frame. An area outside the window is refused.
+- No duplicate image: same WebP bytes (deterministic encoder, so same pixels), size and frame as the last image sent for
+  that window or zoom: not attached again; the reply names the id that takes points; `mode="image"` always attaches it.
+  The zoom reference only follows images the model actually received.
+- `launch` of a Chromium/Electron executable by its path: started with the switches that keep it drawn while covered.
+
+### Changed
+- Native images follow the common Nexus image rule: at most 1568 px and 1.15 MP (area-average reduction), lossless
+  WebP from a built-in VP8L encoder (no dependency; checked pixel-exact against libwebp), near-lossless above 200 KB,
+  never over 500 KB. A point on a reduced image is scaled back to the exact screen pixel, the same one the guards check.
+
+### Fixed
+- Outcome classification: once any input may have left, a failure is an unknown outcome (never replayed, no fallback,
+  accord consumed); a refusal raised before any input (applicability checks, the foreground token wait, the user's
+  pause, the emergency stop, a COM refusal before any pattern call, Chromium before its first activation message) is
+  "nothing sent", so the planned fallback stays available.
+- The emergency stop is checked between Tk scroll lines and is never masked by the end-of-scroll cleanup.
+
+### Security
+- Reviewed by the Codex agent over many rounds (partial input, held press ownership, frames of zoomed images, outcome
+  classification before and after input).
+
 ## [3.0.1] - 2026-10-23 — Family naming, release with provenance
 
 ### Changed

package/README.md

@@ -101,7 +101,7 @@ the JavaScript module cache does not reload.
 | Tool | Purpose |
 |---|---|
 | `windows(filter)` | Application windows (id, app, title, pid), like the taskbar; password managers never listed |
-| `look(window, mode)` | One window by id or a unique part of its title or app: UI Automation tree, plus a capture that works on covered windows with `mode="image"` or `"both"`; selects the window for `act` |
+| `look(window, mode, region, marks)` | One window by id or a unique part of its title or app: UI Automation tree, plus a capture that works on covered windows with `mode="image"` or `"both"`; `region` zooms on an area of the last image; `marks` draws tree line numbers on the image; selects the window for `act` |
 | `act(steps)` | One step, or up to 32 each with its own `expect`, on the window of the last `look`; stops at the first step not verified |
 | `launch(app)` | Start an application (name, path, Store app id), a folder, a document or a Settings page (`ms-settings:`) and give the foreground back |
 | `window(action, window)` | `action="close"`: one close message, read back; a "save?" prompt is reported, never answered |
@@ -195,9 +195,17 @@ its option element.
 `observe({mode:'image', capture:'native'})` captures the bound window with `PrintWindow(PW_RENDERFULLCONTENT)`
 without bringing it forward, even when covered: per-monitor DPI aware, cropped to the DWM frame, DPI-unaware windows
 drawn at their logical size and scaled. In the Codex skill, a blank, minimized or failed capture falls back to Sky's
-screenshot; the MCP server has no such fallback and reports the failure. Native images are local PNG files (32 files
-at most, removed after 10 minutes at the next capture, at server start and at a clean stop): evidence for the model,
-and the coordinate frame of a point action that names the image (`screenshotId`).
+screenshot; the MCP server has no such fallback and reports the failure. Native images follow the common Nexus image
+rule: at most 1568 px and 1.15 MP, lossless WebP from a built-in encoder (near-lossless above 200 KB, never over
+500 KB), local files (32 at most, removed after 10 minutes at the next capture, at server start and at a clean stop):
+evidence for the model, and the coordinate frame of a point action that names the image (`screenshotId`), scaled back
+to the screen pixel. `look(region=[x1, y1, x2, y2])` reads an area of the last full image at full resolution, enlarged
+up to 2x for small text; its own `screenshotId` takes points. An image identical to the last one sent for that window
+(same pixels and frame) is not attached again; the reply names the id that takes points, and `mode="image"` always
+attaches it. `look(mode="both", marks="auto"|"all")` reads the tree just before the capture and draws on the image the
+tree line number of each kept element (`auto`: unnamed controls; `all`: every element with a rectangle), at a constant
+size after scaling; no number when the window moved between the two reads, on another window's popup, or when the
+label does not fit whole. Default `none`. Numbers link the image to the tree; `act` still targets a selector or a point.
 
 ### Verified batches (`actBatch`)
 

package/lib/garde-fous.mjs

@@ -205,10 +205,25 @@ export function elementsAt(elements, x, y) {
     return w > 0 && h > 0 && x >= ex && y >= ey && x < ex + w && y < ey + h;
   });
 }
+// Point d'une capture native → pixel écran ENTIER, celui que l'entrée vise (session) et que la confirmation contrôle
+// (screenPoints) : une seule conversion, arrondie au centre de pixel et bornée au cadre (relecture de Codex du 24/10).
+export function nativeScreen(shot, point) {
+  const f = shot.frame;
+  const sx = shot.width > 0 ? f.width / shot.width : 1;
+  const sy = shot.height > 0 ? f.height / shot.height : 1;
+  return {
+    x: Math.max(f.left, Math.min(f.left + f.width - 1, Math.round(f.left + (point.x + 0.5) * sx - 0.5))),
+    y: Math.max(f.top, Math.min(f.top + f.height - 1, Math.round(f.top + (point.y + 0.5) * sy - 0.5))),
+  };
+}
 // Points écran possibles d'un point de capture Sky : origine de la capture + point, ou point ramené à la taille de la
 // fenêtre (capture mise à l'échelle). Les deux sont contrôlés : le doute ne laisse rien passer.
 export function screenPoints(point, screenshot, root) {
   const out = [];
+  if (screenshot?.native && screenshot.frame) { // capture native éventuellement réduite : le pixel même de l'entrée
+    const at = nativeScreen(screenshot, point);
+    out.push([at.x, at.y]);
+  }
   if (Number.isFinite(screenshot?.originX) && Number.isFinite(screenshot?.originY)) out.push([screenshot.originX + point.x, screenshot.originY + point.y]);
   const [rx, ry, rw, rh] = root?.bounds ?? [];
   if (rw > 0 && rh > 0 && screenshot?.width > 0 && screenshot?.height > 0) {

package/lib/mcp.mjs

@@ -2,6 +2,7 @@
 // sans Sky, exposée en outils. Aucune dépendance npm : le protocole utile (initialize, ping, tools/list, tools/call)
 // tient ici, et la chaîne d'approvisionnement reste vide comme pour le reste de l'outil.
 import { readFile } from 'node:fs/promises';
+import { createHash } from 'node:crypto';
 import { summarize } from './session.mjs';
 
 export const PROTOCOL_VERSIONS = ['2025-11-25', '2025-06-18', '2025-03-26', '2024-11-05'];
@@ -42,6 +43,10 @@ const ACTION = {
   },
 };
 const OBSERVE_MODE = { type: 'string', enum: ['text', 'image', 'both'], description: 'text (UI Automation tree, default), image (native capture), both' };
+const REGION = { type: 'array', items: { type: 'number' }, minItems: 4, maxItems: 4,
+  description: '[x1, y1, x2, y2] in pixels of the last native image of this window: that area enlarged up to 2x (small text); its own screenshotId takes points' };
+const NUMBERS = { type: 'string', enum: ['none', 'auto', 'all'],
+  description: 'Numbers drawn on the image of mode both, each the index of its tree line: none (default), auto (unnamed controls), all' };
 const WINDOW = { type: ['integer', 'string'], description: 'Window id from windows, or a unique part of its title or app; omitted: the window of the last look' };
 const READ = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
 const WRITE = { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: true };
@@ -54,9 +59,10 @@ export const TOOLS = [
     inputSchema: { type: 'object', additionalProperties: false, properties: { filter: { type: 'string' } } } },
   { name: 'look', title: 'Look at a window', annotations: { ...READ, title: 'Look at a window' },
     description: 'See a window in the background, without bringing it forward: UI Automation tree (roles, names, ids, values, states) and, ' +
-      'with mode image or both, a native capture that works even when the window is covered. Selects that window for act; ' +
+      'with mode image or both, a native capture that works even when the window is covered (WebP, at most 1568 px; ' +
+      'unchanged since the last image sent: not attached again, mode image always attaches it). Selects that window for act; ' +
       'an observation serves one act and expires after 60 s.',
-    inputSchema: { type: 'object', additionalProperties: false, properties: { window: WINDOW, mode: OBSERVE_MODE } } },
+    inputSchema: { type: 'object', additionalProperties: false, properties: { window: WINDOW, mode: OBSERVE_MODE, region: REGION, marks: NUMBERS } } },
   { name: 'act', title: 'Act in the window', annotations: { ...WRITE, title: 'Act in the window' },
     description: 'Steps on the window of the last look, in the background first (Win32 messages, UI Automation, DevTools for Chromium), ' +
       'the real mouse/keyboard only as a fallback under the shared foreground token. One step, or up to 32 each with its own expect; ' +
@@ -91,17 +97,62 @@ function errorStatus(error) {
 }
 
 // Observation lisible par le modèle : arbre et état, plus l'image native jointe (fichier local, jamais copiée ailleurs).
-async function observationContent(observation) {
+// Pas d'image en double (règle commune des produits Nexus, shared/images.md) : mêmes octets (encodeur WebP déterministe :
+// mêmes pixels), mêmes dimensions et même cadre écran que la dernière image envoyée pour cette fenêtre (ou ce zoom) →
+// non jointe, la réponse dit quel identifiant prend les points. `always` (mode image) la joint toujours. `received` :
+// appelé pour chaque image que le modèle a réellement (jointe, ou identique à la dernière envoyée), jamais pour une
+// image illisible (purgée) qui ne part pas.
+async function observationContent(observation, sent = null, always = false, received = null) {
   const summary = summarize(observation);
   const w = observation.window ?? {};
   const content = [text(`window ${w.id} "${w.title ?? ''}" (${w.app ?? '?'}) · ${observation.elements?.length ?? 0} elements · ${observation.source}`), text(summary)];
   for (const image of observation.images ?? []) {
-    try { content.push({ type: 'image', data: (await readFile(image.path)).toString('base64'), mimeType: 'image/png' }); }
-    catch { /* capture purgée entre-temps : l'arbre reste */ }
+    const mimeType = image.path.endsWith('.webp') ? 'image/webp' : 'image/png';
+    let data;
+    try { data = await readFile(image.path); }
+    catch { continue; /* capture purgée entre-temps : l'arbre reste */ }
+    const key = `${w.id}|${image.zoom ? 'zoom' : 'full'}`;
+    const print = createHash('sha256').update(JSON.stringify([image.width, image.height, image.frame])).update(data).digest('hex');
+    const last = sent?.get(key);
+    if (!always && last?.print === print) {
+      content.push(text(`Image ${image.id} unchanged since ${last.id} (same pixels and frame): not attached again; points take screenshotId ${image.id} (mode image attaches it).`));
+      received?.(image, w.id);
+      continue;
+    }
+    sent?.set(key, { print, id: image.id });
+    content.push({ type: 'image', data: data.toString('base64'), mimeType });
+    if (image.marks) content.push(text(numbersLine(image)));
+    received?.(image, w.id);
   }
   return content;
 }
 
+// Numéros dessinés sur une image jointe : lesquels, et ceux qui manquent (place, autre fenêtre, hors zone).
+function numbersLine(image) {
+  const m = image.marks;
+  if (m.reason) return `Image ${image.id}: ${m.reason}.`;
+  const parts = [`Image ${image.id}: numbers = tree line indexes, drawn for ${m.drawn?.length ?? 0} element(s)`];
+  if (m.omitted?.length) parts.push(`not drawn for lack of room: ${m.omitted.join(', ')}`);
+  if (m.foreign?.length) parts.push(`other windows (not in this capture): ${m.foreign.join(', ')}`);
+  if (m.outside) parts.push(`${m.outside} outside the captured area`);
+  if (m.capped) parts.push(`${m.capped} more element(s) not considered (limit 400)`);
+  return parts.join('; ') + '.';
+}
+
+// Zone [x1, y1, x2, y2] (pixels inclus) d'une image native → zone écran [left, top, width, height] : bords des pixels de
+// l'image ramenés au cadre (même échelle que les points), bornés au cadre.
+export function regionToScreen(image, region) {
+  if (!image?.frame || !(image.width > 0) || !(image.height > 0) || !Array.isArray(region) || region.length !== 4 || !region.every(Number.isFinite)) {
+    throw new Error('region needs [x1, y1, x2, y2] and a native image of this window (look with mode image or both first)');
+  }
+  const f = image.frame, sx = f.width / image.width, sy = f.height / image.height;
+  const [x1, y1, x2, y2] = region;
+  const left = Math.max(0, Math.floor(Math.min(x1, x2) * sx)), top = Math.max(0, Math.floor(Math.min(y1, y2) * sy));
+  const right = Math.min(f.width, Math.ceil((Math.max(x1, x2) + 1) * sx)), bottom = Math.min(f.height, Math.ceil((Math.max(y1, y2) + 1) * sy));
+  if (right - left < 2 || bottom - top < 2) throw new Error('region is empty or outside the image');
+  return [f.left + left, f.top + top, right - left, bottom - top];
+}
+
 function stepLine(step) {
   const how = [step.technique, step.foregroundTaken ? 'foreground taken' : null].filter(Boolean).join(', ');
   return `${mark(step.status)} ${step.kind} — ${step.status}${how ? ` (${how})` : ''}${step.error ? `: ${step.error}` : ''}`;
@@ -109,6 +160,16 @@ function stepLine(step) {
 
 export class WindowsTools {
   #session; #observation = null; #bound = null;
+  #sent = new Map(); // fenêtre|full ou zoom → dernière image envoyée (empreinte, id)
+  #images = new Map(); // fenêtre observée → dernière image native complète envoyée (repère des zones de zoom)
+  #observed = new Map(); // fenêtre liée → fenêtre réellement observée (dialogue suivi)
+  // Toute observation renvoyée (look, act avec refresh) : fenêtre réellement observée ; son image complète devient le
+  // repère des régions seulement si le modèle l'a reçue (#received, relecture de Codex du 24/10).
+  #remember(observation) {
+    const id = observation?.window?.id;
+    if (id !== undefined && this.#bound?.id !== undefined) this.#observed.set(this.#bound.id, id);
+  }
+  #received = (image, windowId) => { if (!image.zoom && windowId !== undefined) this.#images.set(windowId, image); };
   constructor(session) { this.#session = session; }
   async #choose(window) {
     const listed = await this.#session.windows();
@@ -148,9 +209,15 @@ export class WindowsTools {
         return [text(wanted ? listed.filter(w => w.title.toLowerCase().includes(wanted) || w.app.toLowerCase().includes(wanted)) : listed)];
       }
       case 'look': {
+        this.#observation = null; // refus ou échec : plus d'observation ancienne pour act (relecture de Codex du 24/10)
         await this.#bind(args.window);
-        this.#observation = await s.observe({ mode: args.mode ?? 'text', backend: 'native', capture: 'native' });
-        return observationContent(this.#observation);
+        // Repère : dernière image complète envoyée de la fenêtre que cette liaison observe (dialogue suivi compris).
+        const shown = this.#observed.get(this.#bound?.id) ?? this.#bound?.id;
+        const zoom = args.region === undefined ? null : regionToScreen(this.#images.get(shown), args.region);
+        const mode = zoom && (args.mode ?? 'text') === 'text' ? 'both' : args.mode ?? 'text';
+        this.#observation = await s.observe({ mode, backend: 'native', capture: 'native', zoom, ...(zoom ? { zoomWindow: shown } : {}), ...(args.marks && args.marks !== 'none' ? { marks: args.marks } : {}) });
+        this.#remember(this.#observation);
+        return observationContent(this.#observation, this.#sent, mode === 'image', this.#received);
       }
       case 'act': {
         if (!this.#observation) throw new Error('Look at the window first (look)');
@@ -172,7 +239,10 @@ export class WindowsTools {
         const lines = report.steps.map(stepLine).join('\n');
         const content = [text(lines + (this.#observation ? '' : '\nNo fresh observation: look again before the next act.'))];
         // Observation rafraîchie : arbre et, demandée par refresh image|both, l'image native jointe comme pour look.
-        if (report.observation) content.push(...(await observationContent(report.observation)).slice(1));
+        if (report.observation) {
+          this.#remember(report.observation);
+          content.push(...(await observationContent(report.observation, this.#sent, false, this.#received)).slice(1));
+        }
         return content;
       }
       case 'launch': {

package/lib/native.mjs

@@ -5,24 +5,30 @@ import { policyPath } from './garde-fous.mjs';
 
 // Arrêt d'urgence rapporté par le helper : avant toute entrée (refus, rien d'envoyé), ou APRÈS une entrée partielle
 // (« Emergency stop after input: k of n characters sent ») : issue inconnue, accord consommé, jamais rejoué.
-export const stopRefusal = (message, error) => {
+export const stopRefusal = (message, error, inputStarted = false) => {
   if (error !== 'EmergencyStopException') return null;
   const partial = /^Emergency stop after input: (\d+) of (\d+) characters sent/.exec(message ?? '');
-  return Object.assign(new Error(message), { name: 'EmergencyStopError', emergencyStop: true, deferred: !partial,
-    ...(partial ? { afterInput: true, typed: { characters: Number(partial[1]), total: Number(partial[2]) } } : {}) });
+  const after = Boolean(partial) || inputStarted === true; // helper : entrée déjà partie (inputStarted)
+  return Object.assign(new Error(message), { name: 'EmergencyStopError', emergencyStop: true, deferred: !after,
+    ...(after ? { afterInput: true } : {}),
+    ...(partial ? { typed: { characters: Number(partial[1]), total: Number(partial[2]) } } : {}) });
 };
 
 // Erreurs du helper. « notApplicable » : aucune entrée envoyée, la technique suivante est permise. « refused » : aucune
 // entrée envoyée et aucun repli (mot de passe, application exclue, identité changée, fenêtre fermée, cible désactivée).
 // « julienActive » : Julien a repris clavier ou souris pendant l'attente du jeton, rien n'a été envoyé. Toute autre
 // erreur d'action peut suivre une entrée : résultat inconnu, jamais rejoué.
-function nativeError({ message, error }) {
+// inputStarted (helper) : une entrée est déjà partie ; aucune de ces natures « rien n'a été envoyé » ne s'applique plus,
+// l'issue est inconnue et terminale (relecture de Codex du 24/10 : sinon un repli physique rejouait le geste).
+function nativeError({ message, error, inputStarted }) {
+  const after = inputStarted === true;
   return Object.assign(new Error(message || 'Native UIA failed'), {
     nativeError: error ?? null,
-    deferred: error === 'TimeoutException' || error === 'JulienActiveException',
-    julienActive: error === 'JulienActiveException',
-    notApplicable: error === 'BackgroundNotApplicableException' || error === 'CaptureUnavailableException',
-    refused: error === 'ActionRefusedException',
+    deferred: !after && (error === 'TimeoutException' || error === 'JulienActiveException'),
+    julienActive: !after && error === 'JulienActiveException',
+    notApplicable: !after && (error === 'BackgroundNotApplicableException' || error === 'CaptureUnavailableException'),
+    refused: !after && error === 'ActionRefusedException',
+    ...(after ? { afterInput: true } : {}),
   });
 }
 
@@ -103,7 +109,7 @@ export class NativeUIA {
           this.#idle = setTimeout(() => this.close(), this.idleTimeoutMs);
           this.#idle.unref();
           if (response.result.ok) pending.resolve(response.result);
-          else pending.reject(stopRefusal(response.result.message, response.result.error) ?? nativeError(response.result));
+          else pending.reject(stopRefusal(response.result.message, response.result.error, response.result.inputStarted) ?? nativeError(response.result));
         } catch (error) { fail(error); return; }
       }
     });
@@ -132,7 +138,8 @@ export class NativeUIA {
   }
   inspect(window) { return this.call(window, 'inspect'); }
   // Lectures sans entrée (permises pendant l'arrêt d'urgence).
-  capture(window, directory) { return this.call(window, 'capture', { directory }); }
+  // marks : numéros à dessiner ({ marks: [{index, bounds, scope}], treeFrame }), lus avec l'arbre juste avant (session.mjs).
+  capture(window, directory, zoom, marks = null) { return this.call(window, 'capture', { directory, ...(zoom ? { zoom } : {}), ...(marks ?? {}) }); }
   related(window) { return this.call(window, 'related'); }
   alive(window) { return this.call(window, 'alive'); }
   // Page des Paramètres (ms-settings: seul), ouverte sous le jeton et premier plan rendu ; rend la fenêtre notée.

package/lib/session.mjs

@@ -2,7 +2,7 @@ import { randomUUID } from 'node:crypto';
 import { mkdir, writeFile, rename, stat, unlink, readdir } from 'node:fs/promises';
 import path from 'node:path';
 import { ApplicationMemory } from './memory.mjs';
-import { parseTree, uniqueElement, findElements, structureHash, stateDigest, checkExpectation, validateExpectation, digest } from './tree.mjs';
+import { parseTree, markCandidates, uniqueElement, findElements, structureHash, stateDigest, checkExpectation, validateExpectation, digest } from './tree.mjs';
 import { emergencyStop } from './stop.mjs';
 import { JulienActiveError, PresenceUnknownError, QUIET_MS, MAX_WAIT_MS } from './presence.mjs';
 import * as gardeFous from './garde-fous.mjs';
@@ -105,7 +105,10 @@ const actionFields = {
 function nativeShot(observation, id) {
   const image = (observation.images ?? []).find(i => i.id === id);
   if (!image?.frame) return undefined;
-  return { id, width: image.width, height: image.height, originX: image.frame.left, originY: image.frame.top, frame: image.frame, native: true };
+  // frame : zone que couvre l'image (points) ; windowFrame : cadre entier de la fenêtre (garde de mouvement du helper),
+  // distincts pour une image zoomée (relecture de Codex du 24/10).
+  return { id, width: image.width, height: image.height, originX: image.frame.left, originY: image.frame.top, frame: image.frame,
+    windowFrame: image.windowFrame ?? image.frame, native: true };
 }
 function validateAction(spec) {
   if (!spec || typeof spec !== 'object' || Array.isArray(spec) || !Object.hasOwn(actionFields, spec.kind)) throw new Error('Unsupported action');
@@ -525,7 +528,7 @@ export class WindowsSession {
   }
   async #pruneCaptures(directory) {
     let files;
-    try { files = (await readdir(directory)).filter(f => f.endsWith('.png')); } catch { return; }
+    try { files = (await readdir(directory)).filter(f => f.endsWith('.png') || f.endsWith('.webp')); } catch { return; }
     const dated = [];
     for (const name of files) {
       try { dated.push({ file: path.join(directory, name), at: (await stat(path.join(directory, name))).mtimeMs }); } catch { /* déjà retiré */ }
@@ -536,12 +539,13 @@ export class WindowsSession {
       if (i >= CAPTURE_KEEP || now - entry.at > CAPTURE_RETENTION_MS) await unlink(entry.file).catch(() => {});
     }
   }
-  async #capture(mode, backend = 'auto', { capture = 'sky', windowChange = null } = {}) {
+  async #capture(mode, backend = 'auto', { capture = 'sky', windowChange = null, zoom = null, marks = 'none' } = {}) {
     this.#invalidate();
     if (!this.#window) throw new Error('Bind a returned window before observing');
     if (!['text', 'image', 'both'].includes(mode)) throw new Error('mode must be text, image or both');
     if (!['auto', 'sky', 'native'].includes(backend)) throw new Error('backend must be auto, sky or native');
     if (!['sky', 'native'].includes(capture)) throw new Error('capture must be sky or native');
+    if (!['none', 'auto', 'all'].includes(marks)) throw new Error('marks must be none, auto or all');
     if (backend === 'native' && !this.#native) throw new Error('Native UIA is not configured');
     const nativeOnly = this.#window.nativeOnly === true;
     if (nativeOnly) {
@@ -553,21 +557,33 @@ export class WindowsSession {
     if (!sameWindow(current, this.#window)) throw new Error('Window identity changed; bind it again');
     const useNative = nativeOnly || backend === 'native' || (backend === 'auto' && this.#nativePreferred);
     // Capture propre (PrintWindow, fenêtre recouverte, sans premier plan) ; image vide ou échec : capture Sky.
+    // Numéros [N] (shared/images.md point 5, avis de Codex du 24/10) : arbre natif lu AVANT la capture, une seule fois et
+    // sous le même verrou ; ses index et rectangles vont au helper, qui ne les dessine que si le cadre de la fenêtre est
+    // resté celui de la lecture. L'âge de l'observation part du début de cette lecture (FRESH_MS).
+    let marksNote = null, preread = null, treeStartedAt = null, candidates = null;
+    if (marks !== 'none') {
+      if (mode !== 'both') marksNote = 'numbers need mode both (tree and image together): none drawn';
+      else if (capture !== 'native' || !useNative || !this.#native?.inspect || !this.#native?.capture || !this.#root) marksNote = 'numbers need the native tree and capture: none drawn';
+      else { treeStartedAt = Date.now(); preread = await this.#native.inspect(current); }
+    }
     let images = [], imageFallback = null;
     if (capture === 'native' && mode !== 'text') {
       if (!this.#native?.capture || !this.#root) imageFallback = 'Native capture is not configured';
       else {
         const directory = path.join(this.#root, 'captures');
         try {
-          const shot = await this.#native.capture(current, directory);
+          const shot = await this.#native.capture(current, directory, zoom,
+            preread ? { marks: (candidates = markCandidates(preread.elements, marks)), treeFrame: preread.frame ?? null } : null);
           if (shot.blank) imageFallback = 'PrintWindow returned a uniform (blank) image';
           // id : repère de coordonnées natif (banc commun du 21/10, plan de Codex étape 5), distinct des captures Sky ;
-          // un point de cette image vise l'écran à frame.left + x, frame.top + y (image à la taille affichée).
-          else images = [{ id: `native-${++this.#nativeShots}`, path: shot.path, width: shot.width, height: shot.height, method: shot.method, scale: shot.scale, ms: shot.ms, frame: shot.frame }];
+          // un point de cette image vise l'écran à frame.left + x * frame.width / width (image réduite au plus à
+          // 1568 px et 1,15 Mpx, règle commune des produits Nexus : nativeScreen).
+          else images = [{ id: `native-${++this.#nativeShots}`, path: shot.path, width: shot.width, height: shot.height, method: shot.method, scale: shot.scale, ms: shot.ms, frame: shot.frame, windowFrame: shot.windowFrame ?? shot.frame, ...(zoom ? { zoom: true } : {}), ...(preread ? { marks: { ...(shot.marks ?? { reason: 'the helper drew no numbers' }), ...(candidates?.capped ? { capped: candidates.capped } : {}) } } : {}) }];
         } catch (error) { imageFallback = error.message; }
         await this.#pruneCaptures(directory);
       }
     }
+    if (preread && !images.length) marksNote = 'no native image: none drawn';
     const skyText = mode !== 'image' && !useNative;
     const skyImage = mode !== 'text' && !images.length && !nativeOnly;
     const state = !skyText && !skyImage
@@ -584,14 +600,14 @@ export class WindowsSession {
     const nativeRead = mode !== 'image' && this.#native?.inspect && (useNative || (!elements.length && backend === 'auto'));
     let revision = nativeRead ? null : await this.#getRevision(state.window.app); // une lecture native donne la sienne
     if (nativeRead) {
-      const native = await this.#native.inspect(state.window);
+      const native = preread ?? await this.#native.inspect(state.window);
       accessibility = native.accessibility; elements = native.elements; source = 'native-uia';
       revision = native.revision;
       this.#nativePreferred = true;
     }
     else if (mode !== 'image') this.#nativePreferred = false;
     const observation = freeze({
-      token: randomUUID(), epoch: this.#coordinator.epoch, capturedAt: Date.now(), mode, source, revision,
+      token: randomUUID(), epoch: this.#coordinator.epoch, capturedAt: treeStartedAt ?? Date.now(), mode, source, revision,
       window: clone(state.window), windowStack: this.#stack.map(w => w.id), windowChange: clone(windowChange), popups: clone(this.#popups),
       elements, structureHash: structureHash(elements),
       accessibility: clone(accessibility),
@@ -599,15 +615,23 @@ export class WindowsSession {
       screenshots: (state.screenshots ?? []).map(({ id, width, height, originX, originY, zIndex }) =>
         ({ id, width, height, originX, originY, zIndex })),
       // Captures natives : fichiers locaux (LocalAppData) à regarder avec view_image ; jamais des cibles de coordonnées.
-      images, imageFallback,
+      images, imageFallback, ...(marksNote ? { marksNote } : {}),
     });
     this.#observation = observation;
     return observation;
   }
-  async observe({ mode = 'text', backend = 'auto', capture = 'sky' } = {}) {
+  // zoom : zone écran [left, top, width, height] d'une capture native (lookZoom), agrandie jusqu'à ×2.
+  // marks : none (défaut), auto (contrôles sans nom), all ; mode both et lecture native seulement (sinon marksNote).
+  // zoomWindow : fenêtre de l'image d'où vient la zone ; un dialogue suivi entre-temps la rend caduque (refus, rien lu
+  // dans un autre repère ; relecture de Codex du 24/10).
+  async observe({ mode = 'text', backend = 'auto', capture = 'sky', zoom = null, marks = 'none', zoomWindow = null } = {}) {
     return this.#exclusive(async () => {
+      this.#invalidate(); // un refus ci-dessous ne laisse aucune observation utilisable (relecture de Codex du 24/10)
       const windowChange = await this.#follow(false);
-      return this.#capture(mode, backend, { capture, windowChange });
+      if (zoom && zoomWindow != null && this.#window?.id !== zoomWindow) {
+        throw new Error(`region comes from an image of window ${zoomWindow}, but the observed window is now ${this.#window?.id}: look again without region`);
+      }
+      return this.#capture(mode, backend, { capture, windowChange, zoom, marks });
     });
   }
   #take(observation) {
@@ -745,8 +769,9 @@ export class WindowsSession {
       if (leased) return null;
       // Point d'une capture native : clic (bouton, double) ou molette postés au contrôle sous le point ; le helper relit
       // le cadre de la fenêtre et refuse s'il a bougé depuis la capture.
-      const f = screenshot.frame;
-      const args = { mode: 'background', x: f.left + spec.point.x, y: f.top + spec.point.y, frame: [f.left, f.top, f.width, f.height] };
+      const f = screenshot.windowFrame ?? screenshot.frame;
+      const at = gardeFous.nativeScreen(screenshot, spec.point);
+      const args = { mode: 'background', x: at.x, y: at.y, frame: [f.left, f.top, f.width, f.height] };
       if (spec.kind === 'click') Object.assign(args, { action: 'click', button: spec.button ?? 'left', count: spec.count ?? 1 });
       else {
         const notches = v => (v ? Math.sign(v) * Math.max(1, Math.round(Math.abs(v) / 120)) : 0);
@@ -993,8 +1018,8 @@ export class WindowsSession {
   // cible d'observation native ; null : pas de repère natif (Sky en secours).
   #nativePoint(spec, basis, selected, screenshot, key = 'point') {
     if (screenshot?.native && spec[key]) {
-      const f = screenshot.frame;
-      return { x: Math.round(f.left + spec[key].x), y: Math.round(f.top + spec[key].y), frame: [f.left, f.top, f.width, f.height] };
+      const f = screenshot.windowFrame ?? screenshot.frame;
+      return { ...gardeFous.nativeScreen(screenshot, spec[key]), frame: [f.left, f.top, f.width, f.height] };
     }
     if (basis.source === 'native-uia' && selected?.bounds?.length === 4 && selected.bounds[2] > 0 && selected.bounds[3] > 0) {
       const [x, y, w, h] = selected.bounds;
@@ -1486,5 +1511,6 @@ export function summarize(observation) {
     windowStack: observation.windowStack, windowChange: observation.windowChange, popups: observation.popups,
     tree: observation.accessibility?.tree ?? '', focused: observation.accessibility?.focused_element ?? '',
     screenshots: observation.screenshots, images: observation.images, imageFallback: observation.imageFallback,
+    ...(observation.marksNote ? { marksNote: observation.marksNote } : {}),
   };
 }

package/lib/tree.mjs

@@ -20,6 +20,31 @@ const stateFlags = new Set(['disabled', 'selected', 'unselected', 'selectable',
   'settable', 'string', 'float', 'integer', 'boolean']);
 export const digest = value => createHash('sha256').update(value).digest('hex');
 
+// Numéros [N] sur l'image (shared/images.md point 5) : éléments natifs avec un rectangle, hors racine et hors éléments
+// dits hors écran. auto : contrôles sans nom (espaces compris) qu'on peut viser ; all : tous. scope : HWND porteur le plus
+// proche (l'élément ou un ancêtre), que le helper compare à la fenêtre capturée (menu surgissant d'une autre fenêtre :
+// jamais d'étiquette fantôme). Index EXACT de la ligne de l'arbre, jamais renuméroté.
+const MARKS_LIMIT = 400;
+const MARKABLE = new Set(['button', 'splitbutton', 'edit', 'checkbox', 'radio', 'combobox', 'listitem', 'menuitem', 'tab',
+  'treeitem', 'link', 'slider', 'spinner', 'dataitem', 'headeritem', 'image']);
+export function markCandidates(elements = [], marks = 'auto') {
+  const byIndex = new Map(elements.map(e => [e.index, e]));
+  const out = [];
+  let capped = 0; // éléments admissibles au-delà du plafond : dits dans le rapport (all n'est alors pas exhaustif)
+  for (const e of elements) {
+    const b = e.bounds;
+    if (!Array.isArray(b) || b.length !== 4 || !b.every(Number.isFinite) || !(b[2] > 0) || !(b[3] > 0)) continue;
+    if (!e.ancestors?.length || e.flags?.includes('offscreen')) continue;
+    if (marks === 'auto' && (String(e.name ?? '').trim() || !(MARKABLE.has(e.role) || e.invoke === true))) continue;
+    const chain = [e, ...[...e.ancestors].reverse().map(i => byIndex.get(i))];
+    const scope = chain.find(x => x?.nativeWindow)?.nativeWindow ?? 0;
+    if (out.length >= MARKS_LIMIT) { capped++; continue; }
+    out.push({ index: e.index, bounds: b.map(Math.round), scope });
+  }
+  if (capped) Object.defineProperty(out, 'capped', { value: capped }); // hors JSON : seul le compte part au rapport
+  return out;
+}
+
 export function parseTree(tree = '') {
   const elements = [], stack = [];
   for (const line of tree.split(/\r?\n/)) {

package/native/bin/NexusCodex.UIA.exe

@@ -1,4 +1,4 @@
 [diffend] Oversized file quarantined before diffing.
 name: package/native/bin/NexusCodex.UIA.exe
-size: 35227685 bytes
-sha256: 1bbe1a15169b998562a5755774cae122e3720ec73f46cba4ede97a30a70920c9
+size: 35244346 bytes
+sha256: e348309a77c415cd4e4102bb6f20045b413589c79a48d815560ba99d67273a68

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nexus-windows-codex",
-  "version": "3.0.1",
+  "version": "3.1.1",
   "description": "NexusWindows for Codex: Windows computer use in the background for OpenAI Codex (MCP server and Codex skill).",
   "type": "module",
   "license": "MIT",