nexus-windows-codex 3.1.2 → 3.1.3

package/CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [3.1.3] - 2026-10-25 — Dialogs, file choosers, form fields
+
+### Added
+- `windows` also lists open dialogs (a Chrome or Edge "Open" box: owned, without WS_EX_APPWINDOW, absent before) with
+  `dialogOf`, their owner; by title the dialog is chosen, by app name the main window comes first.
+- `setup.ps1`: one-step setup for people who never used a terminal (`irm …/setup.ps1 | iex`): installs Node.js LTS
+  through winget when Node 22+ is missing, downloads the server from npm and checks that it answers
+  (`initialize`, `tools/list`), then registers it in Codex's `config.toml` if it is not there yet; safe to run again.
+- Codex plugin (`.codex-plugin/plugin.json`, `.mcp.json`) and the repository as its own plugin marketplace
+  (`codex plugin marketplace add pironjulien/NexusWindows-Codex`).
+
+### Fixed
+- A click on a file chooser trigger ("Choose file", "Browse…", "Upload") is verified only when its dialog opens,
+  waited for about 4 s (Edge opens it after more than 1.6 s): before, the click came back "observed" and the dialog
+  appeared later; without a dialog the step is `unmet` (`fileChooser: not-opened`), never replayed.
+- A line break typed in a form field was refused because of a nearby help button ("Why can't I send this item?"):
+  a neighbour of the field now counts only as a command button (5 words at most, no "?"); a composer's Send button
+  still requires confirmation. The refusal names the field.
+
+### Changed
+- README: installation rewritten for beginners (one step, plugin, Codex app settings, `codex mcp add`); `npx` is
+  started directly by Codex on Windows, no `cmd /c` needed.
+
 ## [3.1.2] - 2026-10-25 — Docs that match the code
 
 ### Fixed

package/README.md

@@ -62,18 +62,47 @@ bench.
 
 ## Installation
 
-Requirements: Windows 10 or 11 and Node 22+. The npm package carries a self-contained helper: no .NET install.
+Windows 10 or 11 and the [Codex app](https://openai.com/codex/) (or the Codex CLI). The server runs on
+[Node.js](https://nodejs.org/) 22 or later; the npm package carries a self-contained helper: no .NET install.
 
-From [npm](https://www.npmjs.com/package/nexus-windows-codex), in Codex's `~/.codex/config.toml`:
+**One step** (installs Node.js LTS if missing, downloads and checks the server, registers it in Codex): open
+**Windows PowerShell** (Start menu, type `powershell`) and paste:
+
+```powershell
+irm https://raw.githubusercontent.com/pironjulien/NexusWindows-Codex/main/setup.ps1 | iex
+```
+
+Then quit Codex completely and open it again. [`setup.ps1`](https://github.com/pironjulien/NexusWindows-Codex/blob/main/setup.ps1)
+is short and can be read first; running it again only does what is missing. Node.js comes from its official package
+through winget (Windows asks once for approval).
+
+**Codex plugin** (the same server; Node.js 22+ required). This repository is a plugin marketplace. In the Codex app:
+Settings → Plugins → Add → Add a marketplace, source `pironjulien/NexusWindows-Codex`, then install
+**NexusWindows** from the Plugins directory. Or in a terminal:
+
+```powershell
+codex plugin marketplace add pironjulien/NexusWindows-Codex
+codex plugin add nexus-windows@nexus-windows-codex
+```
+
+**By hand** (Node.js 22+ required, from the [official installer](https://nodejs.org/)). In the Codex app: Settings →
+Plugins → Add → Add an MCP server, name `nexus-windows`, type STDIO, command `npx`, arguments `-y` and
+`nexus-windows-codex`. Or in a terminal:
+
+```powershell
+codex mcp add nexus-windows -- npx -y nexus-windows-codex
+```
+
+Or in `~/.codex/config.toml` (shared by the Codex app, CLI and IDE extension):
 
 ```toml
 [mcp_servers.nexus-windows]
-command = "cmd"
-args = ["/c", "npx", "-y", "nexus-windows-codex"]
+command = "npx"
+args = ["-y", "nexus-windows-codex"]
 ```
 
-On Windows, `npx` is a `.cmd` script, hence `cmd /c`. The same command works in any MCP client
-(`"command": "cmd", "args": ["/c", "npx", "-y", "nexus-windows-codex"]`).
+Restart Codex, then ask: "Open Notepad and write a shopping list: milk, eggs, bread." Other MCP clients use the same
+command (`npx -y nexus-windows-codex`; a client that cannot start `.cmd` scripts needs `cmd /c npx -y nexus-windows-codex`).
 
 ### Codex skill (alternative, from a clone)
 
@@ -100,7 +129,7 @@ the JavaScript module cache does not reload.
 
 | Tool | Purpose |
 |---|---|
-| `windows(filter)` | Application windows (id, app, title, pid), like the taskbar; password managers never listed |
+| `windows(filter)` | Application windows (id, app, title, pid), like the taskbar, plus open dialogs with `dialogOf` (their owner); password managers never listed |
 | `look(window, mode, region, marks)` | One window by id or a unique part of its title or app: UI Automation tree, plus a capture that works on covered windows with `mode="image"` or `"both"`; `region` zooms on an area of the last image; `marks` draws tree line numbers on the image; selects the window for `act` |
 | `act(steps)` | One step, or up to 32 each with its own `expect`, on the window of the last `look`; stops at the first step not verified |
 | `launch(app)` | Start an application (name, path, Store app id), a folder, a document or a Settings page (`ms-settings:`) and give the foreground back |
@@ -329,6 +358,8 @@ native/          .NET 8 helper: Program.cs (modes --server, --com-worker, --stop
                  ForegroundToken.cs, Win32.cs
 skill/nexus-windows/   SKILL.md and references (API, per-application notes)
 tests/           node:test suites; tests/live: real validation fixture (WinForms, Win32, WPF)
+setup.ps1        one-step setup: Node.js if missing, server from npm checked, registered in Codex
+.codex-plugin/, .mcp.json, .agents/plugins/marketplace.json   Codex plugin and its marketplace
 install.ps1      builds the helper and installs the skill
 test.ps1         offline check; -Full adds the real validation
 ```

package/lib/garde-fous.mjs

@@ -141,6 +141,14 @@ export function keyKinds(spec) {
   }).filter(Boolean);
 }
 
+// Bouton d'envoi possible d'un champ : Button ou SplitButton au libellé de commande (5 mots au plus, sans « ? »). Un lien
+// d'aide « Pourquoi ne puis-je pas envoyer cet élément ? » à côté de « Description* » d'un formulaire n'est pas ce
+// qu'Entrée déclenche (faux refus vu le 25/10 par NexusWindows for Claude) ; « Envoyer » d'un composeur le reste.
+const commandButton = (e, elements) => {
+  if (!['button', 'splitbutton'].includes(e.role)) return false;
+  const label = String(labelOf(e, elements)).trim();
+  return !label.includes('?') && label.split(/\s+/).filter(Boolean).length <= 5;
+};
 // Éléments d'action du groupe du champ : sous-arbre de son ancêtre au plus NEIGHBOR_LEVELS au-dessus, jamais la racine.
 function neighbors(field, elements, policy) {
   const ancestors = field.ancestors ?? [];
@@ -149,7 +157,7 @@ function neighbors(field, elements, policy) {
   let found = [];
   for (let levels = Math.min(NEIGHBOR_LEVELS, ancestors.length - 1); levels > 0 && levels <= ancestors.length - 1; levels++) {
     const group = ancestors[ancestors.length - levels];
-    found = elements.filter(e => e !== field && e.ancestors?.includes(group) && liveTrigger(e, policy));
+    found = elements.filter(e => e !== field && e.ancestors?.includes(group) && liveTrigger(e, policy) && commandButton(e, elements));
     if (found.some(e => policy.confirm?.test(plain(labelOf(e, elements))))) break;
   }
   return found;

package/lib/mcp.mjs

@@ -55,7 +55,7 @@ const WRITE = { readOnlyHint: false, destructiveHint: true, idempotentHint: fals
 // compte rendu d'étape commence par ✓ vérifié, = déjà fait, ~ issue inconnue (jamais rejouée), ✗ échec ou refus.
 export const TOOLS = [
   { name: 'windows', title: 'List windows', annotations: { ...READ, title: 'List windows' },
-    description: 'Application windows on the desktop (id, app, title, pid), like the taskbar. Password managers are never listed. filter: part of a title or app.',
+    description: 'Application windows on the desktop (id, app, title, pid), like the taskbar, plus open dialogs (a file "Open" box…) with dialogOf: the window that owns them; look at a dialog by its id or title. Password managers are never listed. filter: part of a title or app.',
     inputSchema: { type: 'object', additionalProperties: false, properties: { filter: { type: 'string' } } } },
   { name: 'look', title: 'Look at a window', annotations: { ...READ, title: 'Look at a window' },
     description: 'See a window in the background, without bringing it forward: UI Automation tree (roles, names, ids, values, states) and, ' +
@@ -155,7 +155,9 @@ export function regionToScreen(image, region) {
 
 function stepLine(step) {
   const how = [step.technique, step.foregroundTaken ? 'foreground taken' : null].filter(Boolean).join(', ');
-  return `${mark(step.status)} ${step.kind} — ${step.status}${how ? ` (${how})` : ''}${step.error ? `: ${step.error}` : ''}`;
+  const chooser = step.fileChooser === 'opened' ? ': file dialog opened (look at it by its title)'
+    : step.fileChooser === 'not-opened' ? ': no file dialog opened (real click sent, effect not seen); look again' : '';
+  return `${mark(step.status)} ${step.kind} — ${step.status}${how ? ` (${how})` : ''}${chooser}${step.error ? `: ${step.error}` : ''}`;
 }
 
 export class WindowsTools {
@@ -178,7 +180,15 @@ export class WindowsTools {
     else {
       const wanted = String(window).toLowerCase();
       found = listed.filter(w => w.title.toLowerCase() === wanted);
-      if (!found.length) found = listed.filter(w => w.title.toLowerCase().includes(wanted) || w.app.toLowerCase().includes(wanted));
+      if (!found.length) {
+        // Par titre, un dialogue est visé comme une autre fenêtre ; par nom d'application seulement, sa fenêtre principale
+        // passe d'abord (la boîte « Ouvrir » de Chrome ne rend pas « chrome » ambigu).
+        const byTitle = listed.filter(w => w.title.toLowerCase().includes(wanted));
+        const byApp = listed.filter(w => !byTitle.includes(w) && w.app.toLowerCase().includes(wanted));
+        const mains = byApp.filter(w => !w.dialogOf);
+        found = [...byTitle, ...mains];
+        if (!found.length) found = byApp;
+      }
     }
     if (found.length !== 1) {
       throw new Error(found.length ? `${found.length} windows match "${window}": ${found.map(w => `${w.id} "${w.title}"`).join(', ')}` : `No window matches "${window}"; list windows`);

package/lib/native-provider.mjs

@@ -11,8 +11,10 @@ export class NoSkyError extends Error {
 
 export function nativeProvider(native) {
   if (!native?.listWindows) throw new Error('The native helper is required');
+  // Dialogue (fenêtre possédée : boîte « Ouvrir » de Chrome) : son propriétaire est imposé au helper à chaque requête.
   const list = async () => (await native.listWindows(0)).windows.map(w => ({
     id: w.id, app: w.app, title: w.title ?? '', pid: w.pid, nativeOnly: true,
+    ...(w.owner ? { owner: w.owner, dialogOf: { id: w.owner, title: w.ownerTitle ?? '' } } : {}),
   }));
   const refuse = what => async () => { throw new NoSkyError(what); };
   return {

package/lib/session.mjs

@@ -90,6 +90,11 @@ const TYPE_CHUNK = 1; // frappe physique par graphème : l'arrêt d'urgence est
 const PRESENCE_POLL_MS = 50, PRESENCE_POLLS = 33; // ≈ 1,618 s d'attente du retour du garde avant refus
 // 'key' : touches et raccourcis Ctrl/Maj postés en Win32/WinForms (banc commun du 21/10) ; ailleurs, non applicable → Sky.
 const BACKGROUND_KINDS = new Set(['click', 'setValue', 'typeText', 'secondary', 'select', 'menu', 'key']);
+// Déclencheur d'un sélecteur de fichiers : « Upload », « Parcourir… » seuls, ou un verbe suivi d'un objet fichier
+// (« Choisir un fichier: Aucun fichier… » de Chromium, « Choose files », « Ajouter des pièces jointes »). La boîte est
+// attendue au plus ≈ 1,618 × 2,618 s : Edge l'a ouverte après plus de 1,6 s le 25/10.
+const FILE_CHOOSER = /^\s*(?:(?:upload|browse|parcourir|t[eé]l[eé]verser)(?:\s+(?:a|the|your|un|une|des|vos))?(?:\s+(?:fichiers?|files?|documents?|images?|photos?))?|(?:choisir|s[eé]lectionner|ajouter|importer|joindre|choose|select|add|attach)(?:\s+(?:un|une|des|le|les|vos|a|the|your))?\s+(?:fichiers?|files?|documents?|images?|photos?|pi[eè]ces?\s+jointes?|attachments?))\s*(?:…|\.\.\.)?\s*(?::|$)/i;
+const CHOOSER_MS = 4236;
 // Champ destinataire (À, Cc, Cci, To, Destinataire, numéro) : montré à part dans le résumé de l'accord.
 const RECIPIENT_FIELD = /^((a|to|cc|cci|bcc)( recipients?| destinataires?)?|destinataires?|recipients?|e ?mail|courriel|adresse (e ?mail|electronique)|email address|tel|telephone|numero de telephone|phone( number)?|beneficiaire|iban)( |$)/;
 const plainName = name => String(name).normalize('NFKD').replace(/\p{M}/gu, '').toLowerCase().replace(/[^0-9a-z]+/g, ' ').trim();
@@ -893,6 +898,8 @@ export class WindowsSession {
         if (matches.length === 1) field = matches[0];
       }
       field ??= basis.filter(e => e.focused === true).sort((a, b) => b.depth - a.depth)[0] ?? null;
+      // Le refus et le résumé de l'accord nomment le champ (« saut de ligne tapé dans « Description* » »).
+      if (spec.kind === 'typeText' && field) what = what.replace(' dans le texte', ` tapé dans « ${String(gardeFous.labelOf(field, basis)).trim().slice(0, 60)} »`);
       for (const kind of kinds) fail(gardeFous.keyTriggers(basis, field, kind, policy));
     }
     if (!outward.length) return null;
@@ -1194,7 +1201,7 @@ export class WindowsSession {
     this.#observation = observation;
     return observation;
   }
-  async #settle(spec, coordinate) {
+  async #settle(spec, coordinate, { dialogMs = 0 } = {}) {
     const refresh = spec.refresh ?? (coordinate ? 'both' : 'text');
     // Fenêtre liée fermée par l'action (Alt+F4, bouton Fermer, dialogue qui ferme l'application) : état terminal relu
     // par le helper à toute profondeur de pile, avant chaque capture et après une capture en échec, sans recapturer le
@@ -1244,6 +1251,16 @@ export class WindowsSession {
     let after = await capture(spec.expect && refresh === 'image' ? 'both' : refresh, change);
     if (!after) return closed();
     if (recovered) change = recovered;
+    // Boîte attendue (sélecteur de fichiers) : lectures seules jusqu'à son ouverture ; Edge l'ouvre après plus de 1,6 s.
+    if (dialogMs > 0 && change?.reason !== 'opened') {
+      const deadline = Date.now() + dialogMs;
+      while (Date.now() < deadline && change?.reason !== 'opened') {
+        await sleep(100);
+        const followed = await follow();
+        if (followed === GONE) return closed();
+        if (followed) { change = followed; after = await capture(refresh, change); if (!after) return closed(); }
+      }
+    }
     if (spec.expect && !checkExpectation(after.elements, spec.expect)) {
       const deadline = Date.now() + (spec.timeoutMs ?? 1618);
       while (Date.now() < deadline && !checkExpectation(after.elements, spec.expect)) {
@@ -1350,6 +1367,9 @@ export class WindowsSession {
     let accord;
     try { accord = await this.#confirmation(spec, before, selected, screenshot); }
     catch (error) { if (error.refused) this.#record(spec.kind, error.confirmationRequired ? 'confirmation-required' : 'refused', Date.now() - start); throw error; }
+    // Sélecteur de fichiers (<input type=file> « Choisir un fichier », « Parcourir… », « Upload ») : boîte attendue.
+    const chooser = spec.kind === 'click' && Boolean(selected) && (spec.count ?? 1) === 1 && (spec.button ?? 'left') === 'left' &&
+      FILE_CHOOSER.test(String(gardeFous.labelOf(selected, before.elements)));
     const background = this.#backgroundPlan(spec, before, selected, coordinate, screenshot);
     // Sans technique d'arrière-plan possible, les contrôles de l'entrée physique précèdent toute consommation.
     if (!background) selected = this.#physicalTarget(spec, original, before) ?? selected;
@@ -1382,7 +1402,7 @@ export class WindowsSession {
         }
       }
       if (outcome) {
-        ({ after, change } = await this.#settle(spec, coordinate));
+        ({ after, change } = await this.#settle(spec, coordinate, { dialogMs: chooser ? CHOOSER_MS : 0 }));
         // Repli physique seulement après un « none » RELU par le helper : rien n'a pris, rien n'est parti. Après « sent »
         // (transmis sans relecture possible), un état encore inchangé ne prouve rien : l'effet peut être lent (revue du
         // 16/10 : effet arrivé à 2,5 s, après la fenêtre de 1,618 s ; le repli refaisait un vrai clic, double action
@@ -1417,7 +1437,7 @@ export class WindowsSession {
         physical = await this.#runPhysical(spec, before, selected, screenshot);
         outcome = physical.outcome;
         attempts.push({ technique: outcome.technique, mechanism: outcome.mechanism, effect: outcome.effect });
-        ({ after, change } = await this.#settle(spec, coordinate));
+        ({ after, change } = await this.#settle(spec, coordinate, { dialogMs: chooser ? CHOOSER_MS : 0 }));
       }
       // Helper du bail perdu pendant l'envoi ou la relecture (fond compris) : l'entrée a pu partir sans jeton ; inconnue
       // dès cette action, jamais « verified » (relecture de Codex, 21/10).
@@ -1440,8 +1460,12 @@ export class WindowsSession {
       this.#record(spec.kind, 'unknown', Date.now() - start, outcome?.technique ?? null, outcome?.foregroundTaken === true);
       throw new Error('Input or refresh outcome is unknown. Observe again before deciding; do not replay automatically.', { cause });
     }
-    const verified = Boolean(spec.expect && checkExpectation(after.elements, spec.expect));
-    const status = physical?.interrupted ? 'interrupted' : verified ? 'verified' : unconfirmed ? 'sent-unconfirmed' : spec.expect ? 'unmet' : 'observed';
+    // Déclencheur d'un sélecteur de fichiers : le clic n'est attesté que par la boîte ouverte et suivie ; sans elle,
+    // « unmet » (un changement d'image ne prouve rien, vu le 25/10 dans Chrome par NexusWindows for Claude).
+    const chooserOpened = chooser && change?.reason === 'opened';
+    const verified = chooser ? chooserOpened && (!spec.expect || checkExpectation(after.elements, spec.expect))
+      : Boolean(spec.expect && checkExpectation(after.elements, spec.expect));
+    const status = physical?.interrupted ? 'interrupted' : verified ? 'verified' : chooser ? 'unmet' : unconfirmed ? 'sent-unconfirmed' : spec.expect ? 'unmet' : 'observed';
     const result = freeze({
       status, verified: verified && !physical?.interrupted, performed: true, technique: outcome.technique, mechanism: outcome.mechanism,
       foregroundTaken: outcome.foregroundTaken, attempts, julienInput: Boolean(physical?.julienInput),
@@ -1451,6 +1475,7 @@ export class WindowsSession {
       ...(Object.hasOwn(outcome, 'contextMenu') ? { contextMenu: outcome.contextMenu } : {}),
       ...(physical?.interrupted ? { interrupted: physical.interrupted, typed: physical.typed } : {}),
       ...(physical ? { presence: physical.presence ?? 'none', waitedMs: physical.waitedMs ?? 0 } : {}),
+      ...(chooser ? { fileChooser: chooserOpened ? 'opened' : 'not-opened' } : {}),
       windowChange: change, durationMs: Date.now() - start, observation: after,
     });
     this.#record(spec.kind, status, result.durationMs, outcome.technique, outcome.foregroundTaken === true);

package/native/bin/NexusCodex.UIA.exe

@@ -1,4 +1,4 @@
 [diffend] Oversized file quarantined before diffing.
 name: package/native/bin/NexusCodex.UIA.exe
-size: 35244346 bytes
-sha256: 15e9d75e1756c8b0bb63c0bdaad3ee5e18aa1c5a09ab6047686b7326716c44b2
+size: 35244814 bytes
+sha256: 0ea640868ef2c16c6eb9f0b14a8593e29de71912e2c8cc5dc7e4adcd60773ab0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nexus-windows-codex",
-  "version": "3.1.2",
+  "version": "3.1.3",
   "description": "NexusWindows for Codex: Windows computer use in the background for OpenAI Codex (MCP server and Codex skill).",
   "type": "module",
   "license": "MIT",