@nexus-cortex/server 4.131.7 → 4.132.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: desktop
|
|
3
|
+
description: How to drive the computer with the `desktop` tool (nexus-desktop) — keyboard first, perceive before you click, the failsafe and stop file, coordinates, typing into fields, scripts/macros, and what to do when it refuses. Load before any desktop task (native apps, dialogs, installers, anything a browser tool cannot reach).
|
|
4
|
+
triggers:
|
|
5
|
+
- desktop
|
|
6
|
+
- screen
|
|
7
|
+
- native app
|
|
8
|
+
- dialog
|
|
9
|
+
- mouse
|
|
10
|
+
- keyboard shortcut
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
# The `desktop` tool
|
|
14
|
+
|
|
15
|
+
`desktop` controls a whole screen: a private virtual display (`NEXUS_DESKTOP=virtual`) or the user's own screen (`real`, enabled only by the
|
|
16
|
+
user with `cortex desktop enable --real`). It is off unless enabled. Use the browser tools for web pages; use `desktop` for what they cannot
|
|
17
|
+
reach — native applications, system dialogs, file pickers, installers, apps inside a virtual machine.
|
|
18
|
+
|
|
19
|
+
## 1. Start every task the same way
|
|
20
|
+
1. `desktop {action:"info"}` — screen size, screenshot scale, permissions, the failsafe spot, the launch allow-list. If permissions show
|
|
21
|
+
false (macOS Screen Recording / Accessibility), stop and tell the user what to grant; mutating actions refuse without them.
|
|
22
|
+
2. `desktop {action:"context"}` — the front app, its windows, the focused element and recent events, in one cheap call.
|
|
23
|
+
3. Perceive before acting: `scan` (accessibility elements with refs) and `grab` read text and find controls WITHOUT a screenshot. Take a
|
|
24
|
+
`screenshot` or `zoom` only when there is no accessible element to read.
|
|
25
|
+
|
|
26
|
+
## 2. Fastest path first (in this order)
|
|
27
|
+
1. A native keyboard shortcut (`hotkey` catalog: app switcher, go-to-folder in any open/save dialog, menu search, save-as …).
|
|
28
|
+
2. A menu item by name (`menu`) — one call instead of several clicks.
|
|
29
|
+
3. Keyboard navigation (Tab, arrows, Enter, accelerators) with `key` / `type`.
|
|
30
|
+
4. A `run` script: several actions in ONE call; save repeated sequences as `macro`s.
|
|
31
|
+
5. Clicks last — on a `ref` from `scan` (`press`) before raw coordinates.
|
|
32
|
+
|
|
33
|
+
## 3. Coordinates and typing
|
|
34
|
+
- Coordinates are pixels in the SCREENSHOT the tool returned (scaled to fit), never your own estimate and never from an older screenshot
|
|
35
|
+
after the window moved. Re-`scan` or re-screenshot after anything that changes the layout.
|
|
36
|
+
- Fill a field with `fill_field` (click, select-all, type, commit, then a zoom of the field to verify) or `set_value` on a scanned ref — not
|
|
37
|
+
a click followed by free typing.
|
|
38
|
+
- Dates and times: type the value into the field; never page through a calendar widget.
|
|
39
|
+
- If typed characters drop or garble (some apps and virtual machines), retry with `type_via:"paste"` or a `char_delay_ms` of 40–80.
|
|
40
|
+
|
|
41
|
+
## 4. Safety (the user's rules, enforced by the tool)
|
|
42
|
+
- FAILSAFE: if the user parks the cursor at the failsafe spot (`info` names it), the tool writes a stop file and REFUSES every further action.
|
|
43
|
+
Do not try to work around it: tell the user the desktop is stopped and that deleting the stop file resumes it.
|
|
44
|
+
- `launch` opens only allow-listed apps; ask the user instead of hunting for another way to start a program.
|
|
45
|
+
- Every action is written to an audit log (typed text by length only). Never type secrets the user did not give you for that field.
|
|
46
|
+
- On the user's real screen, say what you are about to do before a multi-step action, keep focus on the target window, and never close or
|
|
47
|
+
quit apps you did not open.
|
|
48
|
+
|
|
49
|
+
## 5. When something goes wrong
|
|
50
|
+
- A click "did nothing": re-`context` (a dialog may have opened, focus may have moved), then re-`scan`.
|
|
51
|
+
- A ref no longer exists: the window closed or retitled — `scan` again.
|
|
52
|
+
- An app shows no accessible elements (custom-drawn UI): drive it by screenshot + keyboard; `zoom` to read small text.
|
|
53
|
+
- `stats` summarizes recent actions and timings — use it to report what happened.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@nexus-cortex/server",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.132.0",
|
|
4
4
|
"description": "Thin Express server wrapper for Nexus Cortex core library",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -20,8 +20,8 @@
|
|
|
20
20
|
"prepack": "node ../../scripts/copy-pkg-cortex-scaffold.mjs"
|
|
21
21
|
},
|
|
22
22
|
"dependencies": {
|
|
23
|
-
"@nexus-cortex/core": "4.
|
|
24
|
-
"@nexus-cortex/executors": "4.
|
|
23
|
+
"@nexus-cortex/core": "4.132.0",
|
|
24
|
+
"@nexus-cortex/executors": "4.132.0",
|
|
25
25
|
"chalk": "^5.3.0",
|
|
26
26
|
"cors": "^2.8.5",
|
|
27
27
|
"dotenv": "^16.4.5",
|