app-use 0.0.0-stage → 0.1.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,111 @@
1
- # Temporary Holding Version
1
+ # application-use
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A native, blazingly fast macOS application automation CLI designed specifically for AI agents.
4
+
5
+ Similar to Anthropic's **Computer Use**, `application-use` empowers AI agents to operate desktop applications. However, instead of relying on heavy visual inference and imprecise `(x, y)` coordinate clicks from full-screen screenshots, **application-use** provides a **textual understanding interface built directly on top of underlying macOS native APIs (Accessibility) and Apple Vision Framework**.
6
+
7
+ By operating at the OS level, it retrieves a highly structured view of the UI instantly. This approach achieves superior speed, deterministic accuracy, and significantly better effects for LLMs navigating complex desktop interfaces.
8
+
9
+ ## Key Features
10
+
11
+ - **Native OS APIs**: Uses macOS native Accessibility (`AXUIElement`) for instant, reliable UI tree extraction and precise element interaction.
12
+ - **Vision Integration**: Powered by Apple's built-in Vision framework for robust OCR and screen analysis, easily capturing and interacting with elements that standard APIs miss.
13
+ - **AI-Optimized Interface**: Converts visual spatial information into structured text representations (snapshots with alphabet hints like `JK`). LLMs can instantly map these hints back to specific actions.
14
+ - **Fast & Lightweight**: Built meticulously with Go and Swift (via CGO bridge) for maximum native performance without heavy dependencies.
15
+
16
+
17
+ ## Installation
18
+
19
+ ### Global Installation (recommended)
20
+
21
+ Installs the native execution binary directly from NPM:
22
+
23
+ ```bash
24
+ npm install -g application-use@latest
25
+ ```
26
+
27
+ ### AI Coding Assistants (recommended)
28
+
29
+ Add the skill to your AI coding assistant for richer context:
30
+
31
+ ```bash
32
+ npx skills add qdore/application-use
33
+ ```
34
+
35
+ ### From Source (macOS)
36
+
37
+ Dependencies: Go (1.20+) and Xcode Command Line Tools.
38
+
39
+ ```bash
40
+ git clone <repository-url>
41
+ cd application-use
42
+
43
+ # Build the Swift static bridge and Go executable
44
+ make clean && make
45
+ ```
46
+
47
+ ## Quick Start
48
+
49
+ ```bash
50
+ # Search for installed apps
51
+ application-use search "safari"
52
+
53
+ # Open an application
54
+ application-use open --appName "Safari"
55
+
56
+ # Take a structural snapshot (returns an annotated accessibility tree for AI)
57
+ application-use snapshot --appName "Safari"
58
+
59
+ # Click an element by its hint letters (e.g., 'JK' returned from the snapshot)
60
+ application-use click JK --appName "Safari"
61
+
62
+ # Fill text into an element
63
+ application-use fill JK "hello world" --appName "Safari"
64
+
65
+ # Send keystrokes directly
66
+ application-use sendkey cmd+t --appName "Safari"
67
+
68
+ # Take a screenshot
69
+ application-use screenshot result.png --appName "Safari"
70
+
71
+ # Close the application
72
+ application-use close --appName "Safari"
73
+ ```
74
+
75
+ ## Core Commands
76
+
77
+ ```bash
78
+ application-use open --appName <app> # Open a specific application
79
+ application-use snapshot --appName <app> # Snapshot the UI tree & overlay alphabet hints on interactive elements
80
+ application-use click <hint> --appName <app> # Click element by hint (e.g. 'JK'). Supports: --right, --double
81
+ application-use fill [hint] <text> --appName <app> # Fill text into an element or current focus (uses clipboard paste)
82
+ application-use sendkey <key> --appName <app> # Send a key combination (e.g., cmd+v, enter, esc)
83
+ application-use scroll <area> <dir> [px] --appName <app> # Scroll a specific UI block (up/down/left/right)
84
+ application-use screenshot [path] --appName <app> # Take a visual screenshot of the window
85
+ application-use search [query] # Search for installed applications
86
+ application-use close --appName <app> # Gracefully close the application
87
+ application-use upgrade # Check for updates and automatically upgrade
88
+ ```
89
+
90
+ ## Agent Workflow (Recommended for AI)
91
+
92
+ Instead of relying on fragile coordinate clicks (`x,y`), **application-use** implements an LLM-friendly snapshot-and-interact pattern:
93
+
94
+ 1. **Snapshot**: `application-use snapshot` queries the native Accessibility tree and layers Vision OCR data. It assigns a deterministic 2-3 letter hint (like `AA`, `JK`) to every clickable, editable, or readable element.
95
+ 2. **Understand**: The LLM parses the printed structural tree to understand the UI layout.
96
+ - `(+)` markers indicate pure text elements.
97
+ - `(*)` markers indicate elements discovered primarily via OCR.
98
+ 3. **Interact**: The LLM issues a command like `application-use click JK` or `application-use fill AB "admin"`. The CLI uses OS-level handles to instantly perform the action.
99
+
100
+ ## Security & Permissions
101
+
102
+ Because `application-use` interacts natively with OS controls and visual outputs, it requires two macOS permissions on the first run:
103
+
104
+ 1. **Accessibility**: `System Settings > Privacy & Security > Accessibility` (to read the UI tree and inject clicks/keys)
105
+ 2. **Screen Recording**: `System Settings > Privacy & Security > Screen Recording` (required for Vision OCR and taking snapshots)
106
+
107
+ If permissions are missing, the CLI will output specialized error prompts instructing the user to enable them.
108
+
109
+ ## Contributing
110
+
111
+ Currently, `application-use` is designed for macOS. We welcome pull requests to add support for other operating systems (Windows, Linux, etc.).
Binary file
Binary file
Binary file
@@ -0,0 +1,88 @@
1
+ #!/usr/bin/env node
2
+
3
+ /**
4
+ * Cross-platform CLI wrapper for application-use
5
+ */
6
+
7
+ import { spawn } from 'child_process';
8
+ import { existsSync, accessSync, chmodSync, constants } from 'fs';
9
+ import { dirname, join } from 'path';
10
+ import { fileURLToPath } from 'url';
11
+ import { platform, arch } from 'os';
12
+
13
+ const __dirname = dirname(fileURLToPath(import.meta.url));
14
+
15
+ // Map Node.js platform/arch to binary naming convention
16
+ function getBinaryName() {
17
+ const os = platform();
18
+ const cpuArch = arch();
19
+
20
+ let archKey;
21
+ switch (cpuArch) {
22
+ case 'x64':
23
+ case 'x86_64':
24
+ // Linux builds use the Go-style "amd64" suffix.
25
+ archKey = os === 'linux' ? 'amd64' : 'x64';
26
+ break;
27
+ case 'arm64':
28
+ case 'aarch64':
29
+ archKey = 'arm64';
30
+ break;
31
+ default:
32
+ return null;
33
+ }
34
+
35
+ if (os === 'darwin' || os === 'linux') {
36
+ return `application-use-${os}-${archKey}`;
37
+ }
38
+
39
+ return null;
40
+ }
41
+
42
+ function main() {
43
+ const binaryName = getBinaryName();
44
+
45
+ if (!binaryName) {
46
+ console.error(`Error: Unsupported platform: ${platform()}-${arch()}. application-use only supports macOS.`);
47
+ process.exit(1);
48
+ }
49
+
50
+ const binaryPath = join(__dirname, binaryName);
51
+
52
+ if (!existsSync(binaryPath)) {
53
+ console.error(`Error: No binary found for ${platform()}-${arch()}`);
54
+ console.error(`Expected: ${binaryPath}`);
55
+ console.error('');
56
+ console.error('Please ensure the package was installed correctly or run "make package-npm" to build locally.');
57
+ process.exit(1);
58
+ }
59
+
60
+ // Ensure binary is executable
61
+ try {
62
+ accessSync(binaryPath, constants.X_OK);
63
+ } catch {
64
+ try {
65
+ chmodSync(binaryPath, 0o755);
66
+ } catch (chmodErr) {
67
+ console.error(`Error: Cannot make binary executable: ${chmodErr.message}`);
68
+ process.exit(1);
69
+ }
70
+ }
71
+
72
+ // Spawn the native binary with inherited stdio
73
+ const child = spawn(binaryPath, process.argv.slice(2), {
74
+ stdio: 'inherit',
75
+ windowsHide: false,
76
+ });
77
+
78
+ child.on('error', (err) => {
79
+ console.error(`Error executing binary: ${err.message}`);
80
+ process.exit(1);
81
+ });
82
+
83
+ child.on('close', (code) => {
84
+ process.exit(code ?? 0);
85
+ });
86
+ }
87
+
88
+ main();
package/package.json CHANGED
@@ -1,6 +1,26 @@
1
1
  {
2
2
  "name": "app-use",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.1.5",
4
+ "description": "macOS Desktop Automation CLI for AI agents",
5
+ "type": "module",
6
+ "files": [
7
+ "bin"
8
+ ],
9
+ "bin": {
10
+ "application-use": "bin/application-use.js"
11
+ },
12
+ "scripts": {
13
+ "build": "make package-npm",
14
+ "prepublishOnly": "npm run build",
15
+ "test": "echo \"Error: no test specified\" && exit 1"
16
+ },
17
+ "keywords": [
18
+ "macos",
19
+ "automation",
20
+ "cli",
21
+ "agent",
22
+ "desktop"
23
+ ],
24
+ "author": "",
25
+ "license": "Apache-2.0"
26
+ }