app-use 0.0.0-stage → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -1,3 +1,111 @@
|
|
|
1
|
-
#
|
|
1
|
+
# application-use
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A native, blazingly fast macOS application automation CLI designed specifically for AI agents.
|
|
4
|
+
|
|
5
|
+
Similar to Anthropic's **Computer Use**, `application-use` empowers AI agents to operate desktop applications. However, instead of relying on heavy visual inference and imprecise `(x, y)` coordinate clicks from full-screen screenshots, **application-use** provides a **textual understanding interface built directly on top of underlying macOS native APIs (Accessibility) and Apple Vision Framework**.
|
|
6
|
+
|
|
7
|
+
By operating at the OS level, it retrieves a highly structured view of the UI instantly. This approach achieves superior speed, deterministic accuracy, and significantly better effects for LLMs navigating complex desktop interfaces.
|
|
8
|
+
|
|
9
|
+
## Key Features
|
|
10
|
+
|
|
11
|
+
- **Native OS APIs**: Uses macOS native Accessibility (`AXUIElement`) for instant, reliable UI tree extraction and precise element interaction.
|
|
12
|
+
- **Vision Integration**: Powered by Apple's built-in Vision framework for robust OCR and screen analysis, easily capturing and interacting with elements that standard APIs miss.
|
|
13
|
+
- **AI-Optimized Interface**: Converts visual spatial information into structured text representations (snapshots with alphabet hints like `JK`). LLMs can instantly map these hints back to specific actions.
|
|
14
|
+
- **Fast & Lightweight**: Built meticulously with Go and Swift (via CGO bridge) for maximum native performance without heavy dependencies.
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
## Installation
|
|
18
|
+
|
|
19
|
+
### Global Installation (recommended)
|
|
20
|
+
|
|
21
|
+
Installs the native execution binary directly from NPM:
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
npm install -g application-use@latest
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
### AI Coding Assistants (recommended)
|
|
28
|
+
|
|
29
|
+
Add the skill to your AI coding assistant for richer context:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
npx skills add qdore/application-use
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
### From Source (macOS)
|
|
36
|
+
|
|
37
|
+
Dependencies: Go (1.20+) and Xcode Command Line Tools.
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
git clone <repository-url>
|
|
41
|
+
cd application-use
|
|
42
|
+
|
|
43
|
+
# Build the Swift static bridge and Go executable
|
|
44
|
+
make clean && make
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Quick Start
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
# Search for installed apps
|
|
51
|
+
application-use search "safari"
|
|
52
|
+
|
|
53
|
+
# Open an application
|
|
54
|
+
application-use open --appName "Safari"
|
|
55
|
+
|
|
56
|
+
# Take a structural snapshot (returns an annotated accessibility tree for AI)
|
|
57
|
+
application-use snapshot --appName "Safari"
|
|
58
|
+
|
|
59
|
+
# Click an element by its hint letters (e.g., 'JK' returned from the snapshot)
|
|
60
|
+
application-use click JK --appName "Safari"
|
|
61
|
+
|
|
62
|
+
# Fill text into an element
|
|
63
|
+
application-use fill JK "hello world" --appName "Safari"
|
|
64
|
+
|
|
65
|
+
# Send keystrokes directly
|
|
66
|
+
application-use sendkey cmd+t --appName "Safari"
|
|
67
|
+
|
|
68
|
+
# Take a screenshot
|
|
69
|
+
application-use screenshot result.png --appName "Safari"
|
|
70
|
+
|
|
71
|
+
# Close the application
|
|
72
|
+
application-use close --appName "Safari"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Core Commands
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
application-use open --appName <app> # Open a specific application
|
|
79
|
+
application-use snapshot --appName <app> # Snapshot the UI tree & overlay alphabet hints on interactive elements
|
|
80
|
+
application-use click <hint> --appName <app> # Click element by hint (e.g. 'JK'). Supports: --right, --double
|
|
81
|
+
application-use fill [hint] <text> --appName <app> # Fill text into an element or current focus (uses clipboard paste)
|
|
82
|
+
application-use sendkey <key> --appName <app> # Send a key combination (e.g., cmd+v, enter, esc)
|
|
83
|
+
application-use scroll <area> <dir> [px] --appName <app> # Scroll a specific UI block (up/down/left/right)
|
|
84
|
+
application-use screenshot [path] --appName <app> # Take a visual screenshot of the window
|
|
85
|
+
application-use search [query] # Search for installed applications
|
|
86
|
+
application-use close --appName <app> # Gracefully close the application
|
|
87
|
+
application-use upgrade # Check for updates and automatically upgrade
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Agent Workflow (Recommended for AI)
|
|
91
|
+
|
|
92
|
+
Instead of relying on fragile coordinate clicks (`x,y`), **application-use** implements an LLM-friendly snapshot-and-interact pattern:
|
|
93
|
+
|
|
94
|
+
1. **Snapshot**: `application-use snapshot` queries the native Accessibility tree and layers Vision OCR data. It assigns a deterministic 2-3 letter hint (like `AA`, `JK`) to every clickable, editable, or readable element.
|
|
95
|
+
2. **Understand**: The LLM parses the printed structural tree to understand the UI layout.
|
|
96
|
+
- `(+)` markers indicate pure text elements.
|
|
97
|
+
- `(*)` markers indicate elements discovered primarily via OCR.
|
|
98
|
+
3. **Interact**: The LLM issues a command like `application-use click JK` or `application-use fill AB "admin"`. The CLI uses OS-level handles to instantly perform the action.
|
|
99
|
+
|
|
100
|
+
## Security & Permissions
|
|
101
|
+
|
|
102
|
+
Because `application-use` interacts natively with OS controls and visual outputs, it requires two macOS permissions on the first run:
|
|
103
|
+
|
|
104
|
+
1. **Accessibility**: `System Settings > Privacy & Security > Accessibility` (to read the UI tree and inject clicks/keys)
|
|
105
|
+
2. **Screen Recording**: `System Settings > Privacy & Security > Screen Recording` (required for Vision OCR and taking snapshots)
|
|
106
|
+
|
|
107
|
+
If permissions are missing, the CLI will output specialized error prompts instructing the user to enable them.
|
|
108
|
+
|
|
109
|
+
## Contributing
|
|
110
|
+
|
|
111
|
+
Currently, `application-use` is designed for macOS. We welcome pull requests to add support for other operating systems (Windows, Linux, etc.).
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Cross-platform CLI wrapper for application-use
|
|
5
|
+
*/
|
|
6
|
+
|
|
7
|
+
import { spawn } from 'child_process';
|
|
8
|
+
import { existsSync, accessSync, chmodSync, constants } from 'fs';
|
|
9
|
+
import { dirname, join } from 'path';
|
|
10
|
+
import { fileURLToPath } from 'url';
|
|
11
|
+
import { platform, arch } from 'os';
|
|
12
|
+
|
|
13
|
+
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
14
|
+
|
|
15
|
+
// Map Node.js platform/arch to binary naming convention
|
|
16
|
+
function getBinaryName() {
|
|
17
|
+
const os = platform();
|
|
18
|
+
const cpuArch = arch();
|
|
19
|
+
|
|
20
|
+
let archKey;
|
|
21
|
+
switch (cpuArch) {
|
|
22
|
+
case 'x64':
|
|
23
|
+
case 'x86_64':
|
|
24
|
+
// Linux builds use the Go-style "amd64" suffix.
|
|
25
|
+
archKey = os === 'linux' ? 'amd64' : 'x64';
|
|
26
|
+
break;
|
|
27
|
+
case 'arm64':
|
|
28
|
+
case 'aarch64':
|
|
29
|
+
archKey = 'arm64';
|
|
30
|
+
break;
|
|
31
|
+
default:
|
|
32
|
+
return null;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
if (os === 'darwin' || os === 'linux') {
|
|
36
|
+
return `application-use-${os}-${archKey}`;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
return null;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
function main() {
|
|
43
|
+
const binaryName = getBinaryName();
|
|
44
|
+
|
|
45
|
+
if (!binaryName) {
|
|
46
|
+
console.error(`Error: Unsupported platform: ${platform()}-${arch()}. application-use only supports macOS.`);
|
|
47
|
+
process.exit(1);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
const binaryPath = join(__dirname, binaryName);
|
|
51
|
+
|
|
52
|
+
if (!existsSync(binaryPath)) {
|
|
53
|
+
console.error(`Error: No binary found for ${platform()}-${arch()}`);
|
|
54
|
+
console.error(`Expected: ${binaryPath}`);
|
|
55
|
+
console.error('');
|
|
56
|
+
console.error('Please ensure the package was installed correctly or run "make package-npm" to build locally.');
|
|
57
|
+
process.exit(1);
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// Ensure binary is executable
|
|
61
|
+
try {
|
|
62
|
+
accessSync(binaryPath, constants.X_OK);
|
|
63
|
+
} catch {
|
|
64
|
+
try {
|
|
65
|
+
chmodSync(binaryPath, 0o755);
|
|
66
|
+
} catch (chmodErr) {
|
|
67
|
+
console.error(`Error: Cannot make binary executable: ${chmodErr.message}`);
|
|
68
|
+
process.exit(1);
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// Spawn the native binary with inherited stdio
|
|
73
|
+
const child = spawn(binaryPath, process.argv.slice(2), {
|
|
74
|
+
stdio: 'inherit',
|
|
75
|
+
windowsHide: false,
|
|
76
|
+
});
|
|
77
|
+
|
|
78
|
+
child.on('error', (err) => {
|
|
79
|
+
console.error(`Error executing binary: ${err.message}`);
|
|
80
|
+
process.exit(1);
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
child.on('close', (code) => {
|
|
84
|
+
process.exit(code ?? 0);
|
|
85
|
+
});
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
main();
|
package/package.json
CHANGED
|
@@ -1,6 +1,26 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "app-use",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.6",
|
|
4
|
+
"description": "macOS Desktop Automation CLI for AI agents",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"files": [
|
|
7
|
+
"bin"
|
|
8
|
+
],
|
|
9
|
+
"bin": {
|
|
10
|
+
"application-use": "bin/application-use.js"
|
|
11
|
+
},
|
|
12
|
+
"scripts": {
|
|
13
|
+
"build": "make package-npm",
|
|
14
|
+
"prepublishOnly": "npm run build",
|
|
15
|
+
"test": "echo \"Error: no test specified\" && exit 1"
|
|
16
|
+
},
|
|
17
|
+
"keywords": [
|
|
18
|
+
"macos",
|
|
19
|
+
"automation",
|
|
20
|
+
"cli",
|
|
21
|
+
"agent",
|
|
22
|
+
"desktop"
|
|
23
|
+
],
|
|
24
|
+
"author": "",
|
|
25
|
+
"license": "Apache-2.0"
|
|
26
|
+
}
|