@celljs/ta-computer-use 0.0.0-stage → 4.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.en.md ADDED
@@ -0,0 +1,37 @@
1
+ # Cell - Ta Computer Use
2
+
3
+ A built-in pi package for Ta (enabled by default, can be disabled in Web client Settings → Extensions): lets Ta use desktop applications on this machine via [Cua Driver](https://cua.ai/cua-driver) in the background. It lists apps and windows, reads window accessibility trees, performs clicks, types text, scrolls, and takes screenshots, without taking your mouse and keyboard. It's just one pi extension (`src/extension.ts` → `extension/index.mjs`), does not modify Ta's core, and pi can use it as well.
4
+
5
+ [Ta Desktop](../../apps/ta-desktop/README.md) comes with Cua Driver (fixed version) and works out of the box. For the terminal Ta, use your own installation:
6
+
7
+ ```bash
8
+ /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" # macOS, Linux
9
+ irm https://cua.ai/driver/install.ps1 | iex # Windows (PowerShell)
10
+ ```
11
+
12
+ The `/computer-use` page shows which program is in use and checks what your system still needs (guiding you through setup like the first time); if nothing is installed, it shows the installation command.
13
+
14
+ ## How It Works
15
+
16
+ - **Finding the program**: first checks the environment variable `TA_CUA_DRIVER` (set by Ta Desktop when starting ta serve, pointing to its bundled copy), then checks `PATH`, `~/.local/bin` (the default install location), and on macOS `/Applications/CuaDriver.app` and `~/Applications/CuaDriver.app`, in order. It only calls `stat`, never runs anything. If not found, it checks again before each message, so installing mid-session doesn't need a restart.
17
+ - **Hint**: when found, adds a `<computer_use>` section to the system prompt: tool names are `tools.mcp__cua_driver__*`, suggests calling `describeNamespace("mcp__cua_driver")` first to see usage, plus platform limitations (macOS: don't prompt user again while waiting for permissions; Windows: can't operate windows running as administrator; Linux: needs desktop session and AT-SPI). Adds nothing if not found.
18
+ - **On-demand startup**: the MCP server `cua-driver` (exposed in codemode) is not registered at session start. The first time a codemode script mentions `mcp__cua_driver` (per pi's `reachesTools`; scripts using tools must write it out), it registers with `pi.registerMcpServer` and waits for tools to appear (up to 5 seconds). Pi connects all registered MCP servers at session start, creating a `cua-driver mcp` process per session (~10–24 MB); on-demand registration means sessions not using desktop apps have no extra process.
19
+ - **System requirements and permissions** (`src/cua-driver-access.ts`, checked once per process on first use, shared by concurrent calls):
20
+ - **macOS**: Cua Driver needs Accessibility (read and operate apps) and Screen Recording (screenshots), authorized to CuaDriver.app (`com.trycua.driver`, Cua's developer signature), not Ta. Check with `cua-driver call check_permissions` (~20 ms). When permissions are missing, CuaDriver.app shows a two-step guide; Ta posts a notification explaining which two toggles to enable, then checks every second. Once enabled, the call continues without retry. If the guide is dismissed, Ta opens the system settings page. Waits up to 3 minutes; the user can press "Stop" to cancel immediately. Without Accessibility, the call fails, and the model is told to ask the user to enable it; `/computer-use` can guide them again. Without only Screen Recording, the call continues (reading and operating work), with a one-time notification. The daemon returns `permissions_pending` for a few hundred ms after startup, so Ta silently waits up to 3 seconds to avoid false alarms.
21
+ - **Linux**: no permissions needed, but requires a graphical session (X11 or Wayland with XWayland), AT-SPI accessibility bus, and libX11 / libXi / libxkbcommon. Check with `cua-driver doctor --json` (~0.3 s). Without a graphical session (e.g., SSH) or missing libraries, the call fails and explains why plus install commands; AT-SPI warnings notify the user once and continue.
22
+ - **Windows**: no permissions needed; use `doctor` to check for desktop session. Windows running as administrator requires Ta to also run as administrator (UIPI), noted in the hint.
23
+ - When the system can't tell (your daemon hasn't started yet), the call is not blocked; it checks next time.
24
+ - **Bundled Cua Driver on macOS** (`src/cua-driver-daemon.ts`): the MCP process only forwards; the actual daemon must launch as CuaDriver.app via LaunchServices for authorization to belong to it. Ta starts a daemon on its own socket (`~/Library/Caches/ta/cua-driver.sock`), and the MCP server is `cua-driver mcp --embedded --socket <it>`. Here's how:
25
+ - An app nested inside another app bundle is treated by macOS as part of the outer app (permissions attach to Ta and are lost on each build). So on first use, Ta clones it by version to `~/Library/Application Support/Ta/cua-driver/<version>/` (APFS clone, ~30 ms, no extra disk), launches from there, and deletes older version clones.
26
+ - LaunchServices picks variants by bundle id and might pick CuaDriver.app from another app (different version, forwarding reports incompatibility). So before launch, Ta registers this copy with `lsregister -f`, then verifies the actual running program's path after launch. If it doesn't match, Ta stops it and suggests `cua-driver doctor --fix`. An existing daemon on the socket (from another Ta process or left over) is checked the same way.
27
+ - The bundled copy disables Cua's product telemetry and update checks (`CUA_DRIVER_RS_TELEMETRY_ENABLED=0`, `CUA_DRIVER_RS_UPDATE_CHECK=0`); Ta's version updates handle upgrades. Your own installation keeps your settings.
28
+ - Before each use, the daemon is confirmed alive (one socket connect); if it crashed, it restarts. After 10 minutes idle, it exits (~60 MB), and the forwarder exits too; pi auto-reconnects on the next call. When the process that started it exits, it stops the daemon too.
29
+
30
+ Measured (M4 Pro, fake model running `ta -p`, bundled Cua Driver 0.47.5, from Ta.app bundle): the first call is ~0.3 s slower (clone + daemon start). Each subsequent first call in a new process is within measurement error compared to not calling (~1.18 s vs ~1.20 s). Daemon startup ~150–250 ms, permission check ~20 ms, idle ~33–60 MB.
31
+
32
+ ## Limitations
33
+
34
+ - If `mcp.json` already has a server named `cua-driver`, the file takes precedence. Cua Driver configured under another name (e.g., from `cua-driver connect` as `cua-computer-use`) won't be recognized, and tools appear twice (you can disable this extension).
35
+ - When remote execution (`ta serve --exec`) is used, the MCP server runs in the execution environment, but the program is found on the ta serve machine. When paths differ on each side, it won't connect.
36
+ - On macOS, if the user clicks "Quit and Reopen" in the permission guide (Screen Recording requirement), macOS restarts CuaDriver.app without Ta's arguments; Ta then restarts its daemon and stops the extra instance after permissions are granted.
37
+ - Windows and Linux installer packages have not been verified on real machines; Linux's `doctor` output has been tested in containers.
package/README.md CHANGED
@@ -1,3 +1,37 @@
1
- # Temporary Holding Version
1
+ # Cell - Ta Computer Use
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Ta 自带的 pi 包(默认开启,可在 Web 客户端的 设置 → 扩展 关闭):让 Ta 经 [Cua Driver](https://cua.ai/cua-driver) 在后台使用这台电脑上的桌面应用——列出应用和窗口、读取窗口的辅助功能树、点击、输入、滚动、截图,不抢用户的鼠标和键盘。只有一个 pi 扩展(`src/extension.ts` → `extension/index.mjs`),不改 Ta 的核心,pi 也能装。
4
+
5
+ [Ta Desktop](../../apps/ta-desktop/README.md) 自带 Cua Driver(固定版本),装好就能用。终端里的 Ta 用用户自己装的那一份:
6
+
7
+ ```bash
8
+ /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" # macOS、Linux
9
+ irm https://cua.ai/driver/install.ps1 | iex # Windows(PowerShell)
10
+ ```
11
+
12
+ `/computer-use` 显示用的是哪个程序,并检查系统还缺什么(缺什么就像第一次使用时一样引导用户);没装时给出安装命令。
13
+
14
+ ## 机制
15
+
16
+ - **找程序**:先看环境变量 `TA_CUA_DRIVER`(Ta Desktop 给它启动的 ta serve 设置,指向自带的那份),再依次看 `PATH`、`~/.local/bin`(安装脚本的默认位置)、macOS 的 `/Applications/CuaDriver.app` 和 `~/Applications/CuaDriver.app`。只做 `stat`,不运行任何程序。没找到时每条消息前再找一次,会话中途装好也不用重启。
17
+ - **提示**:找到时在系统提示里加一段 `<computer_use>`:工具名是 `tools.mcp__cua_driver__*`,先用 `describeNamespace("mcp__cua_driver")` 看用法,另加一句当前系统的限制(macOS:在等授权时不要再向用户要;Windows:以管理员身份运行的窗口操作不了;Linux:需要桌面会话和 AT-SPI)。没找到时什么都不加。
18
+ - **按需启动**:MCP 服务器 `cua-driver`(codemode 暴露)不在会话开始时注册,第一次有 codemode 脚本点名 `mcp__cua_driver` 时(按 pi 的 `reachesTools`;搜索工具的脚本要在文字里写出它)才用 `pi.registerMcpServer` 注册,等工具出现(最多 5 秒)后这次调用就能用。pi 会在每个会话开始时连接所有已注册的服务器,提前注册就意味着每个会话多一个 `cua-driver mcp` 进程(约 10–24 MB);按需注册后,不用桌面应用的会话没有任何额外进程。
19
+ - **系统要求与授权**(`src/cua-driver-access.ts`,每个进程第一次使用时检查一次,同时到来的调用共用这一次检查):
20
+ - **macOS**:Cua Driver 需要辅助功能(读取和操作应用)和屏幕录制(截图)两项授权,授权属于 CuaDriver.app(`com.trycua.driver`,Cua 的开发者签名),不属于 Ta。用 `cua-driver call check_permissions` 查询(约 20 ms)。缺授权时,CuaDriver.app 会自己弹出两步引导窗口;Ta 发一条通知说明要开哪两个开关,然后每秒查一次,开好后这次调用继续执行,不用重试。引导窗口被关掉时,Ta 直接打开系统设置的对应页面。最多等 3 分钟,用户按“停止”立即结束;没有辅助功能时,这次调用失败,失败原因会告诉模型去请用户开启,`/computer-use` 可以再走一遍引导。只缺屏幕录制时照常继续(不截图也能读取和操作),并提示一次。守护进程刚启动的几百毫秒里会先返回 `permissions_pending`,所以先静默等最多 3 秒,避免误报。
21
+ - **Linux**:不需要授权,但需要图形会话(X11,或带 XWayland 的 Wayland)、AT-SPI 辅助功能总线,以及 libX11 / libXi / libxkbcommon。用 `cua-driver doctor --json` 检查(约 0.3 s):没有图形会话(例如通过 SSH)或缺库时,这次调用失败并说明原因和安装命令;AT-SPI 等警告通知用户一次,照常继续。
22
+ - **Windows**:不需要授权;同样用 `doctor` 检查桌面会话。以管理员身份运行的窗口需要 Ta 也以管理员身份运行(UIPI),这一点写在提示里。
23
+ - 系统说不清的时候(用户自己装的那份守护进程还没启动)不阻止调用,下一次再查。
24
+ - **Ta Desktop 自带的那份,在 macOS 上**(`src/cua-driver-daemon.ts`):MCP 进程只负责转发,真正干活的守护进程必须经 LaunchServices 作为 CuaDriver.app 启动,授权才算在它自己名下。Ta 在自己的 socket(`~/Library/Caches/ta/cua-driver.sock`)上启动一个守护进程,MCP 服务器是 `cua-driver mcp --embedded --socket <它>`。具体做法:
25
+ - 嵌在另一个 app 包里的 app,macOS 会算成外层 app 的一部分(授权会挂到 Ta 名下,每次构建都会丢)。所以第一次使用时,把它按版本克隆到 `~/Library/Application Support/Ta/cua-driver/<版本>/`(APFS 克隆,约 30 ms,不额外占磁盘),从那里启动,同时删掉旧版本的克隆。
26
+ - LaunchServices 按 bundle id 挑副本,可能挑中本机其他应用带的 CuaDriver.app(版本不同,转发会报不兼容)。所以启动前先用 `lsregister -f` 注册这份,启动后核对实际运行的程序路径;对不上就停掉,并提示 `cua-driver doctor --fix`。socket 上已有的守护进程(其他 Ta 进程或上次留下的)同样先核对一次。
27
+ - 自带的那份关闭 Cua 的产品遥测和更新检查(`CUA_DRIVER_RS_TELEMETRY_ENABLED=0`、`CUA_DRIVER_RS_UPDATE_CHECK=0`),由 Ta 的版本更新负责升级;用户自己装的那份保持用户自己的设置。
28
+ - 守护进程每次被用到前都会确认它还活着(一次 socket 连接),挂了就重启;10 分钟没用就退出(约 60 MB),转发进程随之退出,pi 在下一次调用时自动重连;启动它的进程退出时也会把它停掉。
29
+
30
+ 实测(M4 Pro,假模型跑 `ta -p`,自带 Cua Driver 0.47.5,用 Ta.app 打包产物里的那份):第一次调用比不调用多约 0.3 s(克隆加守护进程启动);之后每次新进程的首次调用,和不调用时的差别在测量误差内(约 1.18 s 对 1.20 s)。守护进程启动约 150–250 ms,授权查询约 20 ms,空闲时占约 33–60 MB。
31
+
32
+ ## 限制
33
+
34
+ - `mcp.json` 里已有同名(`cua-driver`)服务器时以文件为准;以其他名字配置的 Cua Driver(例如 `cua-driver connect` 写的 `cua-computer-use`)不会被识别,工具会出现两份,可以关掉本扩展。
35
+ - 远程执行(`ta serve --exec`)时 MCP 服务器在执行环境里启动,而程序是在 ta serve 所在的机器上找的,两边路径不同时连不上。
36
+ - macOS 上,用户在引导窗口里点“退出并重新打开”(屏幕录制要求)后,macOS 会不带 Ta 的参数重新启动 CuaDriver.app;Ta 会重启自己的守护进程,并在授权完成后停掉这个多出来的实例。
37
+ - Windows 和 Linux 的安装包没有在真机上验证过;Linux 的 `doctor` 输出在容器里验证过。
@@ -0,0 +1,551 @@
1
+ // src/extension.ts
2
+ import { execFile as execFile3 } from "node:child_process";
3
+ import { realpathSync } from "node:fs";
4
+ import { join as join3 } from "node:path";
5
+
6
+ // src/cua-driver.ts
7
+ import { statSync } from "node:fs";
8
+ import { homedir } from "node:os";
9
+ import { join } from "node:path";
10
+ var CuaDriver;
11
+ ((CuaDriver) => {
12
+ CuaDriver.SERVER = "cua-driver";
13
+ CuaDriver.NAMESPACE = "mcp__cua_driver";
14
+ CuaDriver.SECTION = "computer_use";
15
+ CuaDriver.BUNDLED_ENV = "TA_CUA_DRIVER";
16
+ function prompt(platform) {
17
+ return [
18
+ "Cua Driver lets you use the desktop apps of this computer in the background, without taking the user's pointer or keyboard: list apps and windows, read a window's accessibility tree, click, type, scroll and take screenshots.",
19
+ `Its tools are \`tools.${CuaDriver.NAMESPACE}__*\` in codemode scripts. It starts the first time a script names \`${CuaDriver.NAMESPACE}\`: begin with \`describeNamespace("${CuaDriver.NAMESPACE}")\` for its tools and how to use them.`,
20
+ "Use it only when a task needs a desktop app, not for files or commands.",
21
+ ...PLATFORM_NOTES[platform] ? [PLATFORM_NOTES[platform]] : []
22
+ ].join(`
23
+ `);
24
+ }
25
+ CuaDriver.prompt = prompt;
26
+ const PLATFORM_NOTES = {
27
+ darwin: "If a call waits for the permissions of macOS, Ta has asked the user for them; do not ask again.",
28
+ win32: "Windows does not let it control the windows of apps run as administrator unless Ta runs as administrator too.",
29
+ linux: "It needs the desktop session of the user (X11, or Wayland with XWayland) and its accessibility bus (AT-SPI)."
30
+ };
31
+ CuaDriver.BUNDLED_ENV_VARS = { CUA_DRIVER_RS_TELEMETRY_ENABLED: "0", CUA_DRIVER_RS_UPDATE_CHECK: "0" };
32
+ CuaDriver.DESCRIPTION = "Use the desktop apps of this computer in the background (Cua Driver)";
33
+ CuaDriver.INSTALL = {
34
+ posix: '/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"',
35
+ win32: "irm https://cua.ai/driver/install.ps1 | iex"
36
+ };
37
+ function locate(host = currentHost()) {
38
+ if (host.bundled && host.isFile(host.bundled)) {
39
+ return { path: host.bundled, bundled: true };
40
+ }
41
+ const windows = host.platform === "win32";
42
+ const name = windows ? "cua-driver.exe" : "cua-driver";
43
+ const dirs = (host.path ?? "").split(windows ? ";" : ":").filter(Boolean);
44
+ const candidates = [...dirs.map((dir) => join(dir, name)), join(host.home, ".local", "bin", name)];
45
+ if (host.platform === "darwin") {
46
+ const app = "CuaDriver.app/Contents/MacOS/cua-driver";
47
+ candidates.push(join("/Applications", app), join(host.home, "Applications", app));
48
+ }
49
+ const own = candidates.find((candidate) => host.isFile(candidate));
50
+ return own ? { path: own, bundled: false } : undefined;
51
+ }
52
+ CuaDriver.locate = locate;
53
+ function appOf(program) {
54
+ const inside = "/Contents/MacOS/cua-driver";
55
+ const app = program.slice(0, -inside.length);
56
+ return program.endsWith(inside) && app.endsWith(".app") ? app : undefined;
57
+ }
58
+ CuaDriver.appOf = appOf;
59
+ function reaches(reach, input) {
60
+ if (reach === undefined) {
61
+ return false;
62
+ }
63
+ if (reach === "all") {
64
+ return JSON.stringify(input ?? null).includes(CuaDriver.NAMESPACE);
65
+ }
66
+ return reach.some((word) => word.includes(CuaDriver.NAMESPACE));
67
+ }
68
+ CuaDriver.reaches = reaches;
69
+ function status(program, platform) {
70
+ if (program) {
71
+ const which = program.bundled ? `the one that comes with Ta Desktop (${program.path})` : program.path;
72
+ return `Cua Driver: ${which}. Ta starts it the first time it uses a desktop app, and checks what the system needs then.`;
73
+ }
74
+ const install = platform === "win32" ? CuaDriver.INSTALL.win32 : CuaDriver.INSTALL.posix;
75
+ return `Ta uses desktop apps through Cua Driver, which is not installed. Install it with \`${install}\`, or use Ta Desktop, which comes with it; Ta finds it on the next message, with no restart.`;
76
+ }
77
+ CuaDriver.status = status;
78
+ function currentHost() {
79
+ return { platform: process.platform, path: process.env.PATH, home: homedir(), bundled: process.env[CuaDriver.BUNDLED_ENV], isFile };
80
+ }
81
+ function isFile(path) {
82
+ try {
83
+ return statSync(path).isFile();
84
+ } catch {
85
+ return false;
86
+ }
87
+ }
88
+ })(CuaDriver ||= {});
89
+
90
+ // src/cua-driver-access.ts
91
+ import { execFile } from "node:child_process";
92
+ var CHECK_TIMEOUT_MS = 5000;
93
+ var DOCTOR_TIMEOUT_MS = 15000;
94
+ var POLL_MS = 1000;
95
+ var GRACE_MS = 3000;
96
+ var GRACE_POLL_MS = 100;
97
+ var WAIT_MS = 3 * 60 * 1000;
98
+ var DESKTOP_PROBES = /display|x11|wayland|at-spi|accessib|session|desktop|automation|uia/i;
99
+ var MISSING_LIBRARY = /error while loading shared libraries: (\S+?):/;
100
+ function askMac(env = process.env) {
101
+ return /^zh/i.test(env.TA_LANGUAGE || env.LC_ALL || env.LANG || "") ? "Ta 正在请 macOS 允许 Cua Driver 使用这台 Mac:请在 Cua Driver 打开的窗口或“系统设置”里,打开“辅助功能”(用于点击和输入)和“屏幕录制”(用于查看窗口)中的 Cua Driver。开好后 Ta 会自动继续;点“停止”可放弃。" : "Ta is asking macOS to let Cua Driver use this Mac: turn Cua Driver on under Accessibility (to click and type) and Screen Recording (to see windows), in the window Cua Driver opened or in System Settings. Ta goes on by itself once you have; Stop gives up.";
102
+ }
103
+ var NO_ACCESSIBILITY = "Cua Driver has no Accessibility permission of macOS, which it needs to read and use apps: ask the user to turn Cua Driver on in System Settings → Privacy & Security → Accessibility (`/computer-use` in Ta opens it), then try again.";
104
+ var NO_SCREEN_RECORDING = "Cua Driver has no Screen Recording permission of macOS: it reads and uses apps, but takes no screenshots. Turn it on in System Settings → Privacy & Security → Screen & System Audio Recording.";
105
+
106
+ class CuaDriverAccess {
107
+ program;
108
+ options;
109
+ run;
110
+ constructor(program, options) {
111
+ this.program = program;
112
+ this.options = options;
113
+ this.run = options.run ?? runProgram;
114
+ }
115
+ check(notify, signal) {
116
+ return this.options.platform === "darwin" ? this.checkMac(notify, signal) : this.checkDesktop();
117
+ }
118
+ async permissions() {
119
+ const socket = this.options.socket ? ["--socket", this.options.socket] : [];
120
+ const result = await this.run(this.program, ["call", "check_permissions", '{"prompt":false}', ...socket], CHECK_TIMEOUT_MS, this.options.env);
121
+ return CuaDriverAccess.parsePermissions(result);
122
+ }
123
+ async checkMac(notify, signal) {
124
+ let state = await this.permissions();
125
+ const grace = Date.now() + (this.options.graceMs ?? GRACE_MS);
126
+ while (state?.pending && Date.now() < grace && !signal?.aborted) {
127
+ await sleep(GRACE_POLL_MS, signal);
128
+ state = await this.permissions();
129
+ }
130
+ if (!state || state.accessibility && state.screenRecording) {
131
+ return { ready: true, notes: [], known: state !== undefined };
132
+ }
133
+ notify(askMac());
134
+ if (!state.pending) {
135
+ this.options.openSettings?.(state.accessibility ? "Privacy_ScreenCapture" : "Privacy_Accessibility");
136
+ }
137
+ const deadline = Date.now() + (this.options.waitMs ?? WAIT_MS);
138
+ while (!signal?.aborted && Date.now() < deadline && !(state?.accessibility && (state.screenRecording || !state.pending))) {
139
+ await sleep(this.options.pollMs ?? POLL_MS, signal);
140
+ await this.options.beforeLook?.();
141
+ state = await this.permissions() ?? state;
142
+ }
143
+ if (state?.accessibility) {
144
+ return { ready: true, notes: state.screenRecording ? [] : [NO_SCREEN_RECORDING], known: true };
145
+ }
146
+ return { ready: false, reason: signal?.aborted ? "Stopped while waiting for the permissions of macOS." : NO_ACCESSIBILITY };
147
+ }
148
+ async checkDesktop() {
149
+ const result = await this.run(this.program, ["doctor", "--json"], DOCTOR_TIMEOUT_MS, this.options.env);
150
+ return CuaDriverAccess.parseDoctor(result, this.options.platform);
151
+ }
152
+ }
153
+ ((CuaDriverAccess) => {
154
+ function parsePermissions(result) {
155
+ if (`${result.stdout}${result.stderr}`.includes("permissions_pending")) {
156
+ return { accessibility: false, screenRecording: false, pending: true };
157
+ }
158
+ try {
159
+ const answer = JSON.parse(result.stdout);
160
+ if (typeof answer.accessibility !== "boolean") {
161
+ return;
162
+ }
163
+ return { accessibility: answer.accessibility, screenRecording: answer.screen_recording === true, pending: false };
164
+ } catch {
165
+ return;
166
+ }
167
+ }
168
+ CuaDriverAccess.parsePermissions = parsePermissions;
169
+ function parseDoctor(result, platform) {
170
+ const library = MISSING_LIBRARY.exec(result.stderr)?.[1];
171
+ if (library) {
172
+ const install = platform === "linux" ? " On Debian or Ubuntu: `sudo apt install libx11-6 libxi6 libxkbcommon0 at-spi2-core`." : "";
173
+ return { ready: false, reason: `Cua Driver cannot start: the system has no ${library}.${install}` };
174
+ }
175
+ let probes;
176
+ try {
177
+ probes = JSON.parse(result.stdout).probes ?? [];
178
+ } catch {
179
+ return { ready: true, notes: [], known: true };
180
+ }
181
+ const text = (probe) => [`${probe.label}: ${probe.message}`, probe.detail].filter(Boolean).join(" — ");
182
+ const failed = probes.filter((probe) => probe.status === "error" || probe.status === "warn" && /display server/i.test(probe.label ?? ""));
183
+ if (failed.length > 0) {
184
+ return { ready: false, reason: `Cua Driver cannot use the desktop: ${failed.map(text).join("; ")}` };
185
+ }
186
+ const warned = probes.filter((probe) => probe.status === "warn" && DESKTOP_PROBES.test(probe.label ?? ""));
187
+ return { ready: true, notes: warned.map((probe) => `Cua Driver: ${text(probe)}`), known: true };
188
+ }
189
+ CuaDriverAccess.parseDoctor = parseDoctor;
190
+ })(CuaDriverAccess ||= {});
191
+ function runProgram(program, args, timeoutMs, env) {
192
+ return new Promise((resolve) => {
193
+ execFile(program, args, { timeout: timeoutMs, maxBuffer: 4 * 1024 * 1024, env: { ...process.env, ...env } }, (error, stdout, stderr) => {
194
+ const code = error ? typeof error.code === "number" ? error.code : -1 : 0;
195
+ resolve({ code, stdout: String(stdout), stderr: String(stderr) });
196
+ });
197
+ });
198
+ }
199
+ function sleep(ms, signal) {
200
+ return new Promise((resolve) => {
201
+ const timer = setTimeout(done, ms);
202
+ signal?.addEventListener("abort", done, { once: true });
203
+ function done() {
204
+ clearTimeout(timer);
205
+ signal?.removeEventListener("abort", done);
206
+ resolve();
207
+ }
208
+ });
209
+ }
210
+
211
+ // src/cua-driver-daemon.ts
212
+ import { execFile as execFile2 } from "node:child_process";
213
+ import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync } from "node:fs";
214
+ import { connect } from "node:net";
215
+ import { homedir as homedir2 } from "node:os";
216
+ import { dirname, join as join2 } from "node:path";
217
+ var START_TIMEOUT_MS = 1e4;
218
+ var POLL_MS2 = 20;
219
+ var IDLE_MS = 10 * 60 * 1000;
220
+
221
+ class CuaDriverDaemon {
222
+ source;
223
+ options;
224
+ socket;
225
+ pidFile;
226
+ dir;
227
+ app;
228
+ starting;
229
+ launched = false;
230
+ verified = false;
231
+ idle;
232
+ launch;
233
+ constructor(source, options = {}) {
234
+ this.source = source;
235
+ this.options = options;
236
+ this.dir = options.dir ?? join2(homedir2(), "Library", "Caches", "ta");
237
+ this.app = CuaDriverDaemon.nested(source) ? join2(this.stageDir, CuaDriverDaemon.version(source), "CuaDriver.app") : source;
238
+ this.socket = join2(this.dir, "cua-driver.sock");
239
+ this.pidFile = join2(this.dir, "cua-driver.pid");
240
+ this.launch = options.launch ?? openApp;
241
+ }
242
+ async ensure() {
243
+ if (await this.alive() && (this.verified || await this.adopt())) {
244
+ this.touch();
245
+ return;
246
+ }
247
+ clearTimeout(this.idle);
248
+ this.starting ??= this.start().finally(() => {
249
+ this.starting = undefined;
250
+ });
251
+ await this.starting;
252
+ this.touch();
253
+ }
254
+ alive() {
255
+ if (!existsSync(this.socket)) {
256
+ return Promise.resolve(false);
257
+ }
258
+ return new Promise((resolve) => {
259
+ const socket = connect(this.socket);
260
+ socket.once("connect", () => {
261
+ socket.destroy();
262
+ resolve(true);
263
+ });
264
+ socket.on("error", () => resolve(false));
265
+ });
266
+ }
267
+ stop() {
268
+ clearTimeout(this.idle);
269
+ if (!this.launched) {
270
+ return;
271
+ }
272
+ this.launched = false;
273
+ const pid = this.pid();
274
+ if (pid) {
275
+ try {
276
+ process.kill(pid, "SIGTERM");
277
+ } catch {}
278
+ }
279
+ }
280
+ async reap() {
281
+ const program = this.program;
282
+ const own = this.pid();
283
+ const list = await new Promise((resolve) => execFile2("ps", ["-axo", "pid=,command="], (_error, stdout) => resolve(String(stdout ?? ""))));
284
+ for (const line of list.split(`
285
+ `)) {
286
+ const [, pid, command] = /^\s*(\d+)\s+(.*)$/.exec(line) ?? [];
287
+ if (pid && Number(pid) !== own && command.startsWith(`${program} serve`) && !command.includes(this.socket)) {
288
+ try {
289
+ process.kill(Number(pid), "SIGTERM");
290
+ } catch {}
291
+ }
292
+ }
293
+ }
294
+ async adopt() {
295
+ const running = await this.identify();
296
+ if (!running || running.executable === this.program) {
297
+ this.verified = true;
298
+ return true;
299
+ }
300
+ try {
301
+ process.kill(running.pid, "SIGTERM");
302
+ } catch {}
303
+ return false;
304
+ }
305
+ identify() {
306
+ return new Promise((resolve) => execFile2(this.program, ["call", "check_permissions", '{"prompt":false}', "--socket", this.socket], { timeout: 5000, env: { ...process.env, ...this.options.env } }, (_error, stdout) => {
307
+ try {
308
+ const { source } = JSON.parse(String(stdout));
309
+ resolve(source?.executable && source.pid ? { executable: source.executable, pid: source.pid } : undefined);
310
+ } catch {
311
+ resolve(undefined);
312
+ }
313
+ }));
314
+ }
315
+ get program() {
316
+ return join2(this.app, "Contents", "MacOS", "cua-driver");
317
+ }
318
+ async verify() {
319
+ const pid = this.pid();
320
+ const command = pid ? await this.commandOf(pid) : undefined;
321
+ const program = this.program;
322
+ if (command === undefined || command.includes(`${program} serve`)) {
323
+ this.verified = true;
324
+ return;
325
+ }
326
+ this.stop();
327
+ throw new Error(`macOS started another copy of CuaDriver.app (${command.split(" serve")[0]}). Run \`${program} doctor --fix\`, which makes macOS forget the other copies, then try again.`);
328
+ }
329
+ commandOf(pid) {
330
+ return new Promise((resolve) => execFile2("ps", ["-o", "command=", "-p", String(pid)], (error, stdout) => resolve(error ? undefined : String(stdout).trim())));
331
+ }
332
+ touch() {
333
+ clearTimeout(this.idle);
334
+ this.idle = setTimeout(() => this.stop(), this.options.idleMs ?? IDLE_MS);
335
+ this.idle.unref?.();
336
+ }
337
+ pid() {
338
+ try {
339
+ return Number(readFileSync(this.pidFile, "utf8").trim()) || undefined;
340
+ } catch {
341
+ return;
342
+ }
343
+ }
344
+ get stageDir() {
345
+ return this.options.stageDir ?? join2(homedir2(), "Library", "Application Support", "Ta", "cua-driver");
346
+ }
347
+ async stage() {
348
+ if (this.app === this.source || existsSync(this.app)) {
349
+ return;
350
+ }
351
+ const versions = dirname(dirname(this.app));
352
+ const temp = `${dirname(this.app)}.part`;
353
+ rmSync(temp, { recursive: true, force: true });
354
+ mkdirSync(temp, { recursive: true });
355
+ await run("cp", ["-cR", this.source, temp]).catch(() => run("cp", ["-R", this.source, temp]));
356
+ rmSync(dirname(this.app), { recursive: true, force: true });
357
+ renameSync(temp, dirname(this.app));
358
+ for (const name of readdirSync(versions)) {
359
+ if (join2(versions, name) !== dirname(this.app)) {
360
+ rmSync(join2(versions, name), { recursive: true, force: true });
361
+ }
362
+ }
363
+ }
364
+ async start() {
365
+ await this.stage();
366
+ mkdirSync(this.dir, { recursive: true });
367
+ rmSync(this.socket, { force: true });
368
+ this.verified = false;
369
+ await this.launch(this.app, ["serve", "--socket", this.socket, "--pid-file", this.pidFile], this.options.env ?? {});
370
+ if (!this.launched) {
371
+ this.launched = true;
372
+ process.once("exit", () => this.stop());
373
+ }
374
+ const deadline = Date.now() + START_TIMEOUT_MS;
375
+ while (Date.now() < deadline) {
376
+ if (await this.alive()) {
377
+ return this.verify();
378
+ }
379
+ await new Promise((resolve) => setTimeout(resolve, POLL_MS2));
380
+ }
381
+ throw new Error(`Cua Driver did not start in ${START_TIMEOUT_MS / 1000} s: run \`${join2(this.app, "Contents/MacOS/cua-driver")} doctor\`.`);
382
+ }
383
+ }
384
+ ((CuaDriverDaemon) => {
385
+ function nested(app) {
386
+ return /\.app\//.test(dirname(app));
387
+ }
388
+ CuaDriverDaemon.nested = nested;
389
+ function version(app) {
390
+ try {
391
+ return /<key>CFBundleVersion<\/key>\s*<string>([^<]+)<\/string>/.exec(readFileSync(join2(app, "Contents", "Info.plist"), "utf8"))?.[1] ?? "current";
392
+ } catch {
393
+ return "current";
394
+ }
395
+ }
396
+ CuaDriverDaemon.version = version;
397
+ })(CuaDriverDaemon ||= {});
398
+ var LSREGISTER = "/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister";
399
+ async function openApp(app, args, env) {
400
+ await run(LSREGISTER, ["-f", app]).catch(() => {
401
+ return;
402
+ });
403
+ const vars = Object.entries(env).flatMap(([name, value]) => ["--env", `${name}=${value}`]);
404
+ await run("open", ["-g", "-n", "-a", app, ...vars, "--args", ...args]);
405
+ }
406
+ function run(program, args) {
407
+ return new Promise((resolve, reject) => execFile2(program, args, (error, stdout) => error ? reject(error) : resolve(String(stdout))));
408
+ }
409
+
410
+ // src/extension.ts
411
+ var CONNECT_WAIT_MS = 5000;
412
+ var POLL_MS3 = 15;
413
+ var setups = new Map;
414
+ function setupOf(found, platform) {
415
+ const program = { ...found, path: realPath(found.path) };
416
+ let setup = setups.get(program.path);
417
+ if (!setup) {
418
+ const env = program.bundled ? { ...CuaDriver.BUNDLED_ENV_VARS } : undefined;
419
+ const app = program.bundled && platform === "darwin" ? CuaDriver.appOf(program.path) : undefined;
420
+ const daemon = app ? new CuaDriverDaemon(app, { env }) : undefined;
421
+ const access = new CuaDriverAccess(daemon ? join3(daemon.app, "Contents", "MacOS", "cua-driver") : program.path, {
422
+ platform,
423
+ socket: daemon?.socket,
424
+ env,
425
+ beforeLook: daemon ? () => daemon.ensure() : undefined,
426
+ openSettings: (pane) => execFile3("open", [`x-apple.systempreferences:com.apple.preference.security?${pane}`], () => {
427
+ return;
428
+ })
429
+ });
430
+ const args = daemon ? ["mcp", "--embedded", "--socket", daemon.socket] : ["mcp"];
431
+ setup = { program, daemon, access, args, env, ready: false };
432
+ setups.set(program.path, setup);
433
+ }
434
+ return setup;
435
+ }
436
+ function realPath(path) {
437
+ try {
438
+ return realpathSync(path);
439
+ } catch {
440
+ return path;
441
+ }
442
+ }
443
+ var DEFAULTS = { locate: () => CuaDriver.locate(), platform: process.platform, setup: setupOf };
444
+ function activate(pi, options = DEFAULTS) {
445
+ let setup;
446
+ let connected = false;
447
+ let connecting;
448
+ const prefix = `${CuaDriver.NAMESPACE}__`;
449
+ const ready = () => pi.getAllTools().some((tool) => tool.name.startsWith(prefix));
450
+ const found = () => {
451
+ if (!setup) {
452
+ const program = options.locate();
453
+ setup = program ? options.setup(program, options.platform) : undefined;
454
+ }
455
+ return setup;
456
+ };
457
+ const connect2 = async (driver, signal) => {
458
+ if (!pi.getMcpServers().some((server) => server.name === CuaDriver.SERVER)) {
459
+ pi.registerMcpServer(CuaDriver.SERVER, { command: driver.program.path, args: driver.args, env: driver.env, description: CuaDriver.DESCRIPTION });
460
+ }
461
+ const deadline = Date.now() + CONNECT_WAIT_MS;
462
+ while (!ready() && Date.now() < deadline && !signal?.aborted) {
463
+ await new Promise((resolve) => setTimeout(resolve, POLL_MS3));
464
+ }
465
+ connected = !signal?.aborted;
466
+ };
467
+ const check = async (driver, notify, signal) => {
468
+ driver.checking ??= driver.access.check(notify, signal).finally(() => {
469
+ driver.checking = undefined;
470
+ });
471
+ const result = await driver.checking;
472
+ if (!result.ready) {
473
+ return result.reason;
474
+ }
475
+ if (result.known && !driver.ready) {
476
+ driver.ready = true;
477
+ for (const note of result.notes) {
478
+ notify(note);
479
+ }
480
+ driver.daemon?.reap();
481
+ }
482
+ return;
483
+ };
484
+ pi.on("before_agent_start", (event) => {
485
+ const { sections } = event.systemPromptOptions;
486
+ if (found()) {
487
+ sections[CuaDriver.SECTION] = CuaDriver.prompt(options.platform);
488
+ } else {
489
+ delete sections[CuaDriver.SECTION];
490
+ }
491
+ });
492
+ pi.on("tool_call", async (event, ctx) => {
493
+ const driver = setup;
494
+ if (!driver) {
495
+ return;
496
+ }
497
+ if (driver.ready && connected && !driver.daemon) {
498
+ return;
499
+ }
500
+ const tool = pi.getAllTools().find((candidate) => candidate.name === event.toolName);
501
+ if (!CuaDriver.reaches(tool?.reachesTools?.(event.input), event.input)) {
502
+ return;
503
+ }
504
+ const notify = (message) => ctx.ui.notify(message, "warning");
505
+ try {
506
+ await driver.daemon?.ensure();
507
+ const reason = driver.daemon && !driver.ready ? await check(driver, notify, ctx.signal) : undefined;
508
+ if (reason) {
509
+ return { block: true, reason };
510
+ }
511
+ if (!connected) {
512
+ connecting ??= connect2(driver, ctx.signal).finally(() => {
513
+ connecting = undefined;
514
+ });
515
+ await connecting;
516
+ }
517
+ const after = !driver.daemon && !driver.ready ? await check(driver, notify, ctx.signal) : undefined;
518
+ return after ? { block: true, reason: after } : undefined;
519
+ } catch (error) {
520
+ return { block: true, reason: error.message };
521
+ }
522
+ });
523
+ pi.registerCommand("computer-use", {
524
+ description: "Whether Ta can use the desktop apps of this computer (Cua Driver), and what the system still needs",
525
+ handler: async (_args, ctx) => {
526
+ const program = options.locate();
527
+ ctx.ui.notify(CuaDriver.status(program, options.platform), program ? "info" : "warning");
528
+ if (!program) {
529
+ return;
530
+ }
531
+ const driver = options.setup(program, options.platform);
532
+ await driver.daemon?.ensure();
533
+ const result = await driver.access.check((message) => ctx.ui.notify(message, "warning"), ctx.signal);
534
+ if (!result.ready) {
535
+ ctx.ui.notify(result.reason, "warning");
536
+ } else if (result.known) {
537
+ driver.ready = true;
538
+ ctx.ui.notify(["Cua Driver has what it needs.", ...result.notes].join(`
539
+ `), result.notes.length > 0 ? "warning" : "info");
540
+ }
541
+ }
542
+ });
543
+ }
544
+ function computerUse(pi) {
545
+ activate(pi);
546
+ }
547
+ export {
548
+ setupOf,
549
+ computerUse as default,
550
+ activate
551
+ };