@deepseek-ai/dsh-experimental-computer-use-cua-driver-native 0.1.6-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.i18n.yaml +6 -0
- package/README.md +145 -0
- package/README.zh.md +145 -0
- package/lib/index.js +123 -0
- package/lib/types/index.d.ts +21 -0
- package/package.json +57 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 DeepSeek
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.i18n.yaml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
|
2
|
+
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
|
+
# after editing either side, bring the other along and re-record with:
|
|
4
|
+
# pnpm run verify-translation-pairing --write packages/experimental/computer-use-cua-driver-native/README.md
|
|
5
|
+
README.md: 28e47f17020297ef7e1c2fa6656c8fda0b808a8f
|
|
6
|
+
README.zh.md: ce25309e77fcd05836b4fccddd2482ecf2f1ad17
|
package/README.md
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Run Cua Driver computer-use tools from its native npm SDK, with durable screenshots and explicit host desktop permissions."
|
|
3
|
+
kind: "package-reference"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# @deepseek-ai/dsh-experimental-computer-use-cua-driver-native
|
|
7
|
+
|
|
8
|
+
English | [中文](README.zh.md)
|
|
9
|
+
|
|
10
|
+
## Summary
|
|
11
|
+
|
|
12
|
+
Use Cua Driver to inspect and operate desktop windows without installing its separate CLI or application. The native npm dependency runs inside the DSH host and exposes Cua Driver's own tools. Screenshots reach image-capable models through durable attachments. This published experimental package requires the launching host's desktop permissions and remains an explicit composition choice.
|
|
13
|
+
|
|
14
|
+
## Table of Contents
|
|
15
|
+
|
|
16
|
+
- [Use this package](#use-this-package)
|
|
17
|
+
- [Understand the implementation](#understand-the-implementation)
|
|
18
|
+
- [Further Exploration](#further-exploration)
|
|
19
|
+
- [Model Experience](#model-experience)
|
|
20
|
+
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
|
21
|
+
- [Dev Note](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## Use this package
|
|
27
|
+
|
|
28
|
+
Mount the provider in a composition that already supplies the tool registry and system prompt.
|
|
29
|
+
|
|
30
|
+
### Minimal configuration
|
|
31
|
+
|
|
32
|
+
```yaml
|
|
33
|
+
- name: '@deepseek-ai/dsh-computer-use'
|
|
34
|
+
- name: '@deepseek-ai/dsh-experimental-computer-use-cua-driver-native'
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
The provider has no configuration fields. It loads the exact Cua Driver npm version declared in [package.json](package.json) and uses its same-process defaults. Native import, runtime initialization, malformed catalog, duplicate tool name, or occupied computer-use registration failures reject activation and roll back owned resources. The registered provider name is `cua-driver-native`.
|
|
38
|
+
|
|
39
|
+
Use an attachment store and a model route that explicitly declares image input to receive screenshots. The [MCP result adapter](../../mcp/mcp-client/README.md) owns image admission and diagnostic behavior; programmatic callers retain the canonical raw result when a model cannot receive its images. Calls use Cua Driver's upstream tool parameters and results.
|
|
40
|
+
|
|
41
|
+
### Host requirements
|
|
42
|
+
|
|
43
|
+
The native dependency supplies platform binaries through npm optional dependencies. Keep optional dependencies enabled. Grant desktop permissions to the application that launches DSH; this provider neither installs a permission-owning app nor changes OS grants. The native runtime shares the host process, so native crashes can terminate that process. Use the [installed MCP provider](../computer-use-cua-driver-mcp/README.md) when the separate Cua Driver application should own permissions and execution.
|
|
44
|
+
|
|
45
|
+
### Verify the installed SDK
|
|
46
|
+
|
|
47
|
+
From the repository root, run this opt-in check against the installed native dependency. It discovers tools, reads permission status with `prompt: false`, and verifies teardown; it captures no screenshots, sends no input, and requests no OS permissions. Clearing `NODE_USE_ENV_PROXY` prevents Node from installing the launching shell's proxy before test setup.
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
env -u NODE_USE_ENV_PROXY DSH_COMPUTER_USE_NATIVE_E2E=1 node node_modules/vitest/vitest.mjs run --config vitest.e2e.config.ts packages/experimental/computer-use-cua-driver-native/tests/native.e2e.ts
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
-----
|
|
54
|
+
|
|
55
|
+
<a id="understand-the-implementation"></a>
|
|
56
|
+
## Understand the implementation
|
|
57
|
+
|
|
58
|
+
<details>
|
|
59
|
+
<summary>Implementation internals — click to expand</summary>
|
|
60
|
+
|
|
61
|
+
The provider reserves the shared computer-use registration before loading native code. A child plugin owns discovery, model tools, guidance, and the native runtime. The parent retains the registration until child teardown has removed tools, interrupted native calls and image-capability admission, awaited settlement, and completed native shutdown. Cancellation does not undo input already delivered to an application.
|
|
62
|
+
|
|
63
|
+
| File | Role |
|
|
64
|
+
|---|---|
|
|
65
|
+
| [src/index.ts](src/index.ts) | Native runtime ownership, catalog validation, tool registration, and provider guidance |
|
|
66
|
+
| — | No runtime invariant companion is published; resource ownership has no independently observed state to compare. |
|
|
67
|
+
|
|
68
|
+
Tool definitions reuse the existing MCP result adapter. Cua Driver's JSON catalog determines the schemas; its raw result supplies canonical text, structured output, and image bytes. The computer-use service carries only the provider name and exclusive registration.
|
|
69
|
+
|
|
70
|
+
</details>
|
|
71
|
+
|
|
72
|
+
-----
|
|
73
|
+
|
|
74
|
+
<a id="further-exploration"></a>
|
|
75
|
+
## Further Exploration
|
|
76
|
+
|
|
77
|
+
- [Computer-use service](../../computer-use/computer-use/README.md) — exclusive named registration.
|
|
78
|
+
- [MCP client](../../mcp/mcp-client/README.md) — shared result and image projection.
|
|
79
|
+
- [Cua Driver SDK](https://cua.ai/docs/reference/cua-driver/sdk-reference) — upstream runtime API and host facilities.
|
|
80
|
+
|
|
81
|
+
-----
|
|
82
|
+
|
|
83
|
+
<a id="model-experience"></a>
|
|
84
|
+
## Model Experience
|
|
85
|
+
|
|
86
|
+
### System prompt
|
|
87
|
+
|
|
88
|
+
#### What the model sees
|
|
89
|
+
|
|
90
|
+
The provider contributes the following computer-use guidance while its native tools are mounted.
|
|
91
|
+
|
|
92
|
+
##### Native Cua Driver guidance
|
|
93
|
+
|
|
94
|
+
```markdown
|
|
95
|
+
Cua Driver native computer-use tools operate the host desktop. Discover the exact app and window, then get a fresh window snapshot before acting. Use element_token from that snapshot, or coordinates from its screenshot. A new snapshot of that window invalidates its earlier element tokens. Select either target or the legacy pid/window_id fields; do not combine them.
|
|
96
|
+
|
|
97
|
+
Prefer background delivery. A refusal does not authorize a foreground retry. Verify the requested outcome from fresh state after an action; a delivered click alone does not prove the outcome. After cancellation, inspect current state before retrying because completed input is not rolled back. Other sessions and applications may change the same desktop.
|
|
98
|
+
|
|
99
|
+
On macOS, cursor-overlay operations may return facility_unavailable even when screenshots and input work.
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
#### Token effect
|
|
103
|
+
|
|
104
|
+
This fixed guidance adds system-prompt tokens while the provider is mounted. Upstream guidance resources are not automatically loaded.
|
|
105
|
+
|
|
106
|
+
#### KV Cache effect
|
|
107
|
+
|
|
108
|
+
The unchanged guidance preserves its repeated prompt prefix. Mounting, removing, or editing it changes that prefix and can reduce cache reuse.
|
|
109
|
+
|
|
110
|
+
### Discovered Cua Driver tools and results
|
|
111
|
+
|
|
112
|
+
#### What the model sees
|
|
113
|
+
|
|
114
|
+
Tools use the `cua_driver_native__` prefix followed by the upstream name and retain the upstream descriptions and input schemas. Upstream tool refusals become tool errors. Supported screenshots appear as durable image references beside result text; the canonical raw result remains available to programmatic callers.
|
|
115
|
+
|
|
116
|
+
#### Token effect
|
|
117
|
+
|
|
118
|
+
The discovered catalog adds tool definitions to each request. Accessibility trees, result text, and admitted screenshots add per-call context. Raw base64 remains in execution-local canonical values and is not copied into model history.
|
|
119
|
+
|
|
120
|
+
#### KV Cache effect
|
|
121
|
+
|
|
122
|
+
An unchanged catalog preserves its tool-definition prefix. Tool results append to Session history. Replacing the provider or its catalog changes the model-visible tools and can reduce prefix reuse.
|
|
123
|
+
|
|
124
|
+
## Known Limitations and Deferred Work
|
|
125
|
+
|
|
126
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
127
|
+
|
|
128
|
+
The package preserves the upstream driver's platform and application limits.
|
|
129
|
+
|
|
130
|
+
- **Host permissions and graphics session** — npm installation does not grant desktop access or create a graphical session.
|
|
131
|
+
- **Native cursor overlay** — a headless macOS Node host can receive `facility_unavailable` for overlay operations while screenshots and background input remain usable.
|
|
132
|
+
- **Shared desktop** — the provider does not reserve windows or complete workflows for a Session. Other callers and applications can change the same desktop between calls.
|
|
133
|
+
- **Cancellation** — an aborted call can have delivered input already; inspect fresh state before retrying. The provider waits for SDK shutdown during unload but does not promise native action rollback.
|
|
134
|
+
- **Failed shutdown** — if native shutdown fails, the registration remains occupied. Restart the host before mounting another computer-use provider.
|
|
135
|
+
- **Experimental release** — tool schemas follow the pinned upstream SDK and have no DSH stability promise.
|
|
136
|
+
|
|
137
|
+
<a id="dev-note"></a>
|
|
138
|
+
### Dev Note
|
|
139
|
+
|
|
140
|
+
<details>
|
|
141
|
+
<summary>Working context for maintainers — click to expand</summary>
|
|
142
|
+
|
|
143
|
+
None.
|
|
144
|
+
|
|
145
|
+
</details>
|
package/README.zh.md
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "通过原生 npm SDK 运行 Cua Driver 的电脑操作工具,持久化截图,并明确主机桌面权限要求。"
|
|
3
|
+
kind: "package-reference"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# @deepseek-ai/dsh-experimental-computer-use-cua-driver-native
|
|
7
|
+
|
|
8
|
+
[English](README.md) | 中文
|
|
9
|
+
|
|
10
|
+
## 概述
|
|
11
|
+
|
|
12
|
+
使用 Cua Driver 检查和操作桌面窗口,无需安装其独立 CLI 或应用。原生 npm 依赖在 DSH 主机进程内运行,提供 Cua Driver 自己的工具。截图通过持久化附件传给支持图像的模型。此实验性软件包会发布到 npm,需要启动主机的桌面权限,并且必须在组合配置中显式启用。
|
|
13
|
+
|
|
14
|
+
## 目录
|
|
15
|
+
|
|
16
|
+
- [使用本包](#use-this-package)
|
|
17
|
+
- [理解实现](#understand-the-implementation)
|
|
18
|
+
- [进一步探索](#further-exploration)
|
|
19
|
+
- [模型体验](#model-experience)
|
|
20
|
+
- [已知限制与延后工作](#known-limitations-and-deferred-work)
|
|
21
|
+
- [开发备注](#dev-note)
|
|
22
|
+
|
|
23
|
+
-----
|
|
24
|
+
|
|
25
|
+
<a id="use-this-package"></a>
|
|
26
|
+
## 使用本包
|
|
27
|
+
|
|
28
|
+
在已经提供工具注册表和系统提示词的组合中挂载此提供者。
|
|
29
|
+
|
|
30
|
+
### 最小配置
|
|
31
|
+
|
|
32
|
+
```yaml
|
|
33
|
+
- name: '@deepseek-ai/dsh-computer-use'
|
|
34
|
+
- name: '@deepseek-ai/dsh-experimental-computer-use-cua-driver-native'
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
此提供者没有配置字段。它加载 [package.json](package.json) 声明的确切 Cua Driver npm 版本,并采用其进程内默认配置。原生模块导入、运行时初始化、目录格式、工具重名或电脑操作注册冲突会使激活失败,并回滚所拥有的资源。注册的提供者名称为 `cua-driver-native`。
|
|
38
|
+
|
|
39
|
+
挂载附件存储并使用明确声明支持图像输入的模型路由,才能接收截图。[MCP 结果适配器](../../mcp/mcp-client/README.zh.md) 负责图像接纳和诊断行为;模型无法接收图像时,程序调用方仍保留规范原始结果。调用采用 Cua Driver 上游的工具参数和结果。
|
|
40
|
+
|
|
41
|
+
### 主机要求
|
|
42
|
+
|
|
43
|
+
原生依赖通过 npm 可选依赖提供各平台二进制文件,因此必须保留可选依赖安装。请向启动 DSH 的应用授予桌面权限;此提供者既不安装独立持有权限的应用,也不更改操作系统授权。原生运行时与主机共享进程,因此原生崩溃可能终止该进程。如果需要由独立的 Cua Driver 应用持有权限并执行操作,请使用[已安装的 MCP 提供者](../computer-use-cua-driver-mcp/README.zh.md)。
|
|
44
|
+
|
|
45
|
+
### 验证已安装的 SDK
|
|
46
|
+
|
|
47
|
+
在仓库根目录运行这项显式启用的检查,验证已安装的原生依赖。它发现工具、通过 `prompt: false` 读取权限状态,并验证卸载;它不截图、不发送输入,也不请求操作系统权限。清除 `NODE_USE_ENV_PROXY` 可防止 Node 在测试初始化之前采用启动 shell 的代理设置。
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
env -u NODE_USE_ENV_PROXY DSH_COMPUTER_USE_NATIVE_E2E=1 node node_modules/vitest/vitest.mjs run --config vitest.e2e.config.ts packages/experimental/computer-use-cua-driver-native/tests/native.e2e.ts
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
-----
|
|
54
|
+
|
|
55
|
+
<a id="understand-the-implementation"></a>
|
|
56
|
+
## 理解实现
|
|
57
|
+
|
|
58
|
+
<details>
|
|
59
|
+
<summary>实现内部——点击展开</summary>
|
|
60
|
+
|
|
61
|
+
此提供者在加载原生代码前占用共享电脑操作注册名额。子插件拥有目录发现、模型工具、指导文本和原生运行时。父插件保留注册名额,直到子插件卸载完成工具移除、中断原生调用和图像能力准入、等待调用结束及原生关闭。取消不会撤销已经传给应用的输入。
|
|
62
|
+
|
|
63
|
+
| 文件 | 职责 |
|
|
64
|
+
|---|---|
|
|
65
|
+
| [src/index.ts](src/index.ts) | 原生运行时所有权、目录校验、工具注册和提供者指导文本 |
|
|
66
|
+
| — | 不发布运行时不变量伴随模块;资源所有权没有可独立观测并比较的状态。 |
|
|
67
|
+
|
|
68
|
+
工具定义复用现有 MCP 结果适配器。Cua Driver 的 JSON 目录决定 schema,其原始结果提供规范文本、结构化输出和图像字节。电脑操作服务只保存提供者名称并保证独占注册。
|
|
69
|
+
|
|
70
|
+
</details>
|
|
71
|
+
|
|
72
|
+
-----
|
|
73
|
+
|
|
74
|
+
<a id="further-exploration"></a>
|
|
75
|
+
## 进一步探索
|
|
76
|
+
|
|
77
|
+
- [电脑操作服务](../../computer-use/computer-use/README.zh.md)——独占的具名注册。
|
|
78
|
+
- [MCP 客户端](../../mcp/mcp-client/README.zh.md)——共享结果与图像投影。
|
|
79
|
+
- [Cua Driver SDK](https://cua.ai/docs/reference/cua-driver/sdk-reference)——上游运行时 API 和主机能力。
|
|
80
|
+
|
|
81
|
+
-----
|
|
82
|
+
|
|
83
|
+
<a id="model-experience"></a>
|
|
84
|
+
## 模型体验
|
|
85
|
+
|
|
86
|
+
### 系统提示词
|
|
87
|
+
|
|
88
|
+
#### 模型看到什么
|
|
89
|
+
|
|
90
|
+
在原生工具挂载期间,此提供者加入以下电脑操作指导文本。
|
|
91
|
+
|
|
92
|
+
##### 原生 Cua Driver 指导文本
|
|
93
|
+
|
|
94
|
+
```markdown
|
|
95
|
+
Cua Driver native computer-use tools operate the host desktop. Discover the exact app and window, then get a fresh window snapshot before acting. Use element_token from that snapshot, or coordinates from its screenshot. A new snapshot of that window invalidates its earlier element tokens. Select either target or the legacy pid/window_id fields; do not combine them.
|
|
96
|
+
|
|
97
|
+
Prefer background delivery. A refusal does not authorize a foreground retry. Verify the requested outcome from fresh state after an action; a delivered click alone does not prove the outcome. After cancellation, inspect current state before retrying because completed input is not rolled back. Other sessions and applications may change the same desktop.
|
|
98
|
+
|
|
99
|
+
On macOS, cursor-overlay operations may return facility_unavailable even when screenshots and input work.
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
#### Token 影响
|
|
103
|
+
|
|
104
|
+
提供者挂载期间,这段固定指导文本增加系统提示词 token。上游指导资源不会自动加载。
|
|
105
|
+
|
|
106
|
+
#### KV Cache 影响
|
|
107
|
+
|
|
108
|
+
指导文本不变时,其重复提示词前缀保持稳定。挂载、移除或编辑它会改变此前缀,并可能减少缓存复用。
|
|
109
|
+
|
|
110
|
+
### 发现的 Cua Driver 工具与结果
|
|
111
|
+
|
|
112
|
+
#### 模型看到什么
|
|
113
|
+
|
|
114
|
+
工具名称使用 `cua_driver_native__` 前缀并附加上游名称,保留上游描述和输入 schema。上游工具拒绝转为工具错误。支持的截图作为持久化图像引用出现在结果文本旁;程序调用方仍可读取规范原始结果。
|
|
115
|
+
|
|
116
|
+
#### Token 影响
|
|
117
|
+
|
|
118
|
+
发现的目录为每个请求加入工具定义。无障碍树、结果文本和获准接纳的截图增加每次调用的上下文。原始 base64 保留在执行期间的规范值中,不复制到模型历史。
|
|
119
|
+
|
|
120
|
+
#### KV Cache 影响
|
|
121
|
+
|
|
122
|
+
目录不变时,工具定义前缀保持稳定。工具结果追加到 Session 历史。替换提供者或其目录会改变模型可见工具,并可能减少前缀复用。
|
|
123
|
+
|
|
124
|
+
## 已知限制与延后工作
|
|
125
|
+
|
|
126
|
+
<a id="known-limitations-and-deferred-work"></a>
|
|
127
|
+
|
|
128
|
+
此软件包保留上游驱动的平台和应用限制。
|
|
129
|
+
|
|
130
|
+
- **主机权限与图形会话**——npm 安装不会授予桌面访问权限或创建图形会话。
|
|
131
|
+
- **原生光标覆盖层**——无界面的 macOS Node 主机可能对覆盖层操作返回 `facility_unavailable`,同时截图和后台输入仍可用。
|
|
132
|
+
- **共享桌面**——此提供者不为某个 Session 预留窗口或完整工作流。其他调用方和应用可以在两次调用之间更改同一桌面。
|
|
133
|
+
- **取消**——被取消的调用可能已经传入输入;重试前必须检查新状态。卸载时提供者等待 SDK 关闭,但不承诺回滚原生操作。
|
|
134
|
+
- **关闭失败**——如果原生关闭失败,注册名额保持占用。挂载其他电脑操作提供者之前必须重启主机。
|
|
135
|
+
- **实验性发布**——工具 schema 跟随锁定的上游 SDK,不作 DSH 稳定性承诺。
|
|
136
|
+
|
|
137
|
+
<a id="dev-note"></a>
|
|
138
|
+
### 开发备注
|
|
139
|
+
|
|
140
|
+
<details>
|
|
141
|
+
<summary>维护者工作上下文——点击展开</summary>
|
|
142
|
+
|
|
143
|
+
无。
|
|
144
|
+
|
|
145
|
+
</details>
|
package/lib/index.js
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
import Schema from "@deepseek-ai/schemastery";
|
|
2
|
+
import { ComputerUseProviderName } from "@deepseek-ai/dsh-computer-use/brand";
|
|
3
|
+
import { createMcpToolDefinition } from "@deepseek-ai/dsh-mcp-client";
|
|
4
|
+
import { z } from "zod";
|
|
5
|
+
//#region lib/types/index.js
|
|
6
|
+
/**
|
|
7
|
+
* Computer use through the in-process Cua Driver native SDK and its own tools.
|
|
8
|
+
* @module @deepseek-ai/dsh-experimental-computer-use-cua-driver-native
|
|
9
|
+
*/
|
|
10
|
+
/** Cordis plugin identity for the native Cua Driver provider. */
|
|
11
|
+
const name = "experimental-computer-use-cua-driver-native";
|
|
12
|
+
/** Services required before the native runtime can publish tools. */
|
|
13
|
+
const inject = [
|
|
14
|
+
"computerUse",
|
|
15
|
+
"tools",
|
|
16
|
+
"systemPrompt"
|
|
17
|
+
];
|
|
18
|
+
/** The native provider uses the installed SDK's same-process defaults. */
|
|
19
|
+
const Config = Schema.object({});
|
|
20
|
+
const ToolCatalog = z.object({ tools: z.array(z.object({
|
|
21
|
+
name: z.string().min(1),
|
|
22
|
+
description: z.string().optional(),
|
|
23
|
+
inputSchema: z.record(z.string(), z.unknown()),
|
|
24
|
+
outputSchema: z.unknown().optional()
|
|
25
|
+
})) });
|
|
26
|
+
/** DeepSeek's function-name alphabet and maximum length are protocol constants. */
|
|
27
|
+
const TOOL_NAME = /^[A-Za-z0-9_-]{1,64}$/u;
|
|
28
|
+
const GUIDANCE = `Cua Driver native computer-use tools operate the host desktop. Discover the exact app and window, then get a fresh window snapshot before acting. Use element_token from that snapshot, or coordinates from its screenshot. A new snapshot of that window invalidates its earlier element tokens. Select either target or the legacy pid/window_id fields; do not combine them.
|
|
29
|
+
|
|
30
|
+
Prefer background delivery. A refusal does not authorize a foreground retry. Verify the requested outcome from fresh state after an action; a delivered click alone does not prove the outcome. After cancellation, inspect current state before retrying because completed input is not rolled back. Other sessions and applications may change the same desktop.
|
|
31
|
+
|
|
32
|
+
On macOS, cursor-overlay operations may return facility_unavailable even when screenshots and input work.`;
|
|
33
|
+
/**
|
|
34
|
+
* Own one native runtime and expose its catalog through the MCP result adapter.
|
|
35
|
+
* Startup failures roll back every registration. Unload removes tools, aborts
|
|
36
|
+
* calls and image admission, awaits settlement and SDK shutdown, then releases computer use.
|
|
37
|
+
* @param ctx - context providing the exclusive registration and tool services.
|
|
38
|
+
* @returns after native import, runtime creation, and tool discovery complete.
|
|
39
|
+
*/
|
|
40
|
+
async function apply(ctx) {
|
|
41
|
+
const lifetime = new AbortController();
|
|
42
|
+
const pending = /* @__PURE__ */ new Set();
|
|
43
|
+
let driver;
|
|
44
|
+
ctx.on("internal/plugin", (fiber) => {
|
|
45
|
+
if (fiber === ctx.fiber && fiber.uid === null) lifetime.abort();
|
|
46
|
+
}, { global: true });
|
|
47
|
+
let ready = Promise.resolve();
|
|
48
|
+
const dispose = ctx.effect(function* () {
|
|
49
|
+
yield ctx.computerUse.register(ComputerUseProviderName("cua-driver-native"));
|
|
50
|
+
yield async () => {
|
|
51
|
+
lifetime.abort();
|
|
52
|
+
await ready.catch(() => {});
|
|
53
|
+
await Promise.allSettled(pending);
|
|
54
|
+
if (driver !== void 0) {
|
|
55
|
+
await driver.shutdown();
|
|
56
|
+
driver.uniffiDestroy();
|
|
57
|
+
}
|
|
58
|
+
};
|
|
59
|
+
const child = ctx.plugin({
|
|
60
|
+
name: "computer-use-cua-driver-native-runtime",
|
|
61
|
+
inject: ["tools", "systemPrompt"],
|
|
62
|
+
apply: mountRuntime
|
|
63
|
+
});
|
|
64
|
+
yield child.dispose;
|
|
65
|
+
ready = Promise.resolve(child).then(() => {});
|
|
66
|
+
}, "computer-use-cua-driver-native.runtime");
|
|
67
|
+
try {
|
|
68
|
+
await ready;
|
|
69
|
+
} catch (error) {
|
|
70
|
+
await dispose();
|
|
71
|
+
throw error;
|
|
72
|
+
}
|
|
73
|
+
/** The child owns tool registrations; the outer effect owns native teardown. */
|
|
74
|
+
async function mountRuntime(inner) {
|
|
75
|
+
const { CuaDriver } = await import("@trycua/cua-driver");
|
|
76
|
+
lifetime.signal.throwIfAborted();
|
|
77
|
+
const activeDriver = driver = CuaDriver.create(void 0);
|
|
78
|
+
const catalog = ToolCatalog.parse(JSON.parse(await activeDriver.listToolsJson({ signal: lifetime.signal })));
|
|
79
|
+
lifetime.signal.throwIfAborted();
|
|
80
|
+
const names = /* @__PURE__ */ new Set();
|
|
81
|
+
for (const tool of catalog.tools) {
|
|
82
|
+
const publicName = `cua_driver_native__${tool.name}`;
|
|
83
|
+
if (!TOOL_NAME.test(publicName)) throw new Error(`Cua Driver tool "${tool.name}" exceeds the supported function-name format`);
|
|
84
|
+
if (names.has(publicName)) throw new Error(`Cua Driver listed tool "${tool.name}" more than once`);
|
|
85
|
+
names.add(publicName);
|
|
86
|
+
const definition = createMcpToolDefinition(inner, {
|
|
87
|
+
name: publicName,
|
|
88
|
+
rawName: tool.name,
|
|
89
|
+
description: tool.description ?? "",
|
|
90
|
+
inputSchema: tool.inputSchema,
|
|
91
|
+
outputSchema: tool.outputSchema,
|
|
92
|
+
async call(args, execution) {
|
|
93
|
+
const combined = AbortSignal.any([execution.signal, lifetime.signal]);
|
|
94
|
+
combined.throwIfAborted();
|
|
95
|
+
const result = await activeDriver.callTool(tool.name, JSON.stringify(args), { signal: combined });
|
|
96
|
+
combined.throwIfAborted();
|
|
97
|
+
return JSON.parse(result.rawJson);
|
|
98
|
+
}
|
|
99
|
+
});
|
|
100
|
+
inner.tools.register(definition);
|
|
101
|
+
}
|
|
102
|
+
inner.on("tools/execute", async (exec, next) => {
|
|
103
|
+
if (!names.has(exec.name)) return next();
|
|
104
|
+
const upstream = exec.signal;
|
|
105
|
+
exec.signal = AbortSignal.any([upstream, lifetime.signal]);
|
|
106
|
+
const operation = Promise.resolve().then(next);
|
|
107
|
+
pending.add(operation);
|
|
108
|
+
try {
|
|
109
|
+
return await operation;
|
|
110
|
+
} finally {
|
|
111
|
+
pending.delete(operation);
|
|
112
|
+
exec.signal = upstream;
|
|
113
|
+
}
|
|
114
|
+
});
|
|
115
|
+
inner.systemPrompt.section({
|
|
116
|
+
name: "computer-use:cua-driver-native",
|
|
117
|
+
order: inner.systemPrompt.getSectionOrder("TOOL_COMPUTER_USE"),
|
|
118
|
+
text: GUIDANCE
|
|
119
|
+
});
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
//#endregion
|
|
123
|
+
export { Config, apply, inject, name };
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Computer use through the in-process Cua Driver native SDK and its own tools.
|
|
3
|
+
* @module @deepseek-ai/dsh-experimental-computer-use-cua-driver-native
|
|
4
|
+
*/
|
|
5
|
+
import type { Context } from '@deepseek-ai/cordis';
|
|
6
|
+
import Schema from '@deepseek-ai/schemastery';
|
|
7
|
+
/** Cordis plugin identity for the native Cua Driver provider. */
|
|
8
|
+
export declare const name = "experimental-computer-use-cua-driver-native";
|
|
9
|
+
/** Services required before the native runtime can publish tools. */
|
|
10
|
+
export declare const inject: string[];
|
|
11
|
+
/** The native provider uses the installed SDK's same-process defaults. */
|
|
12
|
+
export declare const Config: Schema<Schemastery.ObjectS<{}>, Schemastery.ObjectT<{}>>;
|
|
13
|
+
/**
|
|
14
|
+
* Own one native runtime and expose its catalog through the MCP result adapter.
|
|
15
|
+
* Startup failures roll back every registration. Unload removes tools, aborts
|
|
16
|
+
* calls and image admission, awaits settlement and SDK shutdown, then releases computer use.
|
|
17
|
+
* @param ctx - context providing the exclusive registration and tool services.
|
|
18
|
+
* @returns after native import, runtime creation, and tool discovery complete.
|
|
19
|
+
*/
|
|
20
|
+
export declare function apply(ctx: Context): Promise<void>;
|
|
21
|
+
//# sourceMappingURL=index.d.ts.map
|
package/package.json
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@deepseek-ai/dsh-experimental-computer-use-cua-driver-native",
|
|
3
|
+
"description": "Experimental computer-use provider embedding the Cua Driver native npm SDK",
|
|
4
|
+
"version": "0.1.6-alpha.1",
|
|
5
|
+
"publishConfig": {
|
|
6
|
+
"access": "public"
|
|
7
|
+
},
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
|
11
|
+
"directory": "packages/experimental/computer-use-cua-driver-native"
|
|
12
|
+
},
|
|
13
|
+
"type": "module",
|
|
14
|
+
"main": "lib/index.js",
|
|
15
|
+
"types": "lib/types/index.d.ts",
|
|
16
|
+
"exports": {
|
|
17
|
+
".": {
|
|
18
|
+
"types": "./lib/types/index.d.ts",
|
|
19
|
+
"default": "./lib/index.js"
|
|
20
|
+
},
|
|
21
|
+
"./src/*": "./src/*",
|
|
22
|
+
"./package.json": "./package.json"
|
|
23
|
+
},
|
|
24
|
+
"files": [
|
|
25
|
+
"lib/index.js",
|
|
26
|
+
"lib/types/**/*.d.ts"
|
|
27
|
+
],
|
|
28
|
+
"license": "MIT",
|
|
29
|
+
"peerDependencies": {
|
|
30
|
+
"@deepseek-ai/cordis": "^4.0.2",
|
|
31
|
+
"@deepseek-ai/dsh-computer-use": "^0.1.6-alpha.1",
|
|
32
|
+
"@deepseek-ai/dsh-system-prompt": "^0.1.6-alpha.1",
|
|
33
|
+
"@deepseek-ai/dsh-tools": "^0.1.6-alpha.1"
|
|
34
|
+
},
|
|
35
|
+
"dependencies": {
|
|
36
|
+
"@trycua/cua-driver": "0.28.0",
|
|
37
|
+
"zod": "^4.4.3",
|
|
38
|
+
"@deepseek-ai/dsh-mcp-client": "^0.1.6-alpha.1",
|
|
39
|
+
"@deepseek-ai/schemastery": "^3.18.2"
|
|
40
|
+
},
|
|
41
|
+
"devDependencies": {
|
|
42
|
+
"@deepseek-ai/cordis": "^4.0.2",
|
|
43
|
+
"@deepseek-ai/cordis-plugin-include": "^1.0.7",
|
|
44
|
+
"@deepseek-ai/dsh-agent": "^0.1.6-alpha.1",
|
|
45
|
+
"@deepseek-ai/cordis-plugin-loader": "^1.0.3",
|
|
46
|
+
"@deepseek-ai/dsh-agent-loop-testkit": "^0.1.6-alpha.1",
|
|
47
|
+
"@deepseek-ai/dsh-agent-loop": "^0.1.6-alpha.1",
|
|
48
|
+
"@deepseek-ai/dsh-attachment": "^0.1.6-alpha.1",
|
|
49
|
+
"@deepseek-ai/dsh-attachment-local": "^0.1.6-alpha.1",
|
|
50
|
+
"@deepseek-ai/dsh-computer-use": "^0.1.6-alpha.1",
|
|
51
|
+
"@deepseek-ai/dsh-llm": "^0.1.6-alpha.1",
|
|
52
|
+
"@deepseek-ai/dsh-session": "^0.1.6-alpha.1",
|
|
53
|
+
"@deepseek-ai/dsh-system-prompt": "^0.1.6-alpha.1",
|
|
54
|
+
"@deepseek-ai/dsh-tools": "^0.1.6-alpha.1",
|
|
55
|
+
"@deepseek-ai/dsh-session-projection": "^0.1.6-alpha.1"
|
|
56
|
+
}
|
|
57
|
+
}
|