@autoark-ai/eva-client-sdk-ts 0.0.2-dev
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/GATEWAY_TERMS.md +11 -0
- package/LICENSE +100 -0
- package/README.md +344 -0
- package/THIRD_PARTY_NOTICES.md +116 -0
- package/dist/assets/silero_vad_v6.onnx +0 -0
- package/dist/browser.d.ts +84 -0
- package/dist/browser.js +939 -0
- package/dist/index.d.ts +406 -0
- package/dist/index.js +3 -0
- package/dist/spi.d.ts +130 -0
- package/dist/spi.js +0 -0
- package/package.json +79 -0
package/GATEWAY_TERMS.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# AutoArk AI Gateway Service Notice
|
|
2
|
+
|
|
3
|
+
Version 1.0 — Effective Date: July 20, 2026
|
|
4
|
+
|
|
5
|
+
The official Eva Gateway is a hosted service separate from the Eva Client SDK. The SDK license does not grant any right to access or use the Gateway.
|
|
6
|
+
|
|
7
|
+
Access to and use of the Gateway are governed by the terms and policies published on the official AutoArk AI website:
|
|
8
|
+
|
|
9
|
+
https://autoarkai.com/
|
|
10
|
+
|
|
11
|
+
Do not access or use the Gateway if you do not agree to those terms and policies.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# AutoArk AI Proprietary SDK License Agreement
|
|
2
|
+
|
|
3
|
+
Version 1.0 — Effective Date: July 20, 2026
|
|
4
|
+
|
|
5
|
+
This Proprietary SDK License Agreement (the “Agreement”) governs your use of the Eva Client SDK package published as `@autoark-ai/eva-client-sdk-ts` (the “SDK”). “Licensor” means AutoArk AI and the legal entity identified as the SDK publisher or operator on the applicable official website, order form, or service console. “You” or “Licensee” means the individual or entity that downloads, installs, copies, or uses the SDK.
|
|
6
|
+
|
|
7
|
+
By downloading, installing, copying, or using the SDK, You accept this Agreement. If You use the SDK for an entity, You represent that You have authority to bind that entity. If You do not agree, do not use the SDK.
|
|
8
|
+
|
|
9
|
+
## 1. Definitions
|
|
10
|
+
|
|
11
|
+
- “Application” means a software product or service developed and controlled by Licensee that incorporates the SDK for use as part of that product or service.
|
|
12
|
+
- “Compiled Output” means JavaScript bundles or other executable artifacts produced from an unmodified SDK through the Permitted Build Operations in Section 3.
|
|
13
|
+
- “Official Gateway” means the Eva Gateway endpoints selected and embedded by Licensor in an authorized SDK release.
|
|
14
|
+
- “Third-Party Components” means software or model assets identified in `THIRD_PARTY_NOTICES.md`.
|
|
15
|
+
|
|
16
|
+
## 2. Limited License Grant
|
|
17
|
+
|
|
18
|
+
Subject to continuous compliance with this Agreement, Licensor grants Licensee a free-of-charge, non-exclusive, non-transferable, non-sublicensable, revocable, limited license to:
|
|
19
|
+
|
|
20
|
+
1. install and use the SDK solely to develop, test, operate, and maintain Licensee’s Applications;
|
|
21
|
+
2. integrate the SDK into Licensee’s Applications; and
|
|
22
|
+
3. reproduce and distribute Compiled Output solely as an inseparable part of Licensee’s Applications to Licensee’s end users.
|
|
23
|
+
|
|
24
|
+
No right is granted to distribute the SDK as a standalone product or to make the SDK available for use independently from an Application. This license does not include any right to access or use the Official Gateway; Gateway access is governed separately under Section 5.
|
|
25
|
+
|
|
26
|
+
## 3. Permitted Build Operations
|
|
27
|
+
|
|
28
|
+
The following ordinary build operations do not violate the prohibition on modification, provided they are performed only to incorporate the SDK into an Application and do not alter its functional behavior, security controls, Gateway binding, notices, or license terms:
|
|
29
|
+
|
|
30
|
+
- compilation or transpilation;
|
|
31
|
+
- linking and module resolution;
|
|
32
|
+
- bundling;
|
|
33
|
+
- minification and compression;
|
|
34
|
+
- Tree Shaking and dead-code elimination;
|
|
35
|
+
- code signing; and
|
|
36
|
+
- format conversion reasonably necessary to package or deploy the Application.
|
|
37
|
+
|
|
38
|
+
The resulting artifact remains subject to this Agreement. This Section does not permit source-level functional changes, patching, instrumentation intended to expose internal interfaces, removal of protections, or creation of a substitute or derivative SDK.
|
|
39
|
+
|
|
40
|
+
## 4. Restrictions
|
|
41
|
+
|
|
42
|
+
Except for the rights expressly granted in Sections 2 and 3, Licensee shall not, and shall not enable or assist any third party to:
|
|
43
|
+
|
|
44
|
+
1. modify, adapt, translate, patch, or create derivative works of the SDK;
|
|
45
|
+
2. reverse engineer, decompile, disassemble, decode, or otherwise attempt to discover the source code, internal protocols, non-public interfaces, models, keys, or underlying ideas of the SDK;
|
|
46
|
+
3. separately publish, redistribute, sell, resell, sublicense, rent, lease, lend, host, time-share, or provide the SDK as a service or managed component;
|
|
47
|
+
4. remove, obscure, or alter any copyright, proprietary, attribution, security, or license notice;
|
|
48
|
+
5. bypass, disable, defeat, or interfere with any technical restriction, usage control, authentication, metering, or security mechanism;
|
|
49
|
+
6. use the SDK, when initiating or participating in large-model or other model inference, with any endpoint other than the Official Gateway;
|
|
50
|
+
7. bypass, replace, proxy, redirect, imitate, emulate, or simulate the Official Gateway, including by reproducing its protocol for production or commercial inference;
|
|
51
|
+
8. use the SDK to develop, benchmark for publication, or provide a competing SDK, Gateway, model-routing service, or substantially similar substitute; or
|
|
52
|
+
9. use the SDK in violation of applicable law, sanctions, export controls, third-party rights, or the terms published for the Official Gateway.
|
|
53
|
+
|
|
54
|
+
Any right not expressly granted is reserved by Licensor.
|
|
55
|
+
|
|
56
|
+
Item 7 does not prohibit local test doubles that perform no model inference and do not bypass authentication, metering, security controls, or the Official Gateway requirement in production.
|
|
57
|
+
|
|
58
|
+
## 5. Gateway Services Are Separate
|
|
59
|
+
|
|
60
|
+
The SDK and the Official Gateway are separate offerings. This Agreement governs only the SDK. Access to and use of the Official Gateway are governed separately as stated in `GATEWAY_TERMS.md`.
|
|
61
|
+
|
|
62
|
+
## 6. Distribution Conditions
|
|
63
|
+
|
|
64
|
+
Licensee may distribute Compiled Output under Section 2 only if:
|
|
65
|
+
|
|
66
|
+
1. the Compiled Output is an inseparable part of an Application with material independent functionality;
|
|
67
|
+
2. the Application does not expose the SDK for standalone use, extraction, or reuse;
|
|
68
|
+
3. Licensee does not represent that it owns the SDK or grant recipients broader rights in the SDK than this Agreement permits;
|
|
69
|
+
4. all proprietary and Third-Party Component notices remain intact where technically included; and
|
|
70
|
+
5. Licensee is solely responsible for its Application, end-user terms, privacy disclosures, security, support, and legal compliance.
|
|
71
|
+
|
|
72
|
+
## 7. Third-Party Components
|
|
73
|
+
|
|
74
|
+
Third-Party Components are licensed under the separate terms identified in `THIRD_PARTY_NOTICES.md`. Those terms apply only to the identified components and do not make the SDK as a whole open source or expand the license granted under this Agreement. If a third-party license directly conflicts with this Agreement for that component, the third-party license controls only for that component.
|
|
75
|
+
|
|
76
|
+
## 8. Ownership and Feedback
|
|
77
|
+
|
|
78
|
+
The SDK is licensed, not sold. As between Licensor and Licensee, Licensor and its licensors retain all right, title, and interest in and to the SDK, its architecture, non-public interfaces, updates, documentation, trademarks, and all related intellectual property. Licensee retains ownership of its Application, excluding the SDK and Third-Party Components.
|
|
79
|
+
|
|
80
|
+
If Licensee voluntarily provides suggestions or feedback, Licensee grants Licensor a perpetual, irrevocable, worldwide, royalty-free right to use and incorporate that feedback without restriction or obligation.
|
|
81
|
+
|
|
82
|
+
## 9. Updates and Compatibility
|
|
83
|
+
|
|
84
|
+
Licensor may update, replace, limit, or discontinue the SDK or any feature. Unless Licensor expressly states otherwise, Licensor has no obligation to provide maintenance, support, backward compatibility, security updates, or any particular release. A new SDK version may include updated terms presented with that version; continued use of the new version constitutes acceptance of those terms.
|
|
85
|
+
|
|
86
|
+
## 10. Term and Termination
|
|
87
|
+
|
|
88
|
+
This Agreement begins when Licensee first downloads, installs, copies, or uses the SDK and continues until terminated. Licensor may revoke or terminate the license immediately if Licensee breaches this Agreement, creates security or legal risk, fails to comply with applicable Gateway terms, or if continued distribution is prohibited by law. Licensor may otherwise revoke the license on at least 30 days’ notice.
|
|
89
|
+
|
|
90
|
+
Upon termination, Licensee shall stop using and distributing the SDK, stop publishing new or updated Applications containing the SDK or Compiled Output, and delete all SDK copies under its control, except copies retained solely where required by law. End users may continue using Compiled Output lawfully distributed before termination, unless continued operation is prohibited by law or the termination notice identifies a specific security or legal risk that requires it to stop. Sections 4, 7, 8, 10, and 11 survive termination.
|
|
91
|
+
|
|
92
|
+
## 11. Governing Law and General Terms
|
|
93
|
+
|
|
94
|
+
This Agreement is governed by the laws of the People’s Republic of China, excluding its conflict-of-law rules. Any dispute that cannot be resolved through good-faith negotiation shall be submitted to the competent people’s court at Licensor’s domicile, except where mandatory law requires otherwise.
|
|
95
|
+
|
|
96
|
+
If any provision is unenforceable, it shall be enforced to the maximum lawful extent and the remaining provisions remain effective. Failure to enforce a provision is not a waiver. Licensee may not assign this Agreement by contract, merger, change of control, or operation of law without Licensor’s prior written consent. Licensor may assign this Agreement as part of a reorganization, financing, or transfer of the SDK business.
|
|
97
|
+
|
|
98
|
+
This Agreement, together with documents expressly incorporated into it, is the entire agreement about the SDK and supersedes prior discussions about the SDK license. Gateway services remain governed separately. Headings are for convenience only.
|
|
99
|
+
|
|
100
|
+
Questions about this Agreement must be directed through the legal or support contact published on the official AutoArk AI website or Gateway console.
|
package/README.md
ADDED
|
@@ -0,0 +1,344 @@
|
|
|
1
|
+
# Eva TypeScript SDK
|
|
2
|
+
|
|
3
|
+
`@autoark-ai/eva-client-sdk-ts` 是浏览器优先的 Eva 多轮语音对话 SDK。它提供一个稳定的 Agent Facade、可观察的消息与事件,以及可替换的音频输入、输出、AEC 和摄像头扩展点。
|
|
4
|
+
|
|
5
|
+
## 安装
|
|
6
|
+
|
|
7
|
+
当前预发布版本通过 `dev` dist-tag 提供:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm install @autoark-ai/eva-client-sdk-ts@dev
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
请固定生产应用实际验收过的完整 package version;不要把 `dev` dist-tag 当作稳定性承诺。
|
|
14
|
+
|
|
15
|
+
## 环境要求
|
|
16
|
+
|
|
17
|
+
- 前端构建环境要求 Node.js `>=20`。
|
|
18
|
+
- 浏览器运行已在 Chromium-based browsers、Safari 验证。
|
|
19
|
+
|
|
20
|
+
## 版本与 Gateway 环境
|
|
21
|
+
|
|
22
|
+
候选版本使用 `<代码版本>-<成熟度>.<Gateway环境>`:`alpha`、`beta`、`rc` 表示成熟度,`dev`、`prod` 表示制品内建的并行 Gateway 环境。例如 `X.Y.Z-alpha.dev` 调用 dev Gateway,`X.Y.Z-alpha.prod` 调用 prod Gateway;正式代码的 dev 制品使用 `X.Y.Z-dev`,正式 prod 制品使用 `X.Y.Z`。
|
|
23
|
+
|
|
24
|
+
每个完整 package version 只对应一组内建 endpoint 和一份确定制品。Gateway 环境不能通过 Agent 配置改写;需要切换环境时应使用对应版本的 SDK 包。
|
|
25
|
+
|
|
26
|
+
包只提供三个入口:
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import {
|
|
30
|
+
createEvaVoiceDialogueAgent,
|
|
31
|
+
EvaSdkError,
|
|
32
|
+
type AgentEvent,
|
|
33
|
+
type ConversationMessage,
|
|
34
|
+
type EvaVoiceDialogueAgent,
|
|
35
|
+
type EvaVoiceDialogueAgentConfig,
|
|
36
|
+
} from "@autoark-ai/eva-client-sdk-ts";
|
|
37
|
+
|
|
38
|
+
import type {
|
|
39
|
+
AecProcessor,
|
|
40
|
+
AudioInputSource,
|
|
41
|
+
AudioOutputSink,
|
|
42
|
+
CameraSnapshotSource,
|
|
43
|
+
MediaTransportsConfig,
|
|
44
|
+
} from "@autoark-ai/eva-client-sdk-ts/spi";
|
|
45
|
+
|
|
46
|
+
import {
|
|
47
|
+
createBrowserAudioInputSource,
|
|
48
|
+
createBrowserAudioOutputSink,
|
|
49
|
+
createBrowserCameraSnapshotSource,
|
|
50
|
+
createPassthroughAecProcessor,
|
|
51
|
+
} from "@autoark-ai/eva-client-sdk-ts/browser";
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## 浏览器最小用法
|
|
55
|
+
|
|
56
|
+
下面的 model 与 voice 组合已在 browser example 验证:
|
|
57
|
+
|
|
58
|
+
```ts
|
|
59
|
+
import {
|
|
60
|
+
createEvaVoiceDialogueAgent,
|
|
61
|
+
type AgentEvent,
|
|
62
|
+
} from "@autoark-ai/eva-client-sdk-ts";
|
|
63
|
+
import type { MediaTransportsConfig } from "@autoark-ai/eva-client-sdk-ts/spi";
|
|
64
|
+
import {
|
|
65
|
+
createBrowserAudioInputSource,
|
|
66
|
+
createBrowserAudioOutputSink,
|
|
67
|
+
createBrowserCameraSnapshotSource,
|
|
68
|
+
createPassthroughAecProcessor,
|
|
69
|
+
} from "@autoark-ai/eva-client-sdk-ts/browser";
|
|
70
|
+
|
|
71
|
+
const transports: MediaTransportsConfig = {
|
|
72
|
+
input: createBrowserAudioInputSource({
|
|
73
|
+
echoCancellation: true,
|
|
74
|
+
noiseSuppression: true,
|
|
75
|
+
autoGainControl: true,
|
|
76
|
+
maxBufferedChunks: 64,
|
|
77
|
+
}),
|
|
78
|
+
output: createBrowserAudioOutputSink(),
|
|
79
|
+
aec: createPassthroughAecProcessor(),
|
|
80
|
+
camera: createBrowserCameraSnapshotSource({
|
|
81
|
+
video: { facingMode: { ideal: "environment" } },
|
|
82
|
+
mimeType: "image/png",
|
|
83
|
+
}),
|
|
84
|
+
};
|
|
85
|
+
|
|
86
|
+
const agent = createEvaVoiceDialogueAgent({
|
|
87
|
+
apiKey: applicationManagedApiKey,
|
|
88
|
+
asr: {
|
|
89
|
+
model: "ark_asr_3b",
|
|
90
|
+
sampleRate: 16_000,
|
|
91
|
+
},
|
|
92
|
+
llm: {
|
|
93
|
+
model: "qiduoduo_chat_vlm",
|
|
94
|
+
},
|
|
95
|
+
tts: {
|
|
96
|
+
model: "cosyvoice_tts",
|
|
97
|
+
voice: "longjielidou_v3",
|
|
98
|
+
sampleRate: 48_000,
|
|
99
|
+
},
|
|
100
|
+
vad: {
|
|
101
|
+
sensitivity: 0.6,
|
|
102
|
+
silenceThresholdMs: 400,
|
|
103
|
+
},
|
|
104
|
+
transports,
|
|
105
|
+
history: { maxTurns: 10 },
|
|
106
|
+
camera: { captureTimeoutMs: 1500 },
|
|
107
|
+
});
|
|
108
|
+
|
|
109
|
+
const unsubscribe = agent.onEvent((event: AgentEvent) => {
|
|
110
|
+
if (event.type === "reply.partial") {
|
|
111
|
+
console.log(event.text);
|
|
112
|
+
}
|
|
113
|
+
});
|
|
114
|
+
|
|
115
|
+
await agent.start();
|
|
116
|
+
await agent.setAudioInputEnabled(true);
|
|
117
|
+
// 摄像头默认关闭;需要时显式开启,开启期间持续持有一个 session。
|
|
118
|
+
await agent.setCameraCaptureEnabled(true);
|
|
119
|
+
await agent.submitText("你好");
|
|
120
|
+
|
|
121
|
+
console.log(agent.getMessages());
|
|
122
|
+
|
|
123
|
+
await agent.setTtsEnabled(false);
|
|
124
|
+
await agent.stop();
|
|
125
|
+
unsubscribe();
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## Agent Facade
|
|
129
|
+
|
|
130
|
+
`createEvaVoiceDialogueAgent(config)` 返回的 `EvaVoiceDialogueAgent` 只有八个方法:
|
|
131
|
+
|
|
132
|
+
```ts
|
|
133
|
+
interface EvaVoiceDialogueAgent {
|
|
134
|
+
start(): Promise<void>;
|
|
135
|
+
submitText(text: string, options?: SubmitTextOptions): Promise<void>;
|
|
136
|
+
setAudioInputEnabled(enabled: boolean): Promise<void>;
|
|
137
|
+
setCameraCaptureEnabled(enabled: boolean): Promise<void>;
|
|
138
|
+
setTtsEnabled(enabled: boolean): Promise<void>;
|
|
139
|
+
getMessages(): readonly ConversationMessage[];
|
|
140
|
+
onEvent(listener: AgentEventListener): Unsubscribe;
|
|
141
|
+
stop(): Promise<void>;
|
|
142
|
+
}
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
| 方法 | created | running | stopped |
|
|
146
|
+
|---|---|---|---|
|
|
147
|
+
| `start()` | 启动 Agent;并发调用共享结果 | 幂等 | 拒绝 |
|
|
148
|
+
| `submitText()` | 拒绝 | 提交文本 turn,可打断当前 turn | 拒绝 |
|
|
149
|
+
| `setAudioInputEnabled()` | 设置初始麦克风状态 | 开启或关闭后续音频输入 | 拒绝 |
|
|
150
|
+
| `setCameraCaptureEnabled()` | 设置初始摄像头状态,默认关闭 | 开启或关闭持续 camera session | 拒绝 |
|
|
151
|
+
| `setTtsEnabled()` | 设置初始 TTS 状态 | 控制后续合成与播放 | 拒绝 |
|
|
152
|
+
| `getMessages()` | 返回空快照 | 返回当前最终消息快照 | 仍可读取 |
|
|
153
|
+
| `onEvent()` | 可订阅 | 可订阅 | 同步拒绝新订阅 |
|
|
154
|
+
| `stop()` | 进入终态 | 完成收尾后进入终态 | 幂等 |
|
|
155
|
+
|
|
156
|
+
Agent 一旦开始停止,就不再接受新的 turn 或事件订阅。需要新会话时,请创建新的 Agent。
|
|
157
|
+
|
|
158
|
+
## 配置
|
|
159
|
+
|
|
160
|
+
`EvaVoiceDialogueAgentConfig` 的公共字段如下:
|
|
161
|
+
|
|
162
|
+
| 字段 | 必填 | 说明 |
|
|
163
|
+
|---|---:|---|
|
|
164
|
+
| `apiKey` | 是 | 应用提供并管理的 Eva Gateway AK |
|
|
165
|
+
| `asr.model` | 是 | ASR model 标识 |
|
|
166
|
+
| `asr.sampleRate` | 是 | ASR 接收的目标 PCM 采样率,必须为正整数 |
|
|
167
|
+
| `llm.model` | 是 | LLM model 标识 |
|
|
168
|
+
| `llm.temperature` | 否 | 采样温度 |
|
|
169
|
+
| `llm.maxTokens` | 否 | 最大生成 token 数 |
|
|
170
|
+
| `tts.model` | 是 | TTS model 标识 |
|
|
171
|
+
| `tts.voice` | 否 | voice 标识 |
|
|
172
|
+
| `tts.speakingRate` | 否 | 语速 |
|
|
173
|
+
| `tts.pitch` | 否 | 音调倍率 |
|
|
174
|
+
| `tts.sampleRate` | 否 | TTS 输出采样率,默认 `16000` |
|
|
175
|
+
| `vad.sensitivity` | 否 | 语音概率阈值,默认 `0.5` |
|
|
176
|
+
| `vad.silenceThresholdMs` | 否 | 判定停止说话所需的连续静音时间,默认 `200` ms |
|
|
177
|
+
| `systemPrompt` | 否 | 每次 LLM 请求使用的系统指令 |
|
|
178
|
+
| `greeting` | 否 | `disabled`、`static` 或 `dynamic` greeting |
|
|
179
|
+
| `history.maxTurns` | 否 | LLM 上下文保留的已完成轮数 |
|
|
180
|
+
| `camera.captureTimeoutMs` | 否 | 单次采图等待上限,默认 `1500` ms |
|
|
181
|
+
| `metadata` | 否 | JSON-compatible Agent metadata |
|
|
182
|
+
| `transports` | 否 | 完整的 `input`、`output`、`aec` 与可选 `camera` 组合;省略时为纯文本 Agent |
|
|
183
|
+
|
|
184
|
+
当配置 `transports.input` 时,应同时提供 `vad`。`submitText()` 可通过 `SubmitTextOptions` 指定 `turnId` 和当前 turn 的 `metadata`。
|
|
185
|
+
|
|
186
|
+
## 消息
|
|
187
|
+
|
|
188
|
+
`getMessages()` 同步返回按提交顺序排列的只读快照。每条 `ConversationMessage` 包含:
|
|
189
|
+
|
|
190
|
+
```ts
|
|
191
|
+
interface ConversationMessage {
|
|
192
|
+
readonly id: string;
|
|
193
|
+
readonly turnId: string;
|
|
194
|
+
readonly role: "user" | "assistant";
|
|
195
|
+
readonly content: string;
|
|
196
|
+
readonly createdAt: number;
|
|
197
|
+
readonly metadata: JsonObject;
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
user 与 assistant 的最终文本才会进入消息列表。同一轮的两条消息共享 `turnId`。`history.maxTurns` 只控制送给 LLM 的上下文,不会裁剪 `getMessages()` 的会话记录。
|
|
202
|
+
|
|
203
|
+
## 事件
|
|
204
|
+
|
|
205
|
+
`onEvent()` 接收 `AgentEvent` discriminated union。`AgentEventType` 是 type-only 字符串联合:
|
|
206
|
+
|
|
207
|
+
| `type` | 主要字段 | 含义 |
|
|
208
|
+
|---|---|---|
|
|
209
|
+
| `speech.started` | — | 检测到开始说话 |
|
|
210
|
+
| `image.captured` | `image` | 当前语音 turn 已在本地成功采图;不表示模型已接受图片 |
|
|
211
|
+
| `speech.stopped` | — | 检测到停止说话 |
|
|
212
|
+
| `transcript.partial` | `text`, `source: "speech"` | 增量转写 |
|
|
213
|
+
| `transcript.final` | `text`, `source` | 最终用户文本 |
|
|
214
|
+
| `interruption` | `reason` | 用户语音或手动文本打断当前 turn |
|
|
215
|
+
| `reply.started` | — | assistant 开始回复 |
|
|
216
|
+
| `reply.partial` | `text` | 新增回复片段 |
|
|
217
|
+
| `reply.final` | `text` | 完整最终回复 |
|
|
218
|
+
| `playback.started` | — | TTS 开始播放 |
|
|
219
|
+
| `playback.stopped` | — | TTS 停止播放 |
|
|
220
|
+
| `turn.latency` | `latency` | turn 总耗时与可用的阶段耗时 |
|
|
221
|
+
| `error` | `error` | 可交给应用处理的结构化错误 |
|
|
222
|
+
|
|
223
|
+
所有事件都有 `streamId`、`partial`、`final` 和只读 `metadata`;除无法定位 turn 的错误外都有 `turnId`。还可能包含 `sequence`、`timestamp` 和 `frameId`。
|
|
224
|
+
|
|
225
|
+
建议用穷尽 switch 消费事件:
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
function consume(event: AgentEvent): void {
|
|
229
|
+
switch (event.type) {
|
|
230
|
+
case "speech.started":
|
|
231
|
+
case "image.captured":
|
|
232
|
+
case "speech.stopped":
|
|
233
|
+
case "reply.started":
|
|
234
|
+
case "playback.started":
|
|
235
|
+
case "playback.stopped":
|
|
236
|
+
return;
|
|
237
|
+
case "transcript.partial":
|
|
238
|
+
case "transcript.final":
|
|
239
|
+
case "reply.partial":
|
|
240
|
+
case "reply.final":
|
|
241
|
+
console.log(event.text);
|
|
242
|
+
return;
|
|
243
|
+
case "interruption":
|
|
244
|
+
console.log(event.reason);
|
|
245
|
+
return;
|
|
246
|
+
case "turn.latency":
|
|
247
|
+
console.log(event.latency);
|
|
248
|
+
return;
|
|
249
|
+
case "error":
|
|
250
|
+
console.error(event.error);
|
|
251
|
+
return;
|
|
252
|
+
default:
|
|
253
|
+
event satisfies never;
|
|
254
|
+
}
|
|
255
|
+
}
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
## 错误
|
|
259
|
+
|
|
260
|
+
同步输入校验和 lifecycle 错误会抛出 `EvaSdkError`。事件中的 `StructuredError` 提供:
|
|
261
|
+
|
|
262
|
+
```ts
|
|
263
|
+
interface StructuredError {
|
|
264
|
+
message: string;
|
|
265
|
+
fatal: boolean;
|
|
266
|
+
source: "sdk" | "provider" | "gateway" | "media";
|
|
267
|
+
provider?: string;
|
|
268
|
+
statusCode?: number;
|
|
269
|
+
role?: "audio-input" | "audio-output" | "aec" | "camera";
|
|
270
|
+
operation?: "start" | "capture" | "stop";
|
|
271
|
+
reason?: "not_configured" | "permission_denied" | "device_unavailable" | "unsupported" | "timeout" | "invalid_data" | "operation_failed";
|
|
272
|
+
}
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
事件和错误不会额外回显 AK、Authorization header 或原始 provider 响应。
|
|
276
|
+
|
|
277
|
+
## 可扩展 Media SPI
|
|
278
|
+
|
|
279
|
+
音频输入、音频输出、AEC 和摄像头都是可替换的公共扩展点。应用可以直接使用 SDK 提供的浏览器默认实现,也可以按角色接入自己的实现。
|
|
280
|
+
|
|
281
|
+
扩展契约从 `@autoark-ai/eva-client-sdk-ts/spi` 导入:
|
|
282
|
+
|
|
283
|
+
- **输入 `AudioInputSource`**:负责采集音频,通过 `frames()` 以 `AsyncIterable<AudioChunk>` 持续产出数据;适合接入浏览器麦克风、原生采集桥、文件或其它实时音频源。
|
|
284
|
+
- **输出 `AudioOutputSink`**:负责接收并播放音频,通过 `enqueue()`、`flush()`、`drain()` 和 `stop()` 管理队列与播放生命周期;可以替换为自定义播放器或原生音频输出。
|
|
285
|
+
- **AEC `AecProcessor`**:接收 far-end 播放音频,并在 near-end 输入进入对话链路前处理回声;可以使用 passthrough、平台 AEC 或应用自带的软件 AEC。
|
|
286
|
+
- **摄像头 `CameraSnapshotSource`**:以必需 `AbortSignal` 管理 `start()` / `capture()`,并通过幂等 `stop()` 释放 session;snapshot 只含图片 bytes、MIME 和实际尺寸。
|
|
287
|
+
|
|
288
|
+
四个角色可以分别替换;其中 `input`、`output` 与 `aec` 是 `MediaTransportsConfig` 必填项,`camera` 可选。构造 Agent 后,已装配对象的 lifecycle 由 Agent 独占驱动,应用不应再并发调用这些对象。
|
|
289
|
+
|
|
290
|
+
```ts
|
|
291
|
+
import type {
|
|
292
|
+
AecProcessor,
|
|
293
|
+
AudioInputSource,
|
|
294
|
+
AudioOutputSink,
|
|
295
|
+
CameraSnapshotSource,
|
|
296
|
+
MediaTransportsConfig,
|
|
297
|
+
} from "@autoark-ai/eva-client-sdk-ts/spi";
|
|
298
|
+
|
|
299
|
+
declare const customInput: AudioInputSource;
|
|
300
|
+
declare const customOutput: AudioOutputSink;
|
|
301
|
+
declare const customAec: AecProcessor;
|
|
302
|
+
declare const customCamera: CameraSnapshotSource;
|
|
303
|
+
|
|
304
|
+
const customTransports: MediaTransportsConfig = {
|
|
305
|
+
input: customInput,
|
|
306
|
+
output: customOutput,
|
|
307
|
+
aec: customAec,
|
|
308
|
+
camera: customCamera,
|
|
309
|
+
};
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
## SDK 提供的浏览器默认实现
|
|
313
|
+
|
|
314
|
+
如果不需要自定义媒体链路,可以从 `@autoark-ai/eva-client-sdk-ts/browser` 直接使用 SDK 提供的四个默认 factory:
|
|
315
|
+
|
|
316
|
+
| 角色 | 默认 factory | 行为 |
|
|
317
|
+
|---|---|---|
|
|
318
|
+
| 输入 | `createBrowserAudioInputSource()` | 使用浏览器麦克风采集音频 |
|
|
319
|
+
| 输出 | `createBrowserAudioOutputSink()` | 使用 Web Audio 播放 TTS 音频 |
|
|
320
|
+
| AEC | `createPassthroughAecProcessor()` | 不做软件回声处理,由浏览器和操作系统负责 AEC |
|
|
321
|
+
| 摄像头 | `createBrowserCameraSnapshotSource()` | 持续持有 video session,并在语音开始时采一张 PNG/JPEG |
|
|
322
|
+
|
|
323
|
+
```ts
|
|
324
|
+
const defaultTransports: MediaTransportsConfig = {
|
|
325
|
+
input: createBrowserAudioInputSource({ echoCancellation: true }),
|
|
326
|
+
output: createBrowserAudioOutputSink(),
|
|
327
|
+
aec: createPassthroughAecProcessor(),
|
|
328
|
+
camera: createBrowserCameraSnapshotSource(),
|
|
329
|
+
};
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
摄像头 factory 只是 `CameraSnapshotSource` 的默认浏览器实现,不会被 agent factory 自动补齐。不开启 camera 时不会申请摄像头权限;开启后可以直接问“图片里有什么”。图片只参与当前 LLM 请求,不进入 `getMessages()` 或后续 history。摄像头权限、设备、超时或本地采图失败会通过结构化 `media` error 可观测,并以纯文本继续当前 turn;已经发出的多模态请求若被模型拒绝,则按普通 gateway/provider error 结束,不会移除图片再做纯文本重试。
|
|
333
|
+
|
|
334
|
+
默认组合开启浏览器 input 的 `echoCancellation`,同时使用 passthrough AEC。如果接入自己的软件 AEC,应将 `echoCancellation` 设为 `false`,避免平台 AEC 与软件 AEC 重复处理。
|
|
335
|
+
|
|
336
|
+
## AK 责任边界
|
|
337
|
+
|
|
338
|
+
`apiKey` 由应用提供和管理。只要 AK 进入浏览器应用,最终用户就可能通过开发者工具观察到它;SDK 不提供浏览器端秘密存储。SDK 的边界是不会通过公共事件、错误、metadata 或配置读取接口再次暴露 AK。
|
|
339
|
+
|
|
340
|
+
## 许可与 Gateway 服务
|
|
341
|
+
|
|
342
|
+
本 SDK 是公开下载的专有软件,并非开源软件。安装或使用 SDK 即表示接受 [Proprietary SDK License Agreement](LICENSE):其中允许在应用中集成 SDK、执行正常的打包/压缩/Tree Shaking 等构建操作,并在应用内分发编译产物;禁止修改、逆向工程、制作衍生 SDK、单独再分发或绕过/替换/模拟官方 Gateway。
|
|
343
|
+
|
|
344
|
+
SDK 许可不包含 Gateway 使用权,也不承诺 Gateway 免费或无限使用;Gateway 适用规则见独立的 [Gateway Service Notice](GATEWAY_TERMS.md)。第三方软件与模型资产适用 [Third-Party Notices](THIRD_PARTY_NOTICES.md) 中列明的各自许可。
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# Third-Party Notices
|
|
2
|
+
|
|
3
|
+
This package redistributes or relies on the third-party components listed below. The notices and
|
|
4
|
+
license terms in this file apply only to those components, not to the Eva SDK as a whole.
|
|
5
|
+
|
|
6
|
+
## Silero VAD v6.2.1 ONNX model
|
|
7
|
+
|
|
8
|
+
Source: `https://github.com/snakers4/silero-vad/blob/v6.2.1/src/silero_vad/data/silero_vad.onnx`
|
|
9
|
+
|
|
10
|
+
Redistributed as: `dist/assets/silero_vad_v6.onnx`
|
|
11
|
+
|
|
12
|
+
SHA-256: `1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3`
|
|
13
|
+
|
|
14
|
+
MIT License
|
|
15
|
+
|
|
16
|
+
Copyright (c) 2020-present Silero Team
|
|
17
|
+
|
|
18
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
19
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
20
|
+
in the Software without restriction, including without limitation the rights
|
|
21
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
22
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
23
|
+
furnished to do so, subject to the following conditions:
|
|
24
|
+
|
|
25
|
+
The above copyright notice and this permission notice shall be included in all
|
|
26
|
+
copies or substantial portions of the Software.
|
|
27
|
+
|
|
28
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
29
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
30
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
31
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
32
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
33
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
34
|
+
SOFTWARE.
|
|
35
|
+
|
|
36
|
+
## ONNX Runtime Web 1.26.0
|
|
37
|
+
|
|
38
|
+
Source: `https://github.com/microsoft/onnxruntime/tree/v1.26.0/js/web`
|
|
39
|
+
|
|
40
|
+
Distributed as the exact direct dependency `onnxruntime-web@1.26.0`.
|
|
41
|
+
|
|
42
|
+
MIT License
|
|
43
|
+
|
|
44
|
+
Copyright (c) Microsoft Corporation
|
|
45
|
+
|
|
46
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
47
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
48
|
+
in the Software without restriction, including without limitation the rights
|
|
49
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
50
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
51
|
+
furnished to do so, subject to the following conditions:
|
|
52
|
+
|
|
53
|
+
The above copyright notice and this permission notice shall be included in all
|
|
54
|
+
copies or substantial portions of the Software.
|
|
55
|
+
|
|
56
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
57
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
58
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
59
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
60
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
61
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
62
|
+
SOFTWARE.
|
|
63
|
+
|
|
64
|
+
## @alexanderolsen/libsamplerate-js 2.1.2
|
|
65
|
+
|
|
66
|
+
Distributed as the exact direct dependency `@alexanderolsen/libsamplerate-js@2.1.2`.
|
|
67
|
+
|
|
68
|
+
MIT License
|
|
69
|
+
|
|
70
|
+
Copyright (c) 2021 Alexander Olsen
|
|
71
|
+
|
|
72
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
73
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
74
|
+
in the Software without restriction, including without limitation the rights
|
|
75
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
76
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
77
|
+
furnished to do so, subject to the following conditions:
|
|
78
|
+
|
|
79
|
+
The above copyright notice and this permission notice shall be included in all
|
|
80
|
+
copies or substantial portions of the Software.
|
|
81
|
+
|
|
82
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
83
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
84
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
85
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
86
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
87
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
88
|
+
SOFTWARE.
|
|
89
|
+
|
|
90
|
+
### Embedded libsamplerate (Secret Rabbit Code)
|
|
91
|
+
|
|
92
|
+
2-clause BSD License
|
|
93
|
+
|
|
94
|
+
Copyright (c) 2012-2016, Erik de Castro Lopo <erikd@mega-nerd.com>
|
|
95
|
+
All rights reserved.
|
|
96
|
+
|
|
97
|
+
Redistribution and use in source and binary forms, with or without
|
|
98
|
+
modification, are permitted provided that the following conditions are met:
|
|
99
|
+
|
|
100
|
+
1. Redistributions of source code must retain the above copyright notice,
|
|
101
|
+
this list of conditions and the following disclaimer.
|
|
102
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
103
|
+
this list of conditions and the following disclaimer in the documentation
|
|
104
|
+
and/or other materials provided with the distribution.
|
|
105
|
+
|
|
106
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
107
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
108
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
109
|
+
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
|
|
110
|
+
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
|
111
|
+
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
|
|
112
|
+
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
113
|
+
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
|
|
114
|
+
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
|
115
|
+
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
|
116
|
+
POSSIBILITY OF SUCH DAMAGE.
|
|
Binary file
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
import type { AecProcessor, AudioInputSource, AudioOutputSink, CameraSnapshotSource } from "./spi.js";
|
|
2
|
+
/** Options for the SDK-provided browser microphone input. */
|
|
3
|
+
export interface BrowserAudioInputOptions {
|
|
4
|
+
/**
|
|
5
|
+
* Maximum encoded chunks buffered before the current capture session fails.
|
|
6
|
+
* @defaultValue 128
|
|
7
|
+
* @remarks Must be a positive integer.
|
|
8
|
+
*/
|
|
9
|
+
maxBufferedChunks?: number;
|
|
10
|
+
/**
|
|
11
|
+
* Requests platform echo cancellation through `getUserMedia`.
|
|
12
|
+
* @defaultValue true
|
|
13
|
+
* @remarks The browser or device may ignore this constraint. Disable it when using active software AEC.
|
|
14
|
+
*/
|
|
15
|
+
echoCancellation?: boolean;
|
|
16
|
+
/**
|
|
17
|
+
* Requests platform noise suppression through `getUserMedia`.
|
|
18
|
+
* @defaultValue true
|
|
19
|
+
* @remarks The browser or device may ignore this constraint.
|
|
20
|
+
*/
|
|
21
|
+
noiseSuppression?: boolean;
|
|
22
|
+
/**
|
|
23
|
+
* Requests platform automatic gain control through `getUserMedia`.
|
|
24
|
+
* @defaultValue true
|
|
25
|
+
* @remarks The browser or device may ignore this constraint.
|
|
26
|
+
*/
|
|
27
|
+
autoGainControl?: boolean;
|
|
28
|
+
}
|
|
29
|
+
/**
|
|
30
|
+
* Creates the SDK-provided browser microphone implementation of `AudioInputSource`.
|
|
31
|
+
* The dialogue runtime owns `start()` / `frames()` / `stop()` after agent construction.
|
|
32
|
+
*/
|
|
33
|
+
export declare function createBrowserAudioInputSource(options?: BrowserAudioInputOptions): AudioInputSource;
|
|
34
|
+
/**
|
|
35
|
+
* Creates the SDK-provided browser playback implementation of `AudioOutputSink`.
|
|
36
|
+
* It initializes Web Audio lazily on the first `enqueue()` call.
|
|
37
|
+
*/
|
|
38
|
+
export declare function createBrowserAudioOutputSink(): AudioOutputSink;
|
|
39
|
+
/** Options for the SDK-provided browser still-camera source. */
|
|
40
|
+
export interface BrowserCameraSnapshotOptions {
|
|
41
|
+
/**
|
|
42
|
+
* Native video track constraints passed to `getUserMedia`.
|
|
43
|
+
* @defaultValue `true`
|
|
44
|
+
* @remarks The browser may adapt or ignore individual ideal constraints.
|
|
45
|
+
*/
|
|
46
|
+
video?: MediaTrackConstraints;
|
|
47
|
+
/**
|
|
48
|
+
* Requested still-image encoding.
|
|
49
|
+
* @defaultValue `image/png`
|
|
50
|
+
*/
|
|
51
|
+
mimeType?: "image/png" | "image/jpeg";
|
|
52
|
+
/**
|
|
53
|
+
* JPEG encoder quality from 0 through 1.
|
|
54
|
+
* @remarks Used only when `mimeType` is `image/jpeg`. Omit to use the browser default.
|
|
55
|
+
*/
|
|
56
|
+
jpegQuality?: number;
|
|
57
|
+
}
|
|
58
|
+
/**
|
|
59
|
+
* Creates the SDK-provided browser implementation of `CameraSnapshotSource`.
|
|
60
|
+
*
|
|
61
|
+
* @remarks
|
|
62
|
+
* The factory does not request permission. After composition, the runtime exclusively owns
|
|
63
|
+
* `start`/`capture`/`stop`: `start` requests one continuously held video-only session and
|
|
64
|
+
* resolves after a drawable frame is ready; `capture` encodes one frame from that same session;
|
|
65
|
+
* `stop` waits for pending acquisition/encoding and releases every track and element reference.
|
|
66
|
+
* The helper permits only one session and one in-flight capture.
|
|
67
|
+
*
|
|
68
|
+
* Abort rejects the outward `start`/`capture` promptly, but the helper retains cleanup ownership:
|
|
69
|
+
* a late MediaStream is stopped and a late encoded Blob is discarded. The runtime applies the
|
|
70
|
+
* fixed 1500ms cancellation settlement deadline; a non-settling operation or unconfirmed release
|
|
71
|
+
* faults the camera role and later camera controls reject deterministically. A successful `stop`
|
|
72
|
+
* permits a later sequential restart.
|
|
73
|
+
*
|
|
74
|
+
* @param options - Browser video constraints and still-image encoding preferences. Omit to use
|
|
75
|
+
* the default video device and PNG encoding at the captured frame's native dimensions.
|
|
76
|
+
* @returns A reusable `CameraSnapshotSource` to pass as `MediaTransportsConfig.camera`.
|
|
77
|
+
*/
|
|
78
|
+
export declare function createBrowserCameraSnapshotSource(options?: BrowserCameraSnapshotOptions): CameraSnapshotSource;
|
|
79
|
+
/**
|
|
80
|
+
* Creates the default no-op `AecProcessor` for platform-managed echo cancellation.
|
|
81
|
+
* Use an active custom AEC implementation instead when platform AEC is disabled.
|
|
82
|
+
*/
|
|
83
|
+
export declare function createPassthroughAecProcessor(): AecProcessor;
|
|
84
|
+
export {};
|