@huakunshen/dsh-plugin-voice 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +63 -0
- package/client.js +532 -0
- package/cordis.patch.yml +3 -0
- package/index.js +376 -0
- package/package.json +30 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Huakun Shen <huakun.shen@huakunshen.com>
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# @huakunshen/dsh-plugin-voice
|
|
2
|
+
|
|
3
|
+
Voice for the [DeepSeek Harness](https://github.com/kernel-machine/deepseek-harness) web UI: read assistant messages aloud and dictate into the composer with one mic click — powered by user-managed speech provider profiles.
|
|
4
|
+
|
|
5
|
+
在 DeepSeek Harness 网页版里朗读 assistant 回复、一键语音输入 —— 语音 provider 由你自己配置、随意组合。
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- **🎤 Dictation** — a mic control in the composer tool row: click to record, click again to stop, the transcript lands directly in the input (falls back to the clipboard when no editor is found). Audio is converted to 16 kHz mono WAV in the browser before upload.
|
|
10
|
+
- **🔊 Read aloud** — a speaker action on every finalized assistant message.
|
|
11
|
+
- **🧩 Provider registry** — add any number of provider profiles and mix them freely; dictation and read-aloud can use completely different providers:
|
|
12
|
+
|
|
13
|
+
| Kind | STT | TTS | Wire dialect |
|
|
14
|
+
|---|---|---|---|
|
|
15
|
+
| `mimo` (Xiaomi MiMo) | ✓ | ✓ | chat-completions `input_audio` / `message.audio` |
|
|
16
|
+
| `fish` (Fish Audio) | ✗ | ✓ | `POST /v1/tts`, true streaming |
|
|
17
|
+
| `openai` | ✓ | ✓ | `POST /audio/transcriptions` + `POST /audio/speech` (works with OpenAI, Groq, SiliconFlow, MiniMax-compatible gateways, …) |
|
|
18
|
+
|
|
19
|
+
- **⚙️ One settings section** — Settings → Voice: per-provider base URL / model / voice / language, API keys stored in the DSH credential store (an environment variable with the same name wins, matching DSH semantics), and the whole registry persists to `~/.dsh/voice.json`.
|
|
20
|
+
- **No repo changes** — installs as an out-of-tree profile plugin.
|
|
21
|
+
|
|
22
|
+
## Install
|
|
23
|
+
|
|
24
|
+
Requires a running DeepSeek Harness (web GUI) deployment.
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
dsh plugin install @huakunshen/dsh-plugin-voice
|
|
28
|
+
# or straight from this repository:
|
|
29
|
+
dsh plugin install https://github.com/HuakunShen/dsh-plugin-voice
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The Web sidebar's **Plugins** page and the agent's `install_bundle` tool accept the same URL. Restart DSH once so the new module generation loads — after that it auto-loads on every start.
|
|
33
|
+
|
|
34
|
+
<details>
|
|
35
|
+
<summary>Manual alternative</summary>
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
git clone https://github.com/HuakunShen/dsh-plugin-voice.git ~/Dev/dsh-plugin-voice
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Then in any DeepSeek Harness session, ask the agent:
|
|
42
|
+
|
|
43
|
+
> Install the bundle at ~/Dev/dsh-plugin-voice into my profile
|
|
44
|
+
|
|
45
|
+
</details>
|
|
46
|
+
|
|
47
|
+
## Configure
|
|
48
|
+
|
|
49
|
+
1. Open **Settings → Voice**.
|
|
50
|
+
2. The registry ships three seeded profiles: `mimo` (base URL points at the Token Plan host; switch to `https://api.xiaomimimo.com/v1` for the pay-as-you-go platform), `fish`, and `openai`.
|
|
51
|
+
3. Paste an API key per provider and save — it goes into the DSH local credential store under the profile's `keyRef` (seeded: `MIMO_API_KEY`, `FISH_API_KEY`, `VOICE_OPENAI_API_KEY`; a same-named environment variable takes precedence).
|
|
52
|
+
4. Pick the dictation provider and the read-aloud provider independently.
|
|
53
|
+
5. Add more providers (e.g. `groq` with `whisper-large-v3` for STT) from the bottom row; delete is one click, and the selects follow.
|
|
54
|
+
|
|
55
|
+
## Notes
|
|
56
|
+
|
|
57
|
+
- MiMo TTS is non-streaming (the model returns base64 audio that plays after synthesis); Fish Audio and OpenAI dialects stream while synthesizing.
|
|
58
|
+
- Dictation records via `MediaRecorder` and transcodes to WAV client-side, so any browser-supported recording format works regardless of what the provider accepts.
|
|
59
|
+
- Config lives in `~/.dsh/voice.json`; keys live in the DSH credential store (`~/.dsh/.credentials.yaml`). Deleting the JSON re-seeds the default registry.
|
|
60
|
+
|
|
61
|
+
## License
|
|
62
|
+
|
|
63
|
+
MIT
|
package/client.js
ADDED
|
@@ -0,0 +1,532 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Browser half of the Voice plugin:
|
|
3
|
+
* - a 🎤 control before the composer's submit action that records, transcribes
|
|
4
|
+
* through the dictation provider, and inserts the text into the composer;
|
|
5
|
+
* - a 🔊 action on finalized assistant messages that speaks the text through
|
|
6
|
+
* the read-aloud provider;
|
|
7
|
+
* - one "Voice" settings section managing the provider registry: add, edit,
|
|
8
|
+
* and remove profiles (MiMo / Fish Audio / OpenAI-compatible) and pick
|
|
9
|
+
* independent providers for dictation and read-aloud.
|
|
10
|
+
* @module dsh-plugin-voice/client
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
window.__ModuleLoader__.load({
|
|
14
|
+
id: 'dsh-plugin-voice/client',
|
|
15
|
+
factory(require) {
|
|
16
|
+
const React = require('react')
|
|
17
|
+
const h = React.createElement
|
|
18
|
+
|
|
19
|
+
const inputStyle = {
|
|
20
|
+
background: 'var(--dsh-color-bg, rgba(127,127,127,0.15))',
|
|
21
|
+
border: '1px solid rgba(127,127,127,0.35)',
|
|
22
|
+
borderRadius: 6,
|
|
23
|
+
padding: '4px 8px',
|
|
24
|
+
fontSize: 13,
|
|
25
|
+
color: 'inherit',
|
|
26
|
+
width: 260,
|
|
27
|
+
}
|
|
28
|
+
const rowStyle = { display: 'flex', alignItems: 'center', gap: 10, margin: '10px 0' }
|
|
29
|
+
const labelStyle = { width: 110, fontSize: 13, opacity: 0.8, flexShrink: 0 }
|
|
30
|
+
const hintStyle = { fontSize: 12, opacity: 0.55, margin: '2px 0 0 120px' }
|
|
31
|
+
const cardStyle = {
|
|
32
|
+
border: '1px solid rgba(127,127,127,0.25)',
|
|
33
|
+
borderRadius: 8,
|
|
34
|
+
padding: '8px 12px',
|
|
35
|
+
margin: '12px 0',
|
|
36
|
+
maxWidth: 620,
|
|
37
|
+
}
|
|
38
|
+
const smallBtn = {
|
|
39
|
+
background: 'rgba(127,127,127,0.2)',
|
|
40
|
+
border: '1px solid rgba(127,127,127,0.35)',
|
|
41
|
+
borderRadius: 6,
|
|
42
|
+
padding: '3px 10px',
|
|
43
|
+
fontSize: 12,
|
|
44
|
+
cursor: 'pointer',
|
|
45
|
+
color: 'inherit',
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
async function api(path, body) {
|
|
49
|
+
const response = await fetch(`/__dsh-voice${path}`, body === undefined
|
|
50
|
+
? undefined
|
|
51
|
+
: { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) })
|
|
52
|
+
const text = await response.text()
|
|
53
|
+
try {
|
|
54
|
+
return JSON.parse(text)
|
|
55
|
+
} catch {
|
|
56
|
+
throw new Error(`Voice 服务响应异常 (HTTP ${response.status}): ${text.slice(0, 120)}`)
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
// ---------------------------------------------------------------- icons
|
|
61
|
+
|
|
62
|
+
function MicIcon() {
|
|
63
|
+
return h('svg', { width: 16, height: 16, viewBox: '0 0 16 16', fill: 'none' },
|
|
64
|
+
h('rect', { x: 5.75, y: 1.5, width: 4.5, height: 8, rx: 2.25, fill: 'currentColor' }),
|
|
65
|
+
h('path', {
|
|
66
|
+
d: 'M3.5 7.5a4.5 4.5 0 0 0 9 0M8 12v2.5',
|
|
67
|
+
stroke: 'currentColor', strokeWidth: 1.3, strokeLinecap: 'round', fill: 'none',
|
|
68
|
+
}))
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
function StopSquareIcon() {
|
|
72
|
+
return h('svg', { width: 16, height: 16, viewBox: '0 0 16 16', fill: 'none' },
|
|
73
|
+
h('rect', { x: 3.5, y: 3.5, width: 9, height: 9, rx: 1.5, fill: 'currentColor' }))
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
function SpeakerIcon() {
|
|
77
|
+
return h('svg', { width: 16, height: 16, viewBox: '0 0 16 16', fill: 'none' },
|
|
78
|
+
h('path', { d: 'M2.2 5.9 H4.9 L8.4 2.9 V13.1 L4.9 10.1 H2.2 Z', fill: 'currentColor' }),
|
|
79
|
+
h('path', { d: 'M10.6 5.6 a3.1 3.1 0 0 1 0 4.8', stroke: 'currentColor', strokeWidth: 1.3, strokeLinecap: 'round', fill: 'none' }),
|
|
80
|
+
h('path', { d: 'M12.4 3.9 a5.6 5.6 0 0 1 0 8.2', stroke: 'currentColor', strokeWidth: 1.3, strokeLinecap: 'round', fill: 'none' }))
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
// ------------------------------------------------------------- audio utils
|
|
84
|
+
|
|
85
|
+
/** Decode any browser-recorded blob, downmix to mono 16 kHz, and encode 16-bit PCM WAV. */
|
|
86
|
+
async function blobToWavBase64(blob) {
|
|
87
|
+
const decoded = await new AudioContext().decodeAudioData(await blob.arrayBuffer())
|
|
88
|
+
const offline = new OfflineAudioContext(1, Math.ceil(decoded.duration * 16000) || 1, 16000)
|
|
89
|
+
const source = offline.createBufferSource()
|
|
90
|
+
source.buffer = decoded
|
|
91
|
+
source.connect(offline.destination)
|
|
92
|
+
source.start()
|
|
93
|
+
const rendered = await offline.startRendering()
|
|
94
|
+
const samples = rendered.getChannelData(0)
|
|
95
|
+
const bytes = new ArrayBuffer(44 + samples.length * 2)
|
|
96
|
+
const view = new DataView(bytes)
|
|
97
|
+
const writeString = (offset, value) => {
|
|
98
|
+
for (let i = 0; i < value.length; i++) view.setUint8(offset + i, value.charCodeAt(i))
|
|
99
|
+
}
|
|
100
|
+
writeString(0, 'RIFF')
|
|
101
|
+
view.setUint32(4, 36 + samples.length * 2, true)
|
|
102
|
+
writeString(8, 'WAVEfmt ')
|
|
103
|
+
view.setUint32(16, 16, true)
|
|
104
|
+
view.setUint16(20, 1, true)
|
|
105
|
+
view.setUint16(22, 1, true)
|
|
106
|
+
view.setUint32(24, 16000, true)
|
|
107
|
+
view.setUint32(28, 32000, true)
|
|
108
|
+
view.setUint16(32, 2, true)
|
|
109
|
+
view.setUint16(34, 16, true)
|
|
110
|
+
writeString(36, 'data')
|
|
111
|
+
view.setUint32(40, samples.length * 2, true)
|
|
112
|
+
for (let i = 0; i < samples.length; i++) {
|
|
113
|
+
const clamped = Math.max(-1, Math.min(1, samples[i]))
|
|
114
|
+
view.setInt16(44 + i * 2, clamped < 0 ? clamped * 0x8000 : clamped * 0x7fff, true)
|
|
115
|
+
}
|
|
116
|
+
let binary = ''
|
|
117
|
+
const chunk = 0x8000
|
|
118
|
+
const raw = new Uint8Array(bytes)
|
|
119
|
+
for (let i = 0; i < raw.length; i += chunk) {
|
|
120
|
+
binary += String.fromCharCode.apply(null, raw.subarray(i, i + chunk))
|
|
121
|
+
}
|
|
122
|
+
return btoa(binary)
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** Insert recognized text into the composer's Lexical contenteditable. */
|
|
126
|
+
function insertIntoComposer(text) {
|
|
127
|
+
const editors = Array.from(document.querySelectorAll('[data-lexical-editor="true"]'))
|
|
128
|
+
.filter((el) => el.isContentEditable && el.offsetParent !== null)
|
|
129
|
+
const target = editors[editors.length - 1]
|
|
130
|
+
if (target === undefined) return false
|
|
131
|
+
target.focus()
|
|
132
|
+
const selection = window.getSelection()
|
|
133
|
+
if (selection !== null) {
|
|
134
|
+
const range = document.createRange()
|
|
135
|
+
range.selectNodeContents(target)
|
|
136
|
+
range.collapse(false)
|
|
137
|
+
selection.removeAllRanges()
|
|
138
|
+
selection.addRange(range)
|
|
139
|
+
}
|
|
140
|
+
return document.execCommand('insertText', false, text)
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
// ------------------------------------------------------------ mic button
|
|
144
|
+
|
|
145
|
+
function MicButton() {
|
|
146
|
+
const statusState = React.useState('idle')
|
|
147
|
+
const status = statusState[0]
|
|
148
|
+
const setStatus = statusState[1]
|
|
149
|
+
const tipState = React.useState('')
|
|
150
|
+
const tip = tipState[0]
|
|
151
|
+
const setTip = tipState[1]
|
|
152
|
+
const recorderRef = React.useRef(null)
|
|
153
|
+
const chunksRef = React.useRef([])
|
|
154
|
+
const timerRef = React.useRef(null)
|
|
155
|
+
|
|
156
|
+
const showTip = function (message) {
|
|
157
|
+
setTip(message)
|
|
158
|
+
clearTimeout(timerRef.current)
|
|
159
|
+
timerRef.current = setTimeout(function () { setTip('') }, 4000)
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
const transcribe = async function (blob) {
|
|
163
|
+
setStatus('transcribing')
|
|
164
|
+
try {
|
|
165
|
+
const wavBase64 = await blobToWavBase64(blob)
|
|
166
|
+
const reply = await api('/asr', { wavBase64 })
|
|
167
|
+
if (reply.error) {
|
|
168
|
+
showTip(reply.error)
|
|
169
|
+
} else if (reply.text && insertIntoComposer(reply.text)) {
|
|
170
|
+
showTip('已插入识别文字')
|
|
171
|
+
} else {
|
|
172
|
+
await navigator.clipboard.writeText(reply.text || '').catch(function () {})
|
|
173
|
+
showTip('未找到输入框,文字已复制到剪贴板')
|
|
174
|
+
}
|
|
175
|
+
} catch (error) {
|
|
176
|
+
showTip(`识别失败:${String(error).slice(0, 160)}`)
|
|
177
|
+
}
|
|
178
|
+
setStatus('idle')
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
const start = async function () {
|
|
182
|
+
try {
|
|
183
|
+
const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
|
|
184
|
+
chunksRef.current = []
|
|
185
|
+
const recorder = new MediaRecorder(stream)
|
|
186
|
+
recorder.ondataavailable = function (event) {
|
|
187
|
+
if (event.data.size > 0) chunksRef.current.push(event.data)
|
|
188
|
+
}
|
|
189
|
+
recorder.onstop = function () {
|
|
190
|
+
stream.getTracks().forEach(function (track) { track.stop() })
|
|
191
|
+
void transcribe(new Blob(chunksRef.current, { type: recorder.mimeType || 'audio/webm' }))
|
|
192
|
+
}
|
|
193
|
+
recorder.start()
|
|
194
|
+
recorderRef.current = recorder
|
|
195
|
+
setStatus('recording')
|
|
196
|
+
} catch (error) {
|
|
197
|
+
showTip(`无法访问麦克风:${String(error).slice(0, 120)}`)
|
|
198
|
+
setStatus('idle')
|
|
199
|
+
}
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
const onClick = function () {
|
|
203
|
+
if (status === 'recording') {
|
|
204
|
+
recorderRef.current?.stop()
|
|
205
|
+
setStatus('transcribing')
|
|
206
|
+
} else if (status === 'idle') {
|
|
207
|
+
void start()
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
return h('span', { style: { display: 'inline-flex', alignItems: 'center', gap: 4 } },
|
|
212
|
+
h('button', {
|
|
213
|
+
title: tip || (status === 'recording' ? '停止录音' : '语音输入'),
|
|
214
|
+
onClick: onClick,
|
|
215
|
+
disabled: status === 'transcribing',
|
|
216
|
+
style: {
|
|
217
|
+
background: status === 'recording' ? 'rgba(229,83,75,0.18)' : 'none',
|
|
218
|
+
border: 'none', cursor: 'pointer', padding: '2px',
|
|
219
|
+
display: 'inline-flex', alignItems: 'center',
|
|
220
|
+
color: status === 'recording' ? '#e5534b' : 'inherit',
|
|
221
|
+
opacity: status === 'transcribing' ? 0.5 : 0.7,
|
|
222
|
+
},
|
|
223
|
+
}, status === 'recording' ? h(StopSquareIcon, null) : h(MicIcon, null)))
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
// -------------------------------------------------------- read-aloud button
|
|
227
|
+
|
|
228
|
+
function SpeakerButton(props) {
|
|
229
|
+
const state = React.useState('idle')
|
|
230
|
+
const status = state[0]
|
|
231
|
+
const setStatus = state[1]
|
|
232
|
+
const srcState = React.useState('')
|
|
233
|
+
const src = srcState[0]
|
|
234
|
+
const setSrc = srcState[1]
|
|
235
|
+
const errState = React.useState('')
|
|
236
|
+
const error = errState[0]
|
|
237
|
+
|
|
238
|
+
const stop = function () {
|
|
239
|
+
setSrc('')
|
|
240
|
+
setStatus('idle')
|
|
241
|
+
}
|
|
242
|
+
const onClick = function () {
|
|
243
|
+
if (status === 'playing') {
|
|
244
|
+
stop()
|
|
245
|
+
return
|
|
246
|
+
}
|
|
247
|
+
errState[1]('')
|
|
248
|
+
setSrc(`/__dsh-voice/tts?sessionId=${encodeURIComponent(props.sessionId)}`
|
|
249
|
+
+ `&messageId=${encodeURIComponent(props.messageId)}&t=${Date.now()}`)
|
|
250
|
+
setStatus('playing')
|
|
251
|
+
}
|
|
252
|
+
const onError = function () {
|
|
253
|
+
fetch(src).then(function (response) {
|
|
254
|
+
return response.text()
|
|
255
|
+
}).then(function (text) {
|
|
256
|
+
let message
|
|
257
|
+
try { message = JSON.parse(text).error } catch { message = text.slice(0, 200) }
|
|
258
|
+
errState[1](message || '播放失败')
|
|
259
|
+
setSrc('')
|
|
260
|
+
setStatus('error')
|
|
261
|
+
}, function () {
|
|
262
|
+
errState[1]('播放失败')
|
|
263
|
+
setSrc('')
|
|
264
|
+
setStatus('error')
|
|
265
|
+
})
|
|
266
|
+
}
|
|
267
|
+
|
|
268
|
+
const label = status === 'error' ? '⚠' : status === 'playing' ? h(StopSquareIcon, null) : h(SpeakerIcon, null)
|
|
269
|
+
return h('span', { style: { display: 'inline-flex', alignItems: 'center', gap: 4 } },
|
|
270
|
+
h('button', {
|
|
271
|
+
title: status === 'error' ? error : '朗读回复',
|
|
272
|
+
onClick: onClick,
|
|
273
|
+
style: {
|
|
274
|
+
background: 'none', border: 'none', cursor: 'pointer',
|
|
275
|
+
opacity: status === 'error' ? 1 : 0.7,
|
|
276
|
+
padding: '2px', display: 'inline-flex', alignItems: 'center',
|
|
277
|
+
color: status === 'error' ? '#e5534b' : 'inherit',
|
|
278
|
+
},
|
|
279
|
+
}, label),
|
|
280
|
+
status === 'playing' && src
|
|
281
|
+
? h('audio', { src: src, autoPlay: true, onEnded: stop, onError: onError, style: { display: 'none' } })
|
|
282
|
+
: null)
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
// ---------------------------------------------------------------- settings
|
|
286
|
+
|
|
287
|
+
const KIND_LABELS = { mimo: 'Xiaomi MiMo', fish: 'Fish Audio', openai: 'OpenAI 兼容 (/audio/*)' }
|
|
288
|
+
|
|
289
|
+
function ProviderCard(props) {
|
|
290
|
+
const id = props.id
|
|
291
|
+
const p = props.profile
|
|
292
|
+
const patch = props.patch
|
|
293
|
+
const remove = props.remove
|
|
294
|
+
const saveKey = props.saveKey
|
|
295
|
+
const keyDraft = React.useState('')
|
|
296
|
+
const saved = React.useState('')
|
|
297
|
+
const savedTimer = React.useRef(null)
|
|
298
|
+
const flash = function () {
|
|
299
|
+
saved[1]('✓ 已保存')
|
|
300
|
+
clearTimeout(savedTimer.current)
|
|
301
|
+
savedTimer.current = setTimeout(function () { saved[1]('') }, 2500)
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
const stt = p.capabilities?.stt
|
|
305
|
+
const tts = p.capabilities?.tts
|
|
306
|
+
return h('div', { style: cardStyle },
|
|
307
|
+
h('div', { style: Object.assign({ gap: 8 }, rowStyle) },
|
|
308
|
+
h('input', {
|
|
309
|
+
value: id,
|
|
310
|
+
title: 'provider 名称',
|
|
311
|
+
onChange: function () {}, // ids are immutable; rename = add + delete
|
|
312
|
+
style: Object.assign({ width: 120, fontWeight: 600, background: 'transparent', border: 'none', color: 'inherit' }, inputStyle),
|
|
313
|
+
}),
|
|
314
|
+
h('select', { value: p.kind, onChange: function (e) { patch(id, { kind: e.target.value }) }, style: Object.assign({ width: 170 }, inputStyle) },
|
|
315
|
+
Object.entries(KIND_LABELS).map(function (entry) {
|
|
316
|
+
return h('option', { value: entry[0] }, entry[1])
|
|
317
|
+
})),
|
|
318
|
+
h('span', { style: { fontSize: 11, opacity: 0.6 } },
|
|
319
|
+
`${stt ? '识别 ✓' : '识别 ✗'} · ${tts ? '朗读 ✓' : '朗读 ✗'}`),
|
|
320
|
+
h('button', { onClick: function () { remove(id) }, style: smallBtn, title: '删除这个 provider' }, '删除')),
|
|
321
|
+
|
|
322
|
+
p.needsBaseUrl !== false && p.kind !== 'fish'
|
|
323
|
+
? h('div', { style: rowStyle },
|
|
324
|
+
h('span', { style: labelStyle }, 'Base URL'),
|
|
325
|
+
h('input', { value: p.baseUrl ?? '', onChange: function (e) { patch(id, { baseUrl: e.target.value }) }, style: inputStyle }))
|
|
326
|
+
: null,
|
|
327
|
+
|
|
328
|
+
h('div', { style: rowStyle },
|
|
329
|
+
h('span', { style: labelStyle }, 'API Key'),
|
|
330
|
+
h('input', {
|
|
331
|
+
type: 'password',
|
|
332
|
+
value: keyDraft[0],
|
|
333
|
+
placeholder: p.keyConfigured ? `已配置(${p.keySource === 'env' ? '环境变量' : p.keySource === 'file' ? '本地存储' : p.keySource ?? '未知来源'}),输入可覆盖` : `凭证名: ${p.keyRef}`,
|
|
334
|
+
onChange: function (e) { keyDraft[1](e.target.value) },
|
|
335
|
+
style: inputStyle,
|
|
336
|
+
}),
|
|
337
|
+
h('button', {
|
|
338
|
+
onClick: function () {
|
|
339
|
+
const value = keyDraft[0].trim()
|
|
340
|
+
if (!value) return
|
|
341
|
+
void api('/key', { profile: id, value }).then(function () {
|
|
342
|
+
keyDraft[1]('')
|
|
343
|
+
flash()
|
|
344
|
+
return props.reload()
|
|
345
|
+
})
|
|
346
|
+
},
|
|
347
|
+
style: smallBtn,
|
|
348
|
+
}, '保存'),
|
|
349
|
+
h('span', { style: { fontSize: 12, opacity: 0.7 } }, saved[0])),
|
|
350
|
+
|
|
351
|
+
stt
|
|
352
|
+
? h('div', null,
|
|
353
|
+
h('div', { style: rowStyle },
|
|
354
|
+
h('span', { style: labelStyle }, '识别模型'),
|
|
355
|
+
h('input', { value: p.asrModel ?? '', onChange: function (e) { patch(id, { asrModel: e.target.value }) }, style: inputStyle })),
|
|
356
|
+
h('div', { style: rowStyle },
|
|
357
|
+
h('span', { style: labelStyle }, '识别语言'),
|
|
358
|
+
h('select', { value: p.asrLanguage ?? 'auto', onChange: function (e) { patch(id, { asrLanguage: e.target.value }) }, style: inputStyle },
|
|
359
|
+
h('option', { value: 'auto' }, '自动检测'),
|
|
360
|
+
h('option', { value: 'zh' }, '中文'),
|
|
361
|
+
h('option', { value: 'en' }, '英文'))))
|
|
362
|
+
: null,
|
|
363
|
+
|
|
364
|
+
tts
|
|
365
|
+
? h('div', null,
|
|
366
|
+
h('div', { style: rowStyle },
|
|
367
|
+
h('span', { style: labelStyle }, 'TTS 模型'),
|
|
368
|
+
h('input', { value: p.ttsModel ?? '', onChange: function (e) { patch(id, { ttsModel: e.target.value }) }, style: inputStyle })),
|
|
369
|
+
h('div', { style: rowStyle },
|
|
370
|
+
h('span', { style: labelStyle }, p.kind === 'fish' ? 'reference_id' : '音色'),
|
|
371
|
+
h('input', {
|
|
372
|
+
value: p.ttsVoice ?? '',
|
|
373
|
+
placeholder: p.kind === 'fish' ? '留空 = 默认音色' : 'mimo_default / 冰糖 / Mia / alloy …',
|
|
374
|
+
onChange: function (e) { patch(id, { ttsVoice: e.target.value }) },
|
|
375
|
+
style: inputStyle,
|
|
376
|
+
})),
|
|
377
|
+
p.kind === 'fish'
|
|
378
|
+
? h('div', { style: rowStyle },
|
|
379
|
+
h('span', { style: labelStyle }, '语速'),
|
|
380
|
+
h('input', {
|
|
381
|
+
type: 'number', min: 0.5, max: 2, step: 0.05,
|
|
382
|
+
value: String(p.speed ?? 1),
|
|
383
|
+
onChange: function (e) {
|
|
384
|
+
const v = parseFloat(e.target.value)
|
|
385
|
+
if (!isNaN(v) && v >= 0.5 && v <= 2) patch(id, { speed: v })
|
|
386
|
+
},
|
|
387
|
+
style: Object.assign({ width: 90 }, inputStyle),
|
|
388
|
+
}))
|
|
389
|
+
: null)
|
|
390
|
+
: null)
|
|
391
|
+
}
|
|
392
|
+
|
|
393
|
+
function VoiceSection() {
|
|
394
|
+
const cfgState = React.useState(null)
|
|
395
|
+
const cfg = cfgState[0]
|
|
396
|
+
const setCfg = cfgState[1]
|
|
397
|
+
const loadError = React.useState('')
|
|
398
|
+
const saved = React.useState('')
|
|
399
|
+
const savedTimer = React.useRef(null)
|
|
400
|
+
const newId = React.useState('')
|
|
401
|
+
|
|
402
|
+
const reload = React.useCallback(function () {
|
|
403
|
+
return api('/config').then(function (nextConfig) {
|
|
404
|
+
setCfg(nextConfig)
|
|
405
|
+
return nextConfig
|
|
406
|
+
})
|
|
407
|
+
}, [])
|
|
408
|
+
|
|
409
|
+
React.useEffect(function () {
|
|
410
|
+
void reload().then(function (nextConfig) {
|
|
411
|
+
if (!nextConfig || typeof nextConfig.providers !== 'object' || nextConfig.providers === null) {
|
|
412
|
+
loadError[1]('Voice host 应答无法识别——dsh host 可能还没重启到 voice 0.2(当前应答:'
|
|
413
|
+
+ (JSON.stringify(nextConfig ?? null) ?? '空').slice(0, 120) + ')')
|
|
414
|
+
return
|
|
415
|
+
}
|
|
416
|
+
setCfg(nextConfig)
|
|
417
|
+
}, function (error) { loadError[1](String(error)) })
|
|
418
|
+
}, [reload])
|
|
419
|
+
|
|
420
|
+
const flashSaved = function () {
|
|
421
|
+
saved[1]('✓ 已保存')
|
|
422
|
+
clearTimeout(savedTimer.current)
|
|
423
|
+
savedTimer.current = setTimeout(function () { saved[1]('') }, 2000)
|
|
424
|
+
}
|
|
425
|
+
|
|
426
|
+
const persist = function (next) {
|
|
427
|
+
setCfg(next)
|
|
428
|
+
return api('/config', { providers: next.providers, dictation: next.dictation, readAloud: next.readAloud }).then(flashSaved)
|
|
429
|
+
}
|
|
430
|
+
|
|
431
|
+
const patchProvider = function (id, patchValue) {
|
|
432
|
+
if (!cfg) return
|
|
433
|
+
const providers = Object.assign({}, cfg.providers)
|
|
434
|
+
providers[id] = Object.assign({ kind: providers[id].kind }, providers[id], patchValue)
|
|
435
|
+
void persist(Object.assign({}, cfg, { providers }))
|
|
436
|
+
}
|
|
437
|
+
const removeProvider = function (id) {
|
|
438
|
+
if (!cfg || Object.keys(cfg.providers).length <= 1) return
|
|
439
|
+
const providers = Object.assign({}, cfg.providers)
|
|
440
|
+
delete providers[id]
|
|
441
|
+
const next = Object.assign({}, cfg, { providers })
|
|
442
|
+
if (next.dictation === id) next.dictation = Object.keys(providers)[0]
|
|
443
|
+
if (next.readAloud === id) next.readAloud = Object.keys(providers)[0]
|
|
444
|
+
void persist(next)
|
|
445
|
+
}
|
|
446
|
+
const addProvider = function () {
|
|
447
|
+
if (!cfg || typeof cfg.providers !== 'object' || cfg.providers === null) {
|
|
448
|
+
loadError[1]('配置尚未加载完成,无法添加 provider。')
|
|
449
|
+
return
|
|
450
|
+
}
|
|
451
|
+
const existing = cfg.providers
|
|
452
|
+
let id = newId[0].trim().toLowerCase().replace(/[^a-z0-9_-]/g, '-') || `provider-${Object.keys(existing).length + 1}`
|
|
453
|
+
if (existing[id] !== undefined) id = `${id}-${Object.keys(existing).length + 1}`
|
|
454
|
+
const providers = Object.assign({}, existing)
|
|
455
|
+
providers[id] = { kind: 'openai', baseUrl: 'https://api.openai.com/v1', keyRef: `VOICE_${id.toUpperCase().replace(/-/g, '_')}_API_KEY`, asrModel: '', asrLanguage: 'auto', ttsModel: '', ttsVoice: '' }
|
|
456
|
+
newId[1]('')
|
|
457
|
+
void persist(Object.assign({}, cfg, {
|
|
458
|
+
providers,
|
|
459
|
+
dictation: cfg.dictation in providers ? cfg.dictation : id,
|
|
460
|
+
readAloud: cfg.readAloud in providers ? cfg.readAloud : id,
|
|
461
|
+
}))
|
|
462
|
+
}
|
|
463
|
+
|
|
464
|
+
if (loadError[0]) {
|
|
465
|
+
return h('div', { style: { padding: 12, color: '#e5534b', fontSize: 13, maxWidth: 560 } },
|
|
466
|
+
'Voice 加载失败:', loadError[0])
|
|
467
|
+
}
|
|
468
|
+
if (!cfg) return null
|
|
469
|
+
|
|
470
|
+
const providerEntries = Object.entries(cfg.providers ?? {})
|
|
471
|
+
const sttChoices = providerEntries.filter(function (entry) { return entry[1].capabilities?.stt })
|
|
472
|
+
const ttsChoices = providerEntries.filter(function (entry) { return entry[1].capabilities?.tts })
|
|
473
|
+
|
|
474
|
+
return h('div', { style: { padding: '8px 4px', maxWidth: 660 } },
|
|
475
|
+
h('h3', { style: { margin: '4px 0 12px', fontSize: 15 } }, 'Voice(语音输入 + 朗读)'),
|
|
476
|
+
|
|
477
|
+
h('div', { style: rowStyle },
|
|
478
|
+
h('span', { style: labelStyle }, '语音输入用'),
|
|
479
|
+
h('select', { value: cfg.dictation, onChange: function (e) { void persist(Object.assign({}, cfg, { dictation: e.target.value })) }, style: inputStyle },
|
|
480
|
+
sttChoices.map(function (entry) { return h('option', { value: entry[0] }, entry[0]) }))),
|
|
481
|
+
h('div', { style: rowStyle },
|
|
482
|
+
h('span', { style: labelStyle }, '朗读用'),
|
|
483
|
+
h('select', { value: cfg.readAloud, onChange: function (e) { void persist(Object.assign({}, cfg, { readAloud: e.target.value })) }, style: inputStyle },
|
|
484
|
+
ttsChoices.map(function (entry) { return h('option', { value: entry[0] }, entry[0]) }))),
|
|
485
|
+
h('div', { style: hintStyle }, '两处可以选完全不同的 provider。'),
|
|
486
|
+
h('div', { style: { fontSize: 12, opacity: 0.7, margin: '0 0 4px 120px' } }, saved[0]),
|
|
487
|
+
|
|
488
|
+
providerEntries.map(function (entry) {
|
|
489
|
+
return h(ProviderCard, {
|
|
490
|
+
key: entry[0],
|
|
491
|
+
id: entry[0],
|
|
492
|
+
profile: entry[1],
|
|
493
|
+
patch: patchProvider,
|
|
494
|
+
remove: removeProvider,
|
|
495
|
+
saveKey: api,
|
|
496
|
+
reload: reload,
|
|
497
|
+
})
|
|
498
|
+
}),
|
|
499
|
+
|
|
500
|
+
h('div', { style: rowStyle },
|
|
501
|
+
h('span', { style: labelStyle }, '新增 provider'),
|
|
502
|
+
h('input', {
|
|
503
|
+
value: newId[0],
|
|
504
|
+
placeholder: '名称(如 minimax、groq)',
|
|
505
|
+
onChange: function (e) { newId[1](e.target.value) },
|
|
506
|
+
style: Object.assign({ width: 160 }, inputStyle),
|
|
507
|
+
}),
|
|
508
|
+
h('button', { onClick: addProvider, style: smallBtn }, '添加')),
|
|
509
|
+
h('div', { style: hintStyle }, '配置持久化在 ~/.dsh/voice.json;key 存 dsh 凭证库,环境变量同名优先。'),
|
|
510
|
+
)
|
|
511
|
+
}
|
|
512
|
+
|
|
513
|
+
return {
|
|
514
|
+
apply(ctx) {
|
|
515
|
+
const slots = ctx.get('slots')
|
|
516
|
+
if (slots === undefined) return
|
|
517
|
+
slots.inject('conversation.input.right', () => slots.register(
|
|
518
|
+
{ name: 'conversation.input.right', id: 'voice-mic', order: 20 },
|
|
519
|
+
MicButton,
|
|
520
|
+
))
|
|
521
|
+
slots.inject('conversation.chat.assistant-actions', () => slots.register(
|
|
522
|
+
{ name: 'conversation.chat.assistant-actions', id: 'voice-read-aloud', order: 90 },
|
|
523
|
+
SpeakerButton,
|
|
524
|
+
))
|
|
525
|
+
slots.inject('settings.section', () => slots.register(
|
|
526
|
+
{ name: 'settings.section', id: 'voice', order: 120, label: 'Voice' },
|
|
527
|
+
VoiceSection,
|
|
528
|
+
))
|
|
529
|
+
},
|
|
530
|
+
}
|
|
531
|
+
},
|
|
532
|
+
})
|
package/cordis.patch.yml
ADDED
package/index.js
ADDED
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Host half of the Voice plugin. Serves same-origin `/__dsh-voice` routes the
|
|
3
|
+
* browser half fetches. Configuration is a user-managed provider registry
|
|
4
|
+
* persisted to `~/.dsh/voice.json`: each profile names a wire dialect
|
|
5
|
+
* (`mimo` | `fish` | `openai`), its endpoint, model, and credential reference.
|
|
6
|
+
* Read-aloud TTS and composer dictation STT each select any profile whose
|
|
7
|
+
* dialect supports the capability.
|
|
8
|
+
* @module dsh-plugin-voice
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
import { readFileSync, writeFileSync } from 'node:fs'
|
|
12
|
+
import { join } from 'node:path'
|
|
13
|
+
|
|
14
|
+
export const inject = ['credentials', 'webServer']
|
|
15
|
+
|
|
16
|
+
const CONFIG_PATH = join(
|
|
17
|
+
process.env.DSH_HOME?.replace(/^~/, process.env.HOME ?? '') ?? join(process.env.HOME ?? '.', '.dsh'),
|
|
18
|
+
'voice.json',
|
|
19
|
+
)
|
|
20
|
+
const MAX_TTS_CHARS = 8000
|
|
21
|
+
const MAX_ASR_BYTES = 25 * 1024 * 1024
|
|
22
|
+
|
|
23
|
+
/** Wire dialects and the capabilities each can serve. */
|
|
24
|
+
export const KINDS = {
|
|
25
|
+
mimo: { stt: true, tts: true, needsBaseUrl: true },
|
|
26
|
+
fish: { stt: false, tts: true, needsBaseUrl: false },
|
|
27
|
+
openai: { stt: true, tts: true, needsBaseUrl: true },
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function seededConfig() {
|
|
31
|
+
return {
|
|
32
|
+
providers: {
|
|
33
|
+
mimo: {
|
|
34
|
+
kind: 'mimo',
|
|
35
|
+
baseUrl: 'https://token-plan-cn.xiaomimimo.com/v1',
|
|
36
|
+
keyRef: 'MIMO_API_KEY',
|
|
37
|
+
asrModel: 'mimo-v2.5-asr',
|
|
38
|
+
asrLanguage: 'auto',
|
|
39
|
+
ttsModel: 'mimo-v2.5-tts',
|
|
40
|
+
ttsVoice: 'mimo_default',
|
|
41
|
+
},
|
|
42
|
+
fish: {
|
|
43
|
+
kind: 'fish',
|
|
44
|
+
keyRef: 'FISH_API_KEY',
|
|
45
|
+
ttsModel: 's2.1-pro-free',
|
|
46
|
+
ttsVoice: '',
|
|
47
|
+
speed: 1,
|
|
48
|
+
},
|
|
49
|
+
openai: {
|
|
50
|
+
kind: 'openai',
|
|
51
|
+
baseUrl: 'https://api.openai.com/v1',
|
|
52
|
+
keyRef: 'OPENAI_API_KEY',
|
|
53
|
+
asrModel: 'gpt-4o-mini-transcribe',
|
|
54
|
+
asrLanguage: 'auto',
|
|
55
|
+
ttsModel: 'gpt-4o-mini-tts',
|
|
56
|
+
ttsVoice: 'alloy',
|
|
57
|
+
},
|
|
58
|
+
},
|
|
59
|
+
dictation: 'mimo',
|
|
60
|
+
readAloud: 'mimo',
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** In-memory config; every mutation persists through saveConfig(). */
|
|
65
|
+
let config = seededConfig()
|
|
66
|
+
|
|
67
|
+
function loadConfig() {
|
|
68
|
+
try {
|
|
69
|
+
const parsed = JSON.parse(readFileSync(CONFIG_PATH, 'utf8'))
|
|
70
|
+
if (parsed && typeof parsed === 'object' && parsed.providers && typeof parsed.providers === 'object') {
|
|
71
|
+
config = { ...seededConfig(), ...parsed }
|
|
72
|
+
}
|
|
73
|
+
} catch {
|
|
74
|
+
// Absent or unreadable file: keep the seeded registry; the first save writes it.
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
function saveConfig() {
|
|
79
|
+
try {
|
|
80
|
+
writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2) + '\n')
|
|
81
|
+
} catch {
|
|
82
|
+
// Persisted config is an optimization; the runtime registry stays usable.
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
function sanitizeProfileId(value) {
|
|
87
|
+
return String(value).toLowerCase().replace(/[^a-z0-9_-]/g, '-').replace(/^-+|-+$/g, '').slice(0, 32)
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
async function readBody(req) {
|
|
91
|
+
const chunks = []
|
|
92
|
+
for await (const chunk of req) chunks.push(chunk)
|
|
93
|
+
try {
|
|
94
|
+
return JSON.parse(Buffer.concat(chunks).toString('utf8'))
|
|
95
|
+
} catch {
|
|
96
|
+
return undefined
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
function json(res, status, value) {
|
|
101
|
+
res.statusCode = status
|
|
102
|
+
res.setHeader('Content-Type', 'application/json; charset=utf-8')
|
|
103
|
+
res.end(JSON.stringify(value))
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
async function resolveKey(credentials, ref) {
|
|
107
|
+
try {
|
|
108
|
+
return await credentials.resolve(ref)
|
|
109
|
+
} catch {
|
|
110
|
+
return undefined
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/** Normalize a MiMo/OpenAI-style base URL and join one path. */
|
|
115
|
+
function endpoint(baseUrl, path) {
|
|
116
|
+
return `${String(baseUrl ?? '').trim().replace(/\/+$/, '')}${path}`
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
/**
|
|
120
|
+
* One chat-completions call carrying either an input_audio part (MiMo ASR) or
|
|
121
|
+
* an audio output request (MiMo TTS). Answer fields: audio (base64) or text.
|
|
122
|
+
*/
|
|
123
|
+
async function mimoChat(profile, key, body, timeoutMs) {
|
|
124
|
+
const response = await fetch(endpoint(profile.baseUrl, '/chat/completions'), {
|
|
125
|
+
method: 'POST',
|
|
126
|
+
headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json' },
|
|
127
|
+
body: JSON.stringify(body),
|
|
128
|
+
signal: AbortSignal.timeout(timeoutMs),
|
|
129
|
+
})
|
|
130
|
+
const text = await response.text()
|
|
131
|
+
let parsed
|
|
132
|
+
try {
|
|
133
|
+
parsed = JSON.parse(text)
|
|
134
|
+
} catch {
|
|
135
|
+
parsed = { raw: text.slice(0, 300) }
|
|
136
|
+
}
|
|
137
|
+
if (!response.ok) {
|
|
138
|
+
return { error: `请求失败(HTTP ${response.status})。${JSON.stringify(parsed).slice(0, 200)}` }
|
|
139
|
+
}
|
|
140
|
+
return { parsed }
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
async function synthesizeMimo(profile, key, text) {
|
|
144
|
+
const audio = { format: 'mp3' }
|
|
145
|
+
if (profile.ttsVoice) audio.voice = profile.ttsVoice
|
|
146
|
+
const reply = await mimoChat(profile, key, {
|
|
147
|
+
model: profile.ttsModel || 'mimo-v2.5-tts',
|
|
148
|
+
messages: [{ role: 'assistant', content: text }],
|
|
149
|
+
audio,
|
|
150
|
+
}, 300_000)
|
|
151
|
+
if (reply.error !== undefined) return { error: `MiMo TTS ${reply.error}` }
|
|
152
|
+
const data = reply.parsed?.choices?.[0]?.message?.audio?.data
|
|
153
|
+
if (typeof data !== 'string' || !data) return { error: 'MiMo TTS 响应中没有音频数据。' }
|
|
154
|
+
return { buffer: Buffer.from(data, 'base64') }
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
async function synthesizeFish(profile, key, text) {
|
|
158
|
+
const body = { text, format: 'mp3', chunk_length: 100 }
|
|
159
|
+
if (profile.ttsVoice) body.reference_id = profile.ttsVoice
|
|
160
|
+
if (profile.speed && profile.speed !== 1) body.prosody = { speed: profile.speed }
|
|
161
|
+
const response = await fetch('https://api.fish.audio/v1/tts', {
|
|
162
|
+
method: 'POST',
|
|
163
|
+
headers: {
|
|
164
|
+
Authorization: `Bearer ${key}`,
|
|
165
|
+
'Content-Type': 'application/json',
|
|
166
|
+
model: profile.ttsModel || 's2.1-pro-free',
|
|
167
|
+
},
|
|
168
|
+
body: JSON.stringify(body),
|
|
169
|
+
signal: AbortSignal.timeout(300_000),
|
|
170
|
+
})
|
|
171
|
+
if (!response.ok || response.body === null) {
|
|
172
|
+
const detail = (await response.text().catch(() => '')).slice(0, 200)
|
|
173
|
+
return { error: `Fish Audio TTS 请求失败(HTTP ${response.status})。${detail}` }
|
|
174
|
+
}
|
|
175
|
+
return { stream: response.body }
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
/** OpenAI /audio/speech: binary audio body streamed straight through. */
|
|
179
|
+
async function synthesizeOpenai(profile, key, text) {
|
|
180
|
+
const response = await fetch(endpoint(profile.baseUrl, '/audio/speech'), {
|
|
181
|
+
method: 'POST',
|
|
182
|
+
headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json' },
|
|
183
|
+
body: JSON.stringify({ model: profile.ttsModel || 'gpt-4o-mini-tts', voice: profile.ttsVoice || 'alloy', input: text, response_format: 'mp3' }),
|
|
184
|
+
signal: AbortSignal.timeout(300_000),
|
|
185
|
+
})
|
|
186
|
+
if (!response.ok || response.body === null) {
|
|
187
|
+
const detail = (await response.text().catch(() => '')).slice(0, 200)
|
|
188
|
+
return { error: `TTS 请求失败(HTTP ${response.status})。${detail}` }
|
|
189
|
+
}
|
|
190
|
+
return { stream: response.body }
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
async function synthesize(profile, key, text) {
|
|
194
|
+
if (profile.kind === 'mimo') return synthesizeMimo(profile, key, text)
|
|
195
|
+
if (profile.kind === 'fish') return synthesizeFish(profile, key, text)
|
|
196
|
+
return synthesizeOpenai(profile, key, text)
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
async function transcribeMimo(profile, key, wavBase64) {
|
|
200
|
+
const body = {
|
|
201
|
+
model: profile.asrModel || 'mimo-v2.5-asr',
|
|
202
|
+
messages: [{
|
|
203
|
+
role: 'user',
|
|
204
|
+
content: [{ type: 'input_audio', input_audio: { data: `data:audio/wav;base64,${wavBase64}` } }],
|
|
205
|
+
}],
|
|
206
|
+
}
|
|
207
|
+
if (profile.asrLanguage && profile.asrLanguage !== 'auto') body.asr_options = { language: profile.asrLanguage }
|
|
208
|
+
const reply = await mimoChat(profile, key, body, 120_000)
|
|
209
|
+
if (reply.error !== undefined) return { error: `语音识别 ${reply.error}` }
|
|
210
|
+
const content = reply.parsed?.choices?.[0]?.message?.content
|
|
211
|
+
if (typeof content !== 'string') return { error: '语音识别响应中没有文字。' }
|
|
212
|
+
return { text: content }
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
async function transcribeOpenai(profile, key, wavBase64) {
|
|
216
|
+
const bytes = Buffer.from(wavBase64, 'base64')
|
|
217
|
+
const form = new FormData()
|
|
218
|
+
form.append('file', new Blob([bytes], { type: 'audio/wav' }), 'audio.wav')
|
|
219
|
+
form.append('model', profile.asrModel || 'gpt-4o-mini-transcribe')
|
|
220
|
+
if (profile.asrLanguage && profile.asrLanguage !== 'auto') form.append('language', profile.asrLanguage)
|
|
221
|
+
const response = await fetch(endpoint(profile.baseUrl, '/audio/transcriptions'), {
|
|
222
|
+
method: 'POST',
|
|
223
|
+
headers: { Authorization: `Bearer ${key}` },
|
|
224
|
+
body: form,
|
|
225
|
+
signal: AbortSignal.timeout(120_000),
|
|
226
|
+
})
|
|
227
|
+
const text = await response.text()
|
|
228
|
+
let parsed
|
|
229
|
+
try {
|
|
230
|
+
parsed = JSON.parse(text)
|
|
231
|
+
} catch {
|
|
232
|
+
parsed = { raw: text.slice(0, 300) }
|
|
233
|
+
}
|
|
234
|
+
if (!response.ok) return { error: `语音识别请求失败(HTTP ${response.status})。${JSON.stringify(parsed).slice(0, 200)}` }
|
|
235
|
+
if (typeof parsed.text !== 'string') return { error: '语音识别响应中没有文字。' }
|
|
236
|
+
return { text: parsed.text }
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
async function transcribe(profile, key, wavBase64) {
|
|
240
|
+
if (profile.kind === 'mimo') return transcribeMimo(profile, key, wavBase64)
|
|
241
|
+
return transcribeOpenai(profile, key, wavBase64)
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
export function apply(ctx) {
|
|
245
|
+
const credentials = ctx.get('credentials')
|
|
246
|
+
const webServer = ctx.get('webServer')
|
|
247
|
+
if (credentials === undefined || webServer === undefined) return
|
|
248
|
+
loadConfig()
|
|
249
|
+
|
|
250
|
+
const dispose = webServer.register({
|
|
251
|
+
kind: 'prefix',
|
|
252
|
+
path: '/__dsh-voice',
|
|
253
|
+
async handler(req, res) {
|
|
254
|
+
const url = new URL(req.url ?? '/', 'http://localhost')
|
|
255
|
+
const path = url.pathname
|
|
256
|
+
|
|
257
|
+
if (req.method === 'GET' && path === '/__dsh-voice/config') {
|
|
258
|
+
if (Object.keys(config.providers).length === 0) {
|
|
259
|
+
config = seededConfig()
|
|
260
|
+
saveConfig()
|
|
261
|
+
}
|
|
262
|
+
const providers = {}
|
|
263
|
+
for (const [id, profile] of Object.entries(config.providers)) {
|
|
264
|
+
const resolved = await resolveKey(credentials, profile.keyRef)
|
|
265
|
+
providers[id] = {
|
|
266
|
+
...profile,
|
|
267
|
+
keyConfigured: resolved !== undefined,
|
|
268
|
+
keySource: resolved?.source,
|
|
269
|
+
capabilities: KINDS[profile.kind] ?? { stt: false, tts: false },
|
|
270
|
+
}
|
|
271
|
+
}
|
|
272
|
+
return json(res, 200, {
|
|
273
|
+
providers,
|
|
274
|
+
dictation: config.dictation,
|
|
275
|
+
readAloud: config.readAloud,
|
|
276
|
+
kinds: KINDS,
|
|
277
|
+
})
|
|
278
|
+
}
|
|
279
|
+
|
|
280
|
+
if (req.method === 'POST' && path === '/__dsh-voice/config') {
|
|
281
|
+
const body = await readBody(req)
|
|
282
|
+
if (body === undefined) return json(res, 400, { error: 'invalid body' })
|
|
283
|
+
const next = { providers: {}, dictation: config.dictation, readAloud: config.readAloud }
|
|
284
|
+
for (const [id, raw] of Object.entries(body.providers ?? {})) {
|
|
285
|
+
const profileId = sanitizeProfileId(id)
|
|
286
|
+
if (!profileId || typeof raw !== 'object') continue
|
|
287
|
+
const kind = KINDS[raw.kind] !== undefined ? raw.kind : 'openai'
|
|
288
|
+
const profile = {
|
|
289
|
+
kind,
|
|
290
|
+
keyRef: typeof raw.keyRef === 'string' && raw.keyRef.trim() ? raw.keyRef.trim() : `VOICE_${profileId.toUpperCase().replace(/-/g, '_')}_API_KEY`,
|
|
291
|
+
}
|
|
292
|
+
if (KINDS[kind].needsBaseUrl && typeof raw.baseUrl === 'string' && /^https?:\/\//.test(raw.baseUrl)) profile.baseUrl = raw.baseUrl.trim()
|
|
293
|
+
if (typeof raw.asrModel === 'string' && raw.asrModel.trim()) profile.asrModel = raw.asrModel.trim()
|
|
294
|
+
if (raw.asrLanguage === 'auto' || raw.asrLanguage === 'zh' || raw.asrLanguage === 'en') profile.asrLanguage = raw.asrLanguage
|
|
295
|
+
if (typeof raw.ttsModel === 'string' && raw.ttsModel.trim()) profile.ttsModel = raw.ttsModel.trim()
|
|
296
|
+
if (typeof raw.ttsVoice === 'string') profile.ttsVoice = raw.ttsVoice.trim()
|
|
297
|
+
if (typeof raw.speed === 'number' && raw.speed >= 0.5 && raw.speed <= 2) profile.speed = raw.speed
|
|
298
|
+
next.providers[profileId] = profile
|
|
299
|
+
}
|
|
300
|
+
if (Object.keys(next.providers).length === 0) return json(res, 400, { error: '至少保留一个 provider。' })
|
|
301
|
+
if (typeof body.dictation === 'string' && next.providers[body.dictation]?.kind !== undefined) next.dictation = body.dictation
|
|
302
|
+
if (typeof body.readAloud === 'string' && next.providers[body.readAloud]?.kind !== undefined) next.readAloud = body.readAloud
|
|
303
|
+
if (next.providers[next.dictation] === undefined) next.dictation = Object.keys(next.providers)[0]
|
|
304
|
+
if (next.providers[next.readAloud] === undefined) next.readAloud = Object.keys(next.providers)[0]
|
|
305
|
+
config = next
|
|
306
|
+
saveConfig()
|
|
307
|
+
return json(res, 200, { ok: true })
|
|
308
|
+
}
|
|
309
|
+
|
|
310
|
+
if (req.method === 'POST' && path === '/__dsh-voice/key') {
|
|
311
|
+
const body = await readBody(req)
|
|
312
|
+
const value = typeof body?.value === 'string' ? body.value.trim() : ''
|
|
313
|
+
const profileId = sanitizeProfileId(body?.profile ?? '')
|
|
314
|
+
const profile = config.providers[profileId]
|
|
315
|
+
if (profile === undefined) return json(res, 404, { error: '未知 provider。' })
|
|
316
|
+
if (!value) return json(res, 400, { ok: false, error: 'empty' })
|
|
317
|
+
await credentials.set(profile.keyRef, value)
|
|
318
|
+
return json(res, 200, { ok: true })
|
|
319
|
+
}
|
|
320
|
+
|
|
321
|
+
if (req.method === 'POST' && path === '/__dsh-voice/asr') {
|
|
322
|
+
const body = await readBody(req)
|
|
323
|
+
const audio = typeof body?.wavBase64 === 'string' ? body.wavBase64 : ''
|
|
324
|
+
if (!audio) return json(res, 400, { error: '缺少音频数据。' })
|
|
325
|
+
if (audio.length > (MAX_ASR_BYTES / 3) * 4) return json(res, 413, { error: '录音太长。' })
|
|
326
|
+
const profile = config.providers[config.dictation]
|
|
327
|
+
if (profile === undefined || !KINDS[profile.kind].stt) {
|
|
328
|
+
return json(res, 409, { error: `语音输入的 provider「${config.dictation}」不支持识别。` })
|
|
329
|
+
}
|
|
330
|
+
const resolved = await resolveKey(credentials, profile.keyRef)
|
|
331
|
+
if (resolved === undefined) return json(res, 409, { error: `未配置 key:Settings → Voice 里给「${config.dictation}」填 API key。` })
|
|
332
|
+
const result = await transcribe(profile, resolved.value, audio)
|
|
333
|
+
if (result.error !== undefined) return json(res, 502, { error: result.error })
|
|
334
|
+
return json(res, 200, { text: result.text })
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
if (req.method === 'GET' && path === '/__dsh-voice/tts') {
|
|
338
|
+
const text = url.searchParams.get('text') ?? ''
|
|
339
|
+
if (!text.trim()) return json(res, 400, { error: '缺少文本。' })
|
|
340
|
+
const limited = text.length > MAX_TTS_CHARS ? text.slice(0, MAX_TTS_CHARS) : text
|
|
341
|
+
const profile = config.providers[config.readAloud]
|
|
342
|
+
if (profile === undefined || !KINDS[profile.kind].tts) {
|
|
343
|
+
return json(res, 409, { error: `朗读的 provider「${config.readAloud}」不支持合成。` })
|
|
344
|
+
}
|
|
345
|
+
const resolved = await resolveKey(credentials, profile.keyRef)
|
|
346
|
+
if (resolved === undefined) return json(res, 409, { error: `未配置 key:Settings → Voice 里给「${config.readAloud}」填 API key。` })
|
|
347
|
+
const result = await synthesize(profile, resolved.value, limited)
|
|
348
|
+
if (result.error !== undefined) return json(res, 502, { error: result.error })
|
|
349
|
+
res.statusCode = 200
|
|
350
|
+
res.setHeader('Content-Type', 'audio/mpeg')
|
|
351
|
+
res.setHeader('Cache-Control', 'no-store')
|
|
352
|
+
if (result.buffer !== undefined) {
|
|
353
|
+
res.end(result.buffer)
|
|
354
|
+
return
|
|
355
|
+
}
|
|
356
|
+
const reader = result.stream.getReader()
|
|
357
|
+
try {
|
|
358
|
+
for (;;) {
|
|
359
|
+
const next = await reader.read()
|
|
360
|
+
if (next.done || res.destroyed) break
|
|
361
|
+
if (!res.write(Buffer.from(next.value))) {
|
|
362
|
+
await new Promise((resolve) => res.once('drain', resolve))
|
|
363
|
+
}
|
|
364
|
+
}
|
|
365
|
+
res.end()
|
|
366
|
+
} catch {
|
|
367
|
+
res.destroy()
|
|
368
|
+
}
|
|
369
|
+
return
|
|
370
|
+
}
|
|
371
|
+
|
|
372
|
+
json(res, 404, { error: 'not found' })
|
|
373
|
+
},
|
|
374
|
+
})
|
|
375
|
+
ctx.effect(() => dispose)
|
|
376
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@huakunshen/dsh-plugin-voice",
|
|
3
|
+
"version": "0.2.0",
|
|
4
|
+
"description": "Voice for the DeepSeek Harness web UI: user-managed speech provider profiles (Xiaomi MiMo, Fish Audio, OpenAI-compatible) with independent provider selection for read-aloud TTS and composer dictation STT.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"author": "Huakun Shen <huakun.shen@huakunshen.com>",
|
|
7
|
+
"type": "module",
|
|
8
|
+
"files": [
|
|
9
|
+
"index.js",
|
|
10
|
+
"client.js",
|
|
11
|
+
"cordis.patch.yml",
|
|
12
|
+
"README.md"
|
|
13
|
+
],
|
|
14
|
+
"exports": {
|
|
15
|
+
".": "./index.js",
|
|
16
|
+
"./client": "./client.js"
|
|
17
|
+
},
|
|
18
|
+
"dsh": {
|
|
19
|
+
"bundle": {
|
|
20
|
+
"patch": "./cordis.patch.yml"
|
|
21
|
+
},
|
|
22
|
+
"client": {
|
|
23
|
+
"platform": "web",
|
|
24
|
+
"inject": [
|
|
25
|
+
"@deepseek-ai/dsh-client-ui-conversation",
|
|
26
|
+
"@deepseek-ai/dsh-client-ui-settings"
|
|
27
|
+
]
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
}
|