@goodandready/dsh-voice 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +110 -0
- package/cordis.patch.yml +19 -0
- package/lib/chain.js +30 -0
- package/lib/client.js +604 -0
- package/lib/index.js +266 -0
- package/lib/providers.js +118 -0
- package/lib/wav.js +39 -0
- package/package.json +61 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-voice contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# dsh-voice
|
|
2
|
+
|
|
3
|
+
Voice input for the DeepSeek Harness Web GUI, in two modes, each with its own
|
|
4
|
+
provider fallback chain.
|
|
5
|
+
|
|
6
|
+
**Dictation** — press the mic, talk, and the text lands in the composer as you
|
|
7
|
+
go: speech is cut into phrases on silence and each phrase is transcribed on its
|
|
8
|
+
own. Press again to stop; sending stays in your hands.
|
|
9
|
+
|
|
10
|
+
**Voice message** — press the wave button, record, press again. The transcript
|
|
11
|
+
is inserted and sent to the agent after a short cancel window.
|
|
12
|
+
|
|
13
|
+
Both modes fall back across providers, so one outage or rate limit does not
|
|
14
|
+
lose your recording. API keys never reach the browser: audio is posted to the
|
|
15
|
+
plugin's own route and the host talks to the providers.
|
|
16
|
+
|
|
17
|
+
## Install
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
# From npm:
|
|
21
|
+
dsh plugin --profile web add @goodandready/dsh-voice
|
|
22
|
+
|
|
23
|
+
# Locally from a checkout:
|
|
24
|
+
dsh plugin --profile web add file:/path/to/dsh-voice
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Restart the Web UI afterwards, then hard-refresh the browser.
|
|
28
|
+
|
|
29
|
+
## Providers
|
|
30
|
+
|
|
31
|
+
| Key | Service | Default model | Credential |
|
|
32
|
+
|---|---|---|---|
|
|
33
|
+
| `deepgram` | Deepgram | `nova-2` | `DEEPGRAM_API_KEY` |
|
|
34
|
+
| `groq` | Groq | `whisper-large-v3-turbo` | `GROQ_API_KEY` |
|
|
35
|
+
| `hf` | HuggingFace Inference | `openai/whisper-large-v3` | `HF_TOKEN` |
|
|
36
|
+
| `local-whisper` | local whisper.cpp server | model given at server start | none, fully offline |
|
|
37
|
+
|
|
38
|
+
Keys are read through the DSH credentials service (Settings → Credentials, or
|
|
39
|
+
`$DSH_HOME/.credentials.yaml`), falling back to the process environment. A
|
|
40
|
+
provider without a key is skipped, not fatal.
|
|
41
|
+
|
|
42
|
+
## Configure (Web GUI)
|
|
43
|
+
|
|
44
|
+
Settings → **Голос** (Voice) has three blocks:
|
|
45
|
+
|
|
46
|
+
- **Dictation** — fallback chain (provider + optional model per row, order is
|
|
47
|
+
the order of attempts), language, and the silence threshold that ends a
|
|
48
|
+
phrase (`vadSilenceMs`, default 700 ms).
|
|
49
|
+
- **Voice message** — its own independent chain, language, and the cancel
|
|
50
|
+
window before the message is sent (`autoSendMs`, default 4000 ms).
|
|
51
|
+
- **General** — local whisper endpoint, binary, model, autostart.
|
|
52
|
+
|
|
53
|
+
Speed matters for dictation and accuracy for messages, which is why the chains
|
|
54
|
+
are separate: a sensible pair is Deepgram → Groq → local for dictation and
|
|
55
|
+
Groq → HuggingFace → local for messages.
|
|
56
|
+
|
|
57
|
+
## Local whisper.cpp
|
|
58
|
+
|
|
59
|
+
The local provider needs a running [whisper.cpp](https://github.com/ggerganov/whisper.cpp)
|
|
60
|
+
server:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
whisper-server -m /path/to/ggml-medium-q8_0.bin --host 127.0.0.1 --port 8001
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Set `whisperModel` (and `whisperBin` if it is not in `PATH`) and the plugin
|
|
67
|
+
launches the server itself when `autoStart` is on. While `whisperModel` is
|
|
68
|
+
empty, autostart stays off.
|
|
69
|
+
|
|
70
|
+
**ffmpeg is required for this provider.** whisper.cpp accepts WAV only and
|
|
71
|
+
rejects the webm/opus the browser records, so the host converts each recording
|
|
72
|
+
to 16 kHz mono WAV before forwarding it. Point `ffmpegBin` at your binary if it
|
|
73
|
+
is not in `PATH`.
|
|
74
|
+
|
|
75
|
+
## Tool
|
|
76
|
+
|
|
77
|
+
The plugin also registers `transcribe_audio(file_path, language?)` for the
|
|
78
|
+
agent, using the voice-message chain. Useful for recordings and interviews that
|
|
79
|
+
are already files on disk.
|
|
80
|
+
|
|
81
|
+
## Routes
|
|
82
|
+
|
|
83
|
+
| Route | Purpose |
|
|
84
|
+
|---|---|
|
|
85
|
+
| `POST /dsh-voice/transcribe` | `{dataBase64, mimeType, mode}` → `{ok, text, provider, tookMs}` |
|
|
86
|
+
| `GET /dsh-voice/status` | whisper server state and the effective chains |
|
|
87
|
+
|
|
88
|
+
## Structure
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
lib/index.js host: config, routes, transcribe_audio, whisper autostart
|
|
92
|
+
lib/providers.js the four providers, pure functions (network injected)
|
|
93
|
+
lib/chain.js fallback walk over a chain
|
|
94
|
+
lib/wav.js webm/opus → WAV 16 kHz mono via ffmpeg
|
|
95
|
+
lib/client.js browser: composer buttons, recording, settings page
|
|
96
|
+
test/ node --test units for the chain and the providers
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Run the tests with `npm test` (no dependencies, Node's built-in runner).
|
|
100
|
+
|
|
101
|
+
## Requirements
|
|
102
|
+
|
|
103
|
+
- DeepSeek Harness with the Web GUI
|
|
104
|
+
- Node 20+
|
|
105
|
+
- ffmpeg, for the local whisper provider
|
|
106
|
+
- a microphone reachable from the browser (HTTPS or localhost)
|
|
107
|
+
|
|
108
|
+
## License
|
|
109
|
+
|
|
110
|
+
MIT
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# dsh-voice bundle layer: applied automatically when the package is installed
|
|
2
|
+
# as a profile bundle (package.json declares dsh.bundle.patch).
|
|
3
|
+
#
|
|
4
|
+
# `name` must stay the full npm package name: the client-modules registry
|
|
5
|
+
# resolves the browser bundle by the loader entry name, so a shortened name
|
|
6
|
+
# leaves the UI half silently out of window.__DSH_BOOT__.
|
|
7
|
+
#
|
|
8
|
+
# Override any field in the profile's own cordis.patch.yml, e.g. to point the
|
|
9
|
+
# plugin at a local whisper.cpp build:
|
|
10
|
+
#
|
|
11
|
+
# - id: dsh-voice
|
|
12
|
+
# config:
|
|
13
|
+
# whisperBin: /opt/whisper.cpp/build/bin/whisper-server
|
|
14
|
+
# whisperModel: /opt/whisper.cpp/models/ggml-medium-q8_0.bin
|
|
15
|
+
#
|
|
16
|
+
- insert:
|
|
17
|
+
- id: dsh-voice
|
|
18
|
+
name: '@goodandready/dsh-voice'
|
|
19
|
+
config: {}
|
package/lib/chain.js
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
// Перебор цепочки провайдеров с фоллбеком. Без сети и без cordis — чистая
|
|
2
|
+
// функция, чтобы поведение при отказах можно было проверить юнит-тестом.
|
|
3
|
+
|
|
4
|
+
function normalizeError(err) {
|
|
5
|
+
const cause = err?.cause?.message || err?.message || String(err)
|
|
6
|
+
return cause.slice(0, 200)
|
|
7
|
+
}
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* @param order {string[]} порядок ключей провайдеров
|
|
11
|
+
* @param providers {Record<string, () => Promise<{ok, provider?, text?, reason?}>>}
|
|
12
|
+
* @returns {Promise<{provider: string, text: string, tookMs: number}>}
|
|
13
|
+
*/
|
|
14
|
+
export async function runChain(order, providers) {
|
|
15
|
+
const t0 = Date.now()
|
|
16
|
+
const keys = (Array.isArray(order) ? order : []).filter((k) => typeof providers[k] === 'function')
|
|
17
|
+
const errors = []
|
|
18
|
+
for (const key of keys) {
|
|
19
|
+
try {
|
|
20
|
+
const out = await providers[key]()
|
|
21
|
+
if (out && out.ok) {
|
|
22
|
+
return { provider: out.provider || key, text: out.text, tookMs: Date.now() - t0 }
|
|
23
|
+
}
|
|
24
|
+
errors.push(`${(out && out.provider) || key}: ${(out && out.reason) || 'unknown'}`)
|
|
25
|
+
} catch (err) {
|
|
26
|
+
errors.push(`${key}: ${normalizeError(err)}`)
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
throw new Error(`all providers failed (${errors.join('; ')})`)
|
|
30
|
+
}
|
package/lib/client.js
ADDED
|
@@ -0,0 +1,604 @@
|
|
|
1
|
+
// dsh-voice — клиентская половина (браузер).
|
|
2
|
+
//
|
|
3
|
+
// Две кнопки в conversation.input.right:
|
|
4
|
+
// микрофон — диктовка: речь режется по паузам, каждый кусок распознаётся и
|
|
5
|
+
// дописывается в строку ввода; отправка остаётся за пользователем;
|
|
6
|
+
// волна — голосовое: одна запись целиком, после распознавания текст
|
|
7
|
+
// уходит агенту по истечении окна отмены.
|
|
8
|
+
//
|
|
9
|
+
// Полоска записи живёт в conversation.input.dock, страница настроек — в
|
|
10
|
+
// settings.section.
|
|
11
|
+
|
|
12
|
+
window.__ModuleLoader__.load({
|
|
13
|
+
id: '@goodandready/dsh-voice',
|
|
14
|
+
factory: (require) => {
|
|
15
|
+
var module = { exports: {} }
|
|
16
|
+
var exports = module.exports
|
|
17
|
+
Object.defineProperty(exports, Symbol.toStringTag, { value: 'Module' })
|
|
18
|
+
let React = require('react')
|
|
19
|
+
|
|
20
|
+
const NS = 'dsh-voice'
|
|
21
|
+
|
|
22
|
+
// ------------------------------------------------------------------ css
|
|
23
|
+
const CSS =
|
|
24
|
+
'.dvo-btn{display:flex;align-items:center;justify-content:center;width:30px;height:30px;border-radius:50%;border:1px solid var(--dsw-alias-border-l1);background:transparent;color:var(--dsw-alias-label-secondary);cursor:pointer;padding:0;box-sizing:border-box}' +
|
|
25
|
+
'.dvo-btn:hover{color:var(--dsw-alias-label-primary);border-color:var(--dsw-alias-border-l2)}' +
|
|
26
|
+
'.dvo-btn[data-err="1"]{color:var(--dsw-alias-state-error-primary);border-color:var(--dsw-alias-state-error-primary)}' +
|
|
27
|
+
'.dvo-pill{display:flex;align-items:center;gap:10px;height:52px;border-radius:26px;background:var(--dsw-alias-bg-layer-1);border:1px solid var(--dsw-alias-border-l1);padding:0 14px;width:100%;max-width:720px;margin:0 auto;box-shadow:0 8px 24px rgba(0,0,0,.18);box-sizing:border-box}' +
|
|
28
|
+
'.dvo-pbtn{display:flex;align-items:center;justify-content:center;width:34px;height:34px;border-radius:50%;cursor:pointer;padding:0;flex:none;box-sizing:border-box;border:1px solid var(--dsw-alias-border-l2);background:transparent;color:var(--dsw-alias-label-primary)}' +
|
|
29
|
+
'.dvo-pbtn:hover{background:var(--dsw-alias-bg-layer-2)}' +
|
|
30
|
+
'.dvo-wave{flex:1;height:40px;width:100%;color:var(--dsw-alias-label-primary)}' +
|
|
31
|
+
'.dvo-status{display:flex;align-items:center;gap:8px;color:var(--dsw-alias-label-secondary);font-size:13px;flex:1;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}' +
|
|
32
|
+
'.dvo-err{color:var(--dsw-alias-state-error-primary)}' +
|
|
33
|
+
'.dvo-count{font-variant-numeric:tabular-nums;font-size:13px;color:var(--dsw-alias-label-secondary)}' +
|
|
34
|
+
'.dvo-spin{animation:dvo-spin 1s linear infinite}' +
|
|
35
|
+
'@keyframes dvo-spin{from{transform:rotate(0)}to{transform:rotate(360deg)}}'
|
|
36
|
+
const cssId = 'dsh-voice/client.module.css'
|
|
37
|
+
if (typeof document !== 'undefined' && !document.querySelector('style[data-plugin-css="' + cssId + '"]')) {
|
|
38
|
+
const tag = document.createElement('style')
|
|
39
|
+
tag.textContent = CSS
|
|
40
|
+
tag.setAttribute('data-plugin', 'dsh-voice')
|
|
41
|
+
tag.dataset.pluginCss = cssId
|
|
42
|
+
document.head.appendChild(tag)
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
// ---------------------------------------------------------------- store
|
|
46
|
+
const voice = {
|
|
47
|
+
phase: 'idle', // idle | recording | processing | pending | error
|
|
48
|
+
mode: 'dictation', // dictation | message
|
|
49
|
+
error: '',
|
|
50
|
+
rec: null,
|
|
51
|
+
levels: [],
|
|
52
|
+
pending: null, // {text, leftMs} — окно отмены режима message
|
|
53
|
+
inputActions: null,
|
|
54
|
+
input: null,
|
|
55
|
+
settings: { vadSilenceMs: 700, autoSendMs: 4000 },
|
|
56
|
+
listeners: new Set(),
|
|
57
|
+
notify() { this.listeners.forEach((l) => l()) },
|
|
58
|
+
set(patch) { Object.assign(this, patch); this.notify() },
|
|
59
|
+
subscribe(l) { this.listeners.add(l); return () => this.listeners.delete(l) },
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
function useVoice() {
|
|
63
|
+
const [, force] = React.useReducer((x) => x + 1, 0)
|
|
64
|
+
React.useEffect(() => voice.subscribe(force), [])
|
|
65
|
+
return voice
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
// ---------------------------------------------------------------- icons
|
|
69
|
+
const ic = { width: 18, height: 18, viewBox: '0 0 24 24', fill: 'none', stroke: 'currentColor', strokeWidth: 2, strokeLinecap: 'round', strokeLinejoin: 'round' }
|
|
70
|
+
const micIcon = () => React.createElement('svg', ic,
|
|
71
|
+
React.createElement('path', { d: 'M12 1a3 3 0 0 0-3 3v8a3 3 0 0 0 6 0V4a3 3 0 0 0-3-3z' }),
|
|
72
|
+
React.createElement('path', { d: 'M19 10v2a7 7 0 0 1-14 0v-2' }),
|
|
73
|
+
React.createElement('line', { x1: 12, y1: 19, x2: 12, y2: 23 }))
|
|
74
|
+
const waveIcon = () => React.createElement('svg', ic,
|
|
75
|
+
React.createElement('line', { x1: 4, y1: 10, x2: 4, y2: 14 }),
|
|
76
|
+
React.createElement('line', { x1: 8, y1: 7, x2: 8, y2: 17 }),
|
|
77
|
+
React.createElement('line', { x1: 12, y1: 4, x2: 12, y2: 20 }),
|
|
78
|
+
React.createElement('line', { x1: 16, y1: 7, x2: 16, y2: 17 }),
|
|
79
|
+
React.createElement('line', { x1: 20, y1: 10, x2: 20, y2: 14 }))
|
|
80
|
+
const xIcon = () => React.createElement('svg', ic,
|
|
81
|
+
React.createElement('line', { x1: 18, y1: 6, x2: 6, y2: 18 }),
|
|
82
|
+
React.createElement('line', { x1: 6, y1: 6, x2: 18, y2: 18 }))
|
|
83
|
+
const stopIcon = () => React.createElement('svg', ic,
|
|
84
|
+
React.createElement('rect', { x: 7, y: 7, width: 10, height: 10, rx: 2.5, fill: 'currentColor', stroke: 'none' }))
|
|
85
|
+
const spinIcon = () => React.createElement('svg', Object.assign({}, ic, { className: 'dvo-spin' }),
|
|
86
|
+
React.createElement('path', { d: 'M21 12a9 9 0 1 1-6.219-8.56' }))
|
|
87
|
+
const warnIcon = () => React.createElement('svg', ic,
|
|
88
|
+
React.createElement('path', { d: 'M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z' }),
|
|
89
|
+
React.createElement('line', { x1: 12, y1: 9, x2: 12, y2: 13 }),
|
|
90
|
+
React.createElement('line', { x1: 12, y1: 17, x2: 12.01, y2: 17 }))
|
|
91
|
+
|
|
92
|
+
// ------------------------------------------------------------ transport
|
|
93
|
+
function blobToBase64(blob) {
|
|
94
|
+
return new Promise((resolve, reject) => {
|
|
95
|
+
if (typeof FileReader === 'undefined') { reject(new Error('FileReader not supported')); return }
|
|
96
|
+
const fr = new FileReader()
|
|
97
|
+
fr.onload = () => { const s = String(fr.result || ''); resolve(s.indexOf(',') >= 0 ? s.slice(s.indexOf(',') + 1) : s) }
|
|
98
|
+
fr.onerror = () => reject(new Error('failed to read audio'))
|
|
99
|
+
fr.readAsDataURL(blob)
|
|
100
|
+
})
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
async function sendAudio(blob, mime, mode) {
|
|
104
|
+
const dataBase64 = await blobToBase64(blob)
|
|
105
|
+
const res = await fetch('/dsh-voice/transcribe', {
|
|
106
|
+
method: 'POST',
|
|
107
|
+
headers: { 'content-type': 'application/json' },
|
|
108
|
+
body: JSON.stringify({ dataBase64, mimeType: mime, mode }),
|
|
109
|
+
})
|
|
110
|
+
let parsed = null
|
|
111
|
+
try { parsed = await res.json() } catch (e) { /* не json */ }
|
|
112
|
+
if (!res.ok || !parsed || !parsed.ok) {
|
|
113
|
+
throw new Error((parsed && parsed.error && parsed.error.message) || `HTTP ${res.status}`)
|
|
114
|
+
}
|
|
115
|
+
return String(parsed.text || '').trim()
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
function appendDraft(text) {
|
|
119
|
+
const actions = voice.inputActions
|
|
120
|
+
if (!actions || typeof actions.setDraft !== 'function') {
|
|
121
|
+
voice.set({ phase: 'error', error: 'Композер недоступен' })
|
|
122
|
+
return
|
|
123
|
+
}
|
|
124
|
+
const draft = voice.input && typeof voice.input.draft === 'string' ? voice.input.draft : ''
|
|
125
|
+
actions.setDraft(draft ? draft + ' ' + text : text)
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// ------------------------------------------------------------ recording
|
|
129
|
+
function teardown(rec) {
|
|
130
|
+
if (!rec) return
|
|
131
|
+
try { rec.stream.getTracks().forEach((t) => t.stop()) } catch (e) { /* уже остановлен */ }
|
|
132
|
+
if (rec.audioCtx) { try { rec.audioCtx.close() } catch (e) { /* уже закрыт */ } }
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
function waitStop(recorder) {
|
|
136
|
+
return new Promise((resolve) => recorder.addEventListener('stop', resolve, { once: true }))
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
async function openMic(mode) {
|
|
140
|
+
if (typeof navigator === 'undefined' || !navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
|
|
141
|
+
throw new Error('Микрофон недоступен: нужен HTTPS или localhost')
|
|
142
|
+
}
|
|
143
|
+
if (typeof MediaRecorder === 'undefined') throw new Error('MediaRecorder не поддерживается браузером')
|
|
144
|
+
const stream = await navigator.mediaDevices.getUserMedia({
|
|
145
|
+
audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
|
|
146
|
+
})
|
|
147
|
+
let mimeType = 'audio/webm;codecs=opus'
|
|
148
|
+
if (!MediaRecorder.isTypeSupported(mimeType)) mimeType = ''
|
|
149
|
+
const recorder = mimeType ? new MediaRecorder(stream, { mimeType }) : new MediaRecorder(stream)
|
|
150
|
+
const rec = {
|
|
151
|
+
recorder, stream, mode,
|
|
152
|
+
chunks: [],
|
|
153
|
+
mime: mimeType || recorder.mimeType || 'audio/webm',
|
|
154
|
+
audioCtx: null, analyser: null,
|
|
155
|
+
cutting: false, closing: false,
|
|
156
|
+
silenceMs: 0, hadSpeech: false,
|
|
157
|
+
}
|
|
158
|
+
recorder.ondataavailable = (e) => { if (e.data && e.data.size > 0) rec.chunks.push(e.data) }
|
|
159
|
+
const AC = typeof AudioContext !== 'undefined' ? AudioContext
|
|
160
|
+
: (typeof webkitAudioContext !== 'undefined' ? webkitAudioContext : null)
|
|
161
|
+
if (AC) {
|
|
162
|
+
rec.audioCtx = new AC()
|
|
163
|
+
const src = rec.audioCtx.createMediaStreamSource(stream)
|
|
164
|
+
rec.analyser = rec.audioCtx.createAnalyser()
|
|
165
|
+
rec.analyser.fftSize = 128
|
|
166
|
+
src.connect(rec.analyser)
|
|
167
|
+
}
|
|
168
|
+
// Без timeslice: только тогда каждый stop() даёт самостоятельный webm-файл.
|
|
169
|
+
recorder.start()
|
|
170
|
+
return rec
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
function currentLevel(rec) {
|
|
174
|
+
if (!rec || !rec.analyser) return 0
|
|
175
|
+
const data = new Uint8Array(rec.analyser.frequencyBinCount)
|
|
176
|
+
rec.analyser.getByteFrequencyData(data)
|
|
177
|
+
let sum = 0
|
|
178
|
+
for (let i = 0; i < data.length; i++) sum += data[i]
|
|
179
|
+
return Math.min(1, (sum / data.length / 255) * 2.2)
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
// Режет текущую фразу: останавливает рекордер, отправляет готовый файл и
|
|
183
|
+
// немедленно начинает новую запись тем же рекордером.
|
|
184
|
+
function cutPhrase() {
|
|
185
|
+
const rec = voice.rec
|
|
186
|
+
if (!rec || rec.cutting || rec.closing) return
|
|
187
|
+
rec.cutting = true
|
|
188
|
+
const stopped = waitStop(rec.recorder)
|
|
189
|
+
try { rec.recorder.stop() } catch (e) { /* уже остановлен */ }
|
|
190
|
+
stopped.then(async () => {
|
|
191
|
+
const blob = new Blob(rec.chunks, { type: rec.mime })
|
|
192
|
+
rec.chunks = []
|
|
193
|
+
rec.silenceMs = 0
|
|
194
|
+
rec.hadSpeech = false
|
|
195
|
+
if (!rec.closing) {
|
|
196
|
+
try { rec.recorder.start() } catch (e) { /* поток закрылся */ }
|
|
197
|
+
}
|
|
198
|
+
rec.cutting = false
|
|
199
|
+
if (blob.size < 1200) return // слишком короткий кусок — это не речь
|
|
200
|
+
try {
|
|
201
|
+
const text = await sendAudio(blob, rec.mime, 'dictation')
|
|
202
|
+
if (text) appendDraft(text)
|
|
203
|
+
} catch (e) {
|
|
204
|
+
voice.set({ error: String(e && e.message ? e.message : e) })
|
|
205
|
+
}
|
|
206
|
+
})
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
function startRecording(mode) {
|
|
210
|
+
if (voice.phase !== 'idle' && voice.phase !== 'error') return
|
|
211
|
+
voice.set({ phase: 'recording', mode, error: '', levels: [] })
|
|
212
|
+
openMic(mode)
|
|
213
|
+
.then((rec) => { voice.rec = rec; voice.notify() })
|
|
214
|
+
.catch((err) => voice.set({ phase: 'error', error: String(err && err.message ? err.message : err), rec: null }))
|
|
215
|
+
}
|
|
216
|
+
|
|
217
|
+
function startDictation() { startRecording('dictation') }
|
|
218
|
+
function startMessage() { startRecording('message') }
|
|
219
|
+
|
|
220
|
+
function cancelCurrent() {
|
|
221
|
+
const rec = voice.rec
|
|
222
|
+
voice.pending = null
|
|
223
|
+
if (!rec) { voice.set({ phase: 'idle', error: '' }); return }
|
|
224
|
+
rec.closing = true
|
|
225
|
+
const stopped = waitStop(rec.recorder)
|
|
226
|
+
try { rec.recorder.stop() } catch (e) { /* уже остановлен */ }
|
|
227
|
+
stopped.then(() => { teardown(rec); voice.rec = null; voice.set({ phase: 'idle', error: '' }) })
|
|
228
|
+
}
|
|
229
|
+
|
|
230
|
+
// Останов по второму нажатию: диктовка досылает хвост, голосовое —
|
|
231
|
+
// отправляет всю запись и открывает окно отмены.
|
|
232
|
+
function stopCurrent() {
|
|
233
|
+
const rec = voice.rec
|
|
234
|
+
if (!rec || rec.closing) return
|
|
235
|
+
rec.closing = true
|
|
236
|
+
const mode = rec.mode
|
|
237
|
+
const stopped = waitStop(rec.recorder)
|
|
238
|
+
try { rec.recorder.stop() } catch (e) { /* уже остановлен */ }
|
|
239
|
+
stopped.then(async () => {
|
|
240
|
+
const blob = new Blob(rec.chunks, { type: rec.mime })
|
|
241
|
+
teardown(rec)
|
|
242
|
+
voice.rec = null
|
|
243
|
+
if (blob.size < 1200) { voice.set({ phase: 'idle' }); return }
|
|
244
|
+
voice.set({ phase: 'processing' })
|
|
245
|
+
try {
|
|
246
|
+
const text = await sendAudio(blob, rec.mime, mode)
|
|
247
|
+
if (!text) { voice.set({ phase: 'error', error: 'Речь не распознана' }); return }
|
|
248
|
+
appendDraft(text)
|
|
249
|
+
if (mode === 'message') {
|
|
250
|
+
voice.set({ phase: 'pending', pending: { text: text, leftMs: voice.settings.autoSendMs } })
|
|
251
|
+
} else {
|
|
252
|
+
voice.set({ phase: 'idle' })
|
|
253
|
+
}
|
|
254
|
+
} catch (e) {
|
|
255
|
+
voice.set({ phase: 'error', error: String(e && e.message ? e.message : e) })
|
|
256
|
+
}
|
|
257
|
+
})
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
function submitPending() {
|
|
261
|
+
voice.pending = null
|
|
262
|
+
voice.set({ phase: 'idle' })
|
|
263
|
+
const actions = voice.inputActions
|
|
264
|
+
if (actions && typeof actions.submit === 'function') {
|
|
265
|
+
setTimeout(() => { try { actions.submit() } catch (e) { /* композер занят */ } }, 0)
|
|
266
|
+
}
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
function keepPending() {
|
|
270
|
+
voice.pending = null
|
|
271
|
+
voice.set({ phase: 'idle' })
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
// ----------------------------------------------------------- components
|
|
275
|
+
function VoiceButtons(props) {
|
|
276
|
+
const v = useVoice()
|
|
277
|
+
voice.inputActions = props.inputActions
|
|
278
|
+
voice.input = props.input
|
|
279
|
+
if (v.phase !== 'idle' && v.phase !== 'error') return null
|
|
280
|
+
const err = v.phase === 'error'
|
|
281
|
+
return React.createElement(React.Fragment, null,
|
|
282
|
+
React.createElement('button', {
|
|
283
|
+
type: 'button', className: 'dvo-btn', 'data-err': err ? '1' : '0',
|
|
284
|
+
title: err ? v.error : 'Голосовой набор', onClick: startDictation,
|
|
285
|
+
}, micIcon()),
|
|
286
|
+
React.createElement('button', {
|
|
287
|
+
type: 'button', className: 'dvo-btn', 'data-err': err ? '1' : '0',
|
|
288
|
+
title: err ? v.error : 'Голосовое сообщение', onClick: startMessage,
|
|
289
|
+
}, waveIcon()),
|
|
290
|
+
)
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
function RecordPill(props) {
|
|
294
|
+
const v = useVoice()
|
|
295
|
+
const canvasRef = React.useRef(null)
|
|
296
|
+
voice.inputActions = props.inputActions
|
|
297
|
+
voice.input = props.input
|
|
298
|
+
const ctx = props.ctx
|
|
299
|
+
|
|
300
|
+
// VAD: копим тишину и режем фразу, когда пауза превысила порог.
|
|
301
|
+
React.useEffect(() => {
|
|
302
|
+
if (v.phase !== 'recording') return
|
|
303
|
+
const tick = 50
|
|
304
|
+
const dispose = ctx.interval(() => {
|
|
305
|
+
const rec = voice.rec
|
|
306
|
+
if (!rec) return
|
|
307
|
+
const level = currentLevel(rec)
|
|
308
|
+
voice.levels.push(level)
|
|
309
|
+
if (voice.levels.length > 150) voice.levels.shift()
|
|
310
|
+
if (level > 0.06) { rec.hadSpeech = true; rec.silenceMs = 0 }
|
|
311
|
+
else if (rec.hadSpeech) rec.silenceMs += tick
|
|
312
|
+
if (rec.mode === 'dictation' && rec.hadSpeech && rec.silenceMs >= voice.settings.vadSilenceMs) {
|
|
313
|
+
cutPhrase()
|
|
314
|
+
}
|
|
315
|
+
}, tick)
|
|
316
|
+
return () => dispose()
|
|
317
|
+
}, [v.phase])
|
|
318
|
+
|
|
319
|
+
// Осциллограмма.
|
|
320
|
+
React.useEffect(() => {
|
|
321
|
+
if (v.phase !== 'recording') return
|
|
322
|
+
const dispose = ctx.interval(() => {
|
|
323
|
+
const canvas = canvasRef.current
|
|
324
|
+
if (!canvas) return
|
|
325
|
+
const g = canvas.getContext('2d')
|
|
326
|
+
const w = canvas.width, h = canvas.height
|
|
327
|
+
g.clearRect(0, 0, w, h)
|
|
328
|
+
if (!voice.waveColor) {
|
|
329
|
+
try { voice.waveColor = getComputedStyle(canvas).color || '#fff' } catch (e) { voice.waveColor = '#fff' }
|
|
330
|
+
}
|
|
331
|
+
const levels = voice.levels
|
|
332
|
+
const midY = h / 2
|
|
333
|
+
for (let i = 0; i < levels.length && i * 7 < w; i++) {
|
|
334
|
+
const level = levels[levels.length - 1 - i]
|
|
335
|
+
const age = i / levels.length
|
|
336
|
+
const x = w - 10 - i * 7
|
|
337
|
+
const hh = Math.max(2.5, level * (h - 6) * 0.5 * (1 - age * 0.35))
|
|
338
|
+
g.globalAlpha = 1 - age * 0.75
|
|
339
|
+
g.fillStyle = voice.waveColor
|
|
340
|
+
g.fillRect(x, midY - hh, 3.5, hh * 2)
|
|
341
|
+
}
|
|
342
|
+
g.globalAlpha = 1
|
|
343
|
+
}, 50)
|
|
344
|
+
return () => dispose()
|
|
345
|
+
}, [v.phase])
|
|
346
|
+
|
|
347
|
+
// Окно отмены режима message.
|
|
348
|
+
React.useEffect(() => {
|
|
349
|
+
if (v.phase !== 'pending') return
|
|
350
|
+
const tick = 100
|
|
351
|
+
const dispose = ctx.interval(() => {
|
|
352
|
+
const p = voice.pending
|
|
353
|
+
if (!p) return
|
|
354
|
+
p.leftMs -= tick
|
|
355
|
+
if (p.leftMs <= 0) { submitPending(); return }
|
|
356
|
+
voice.notify()
|
|
357
|
+
}, tick)
|
|
358
|
+
return () => dispose()
|
|
359
|
+
}, [v.phase])
|
|
360
|
+
|
|
361
|
+
if (v.phase === 'idle') return null
|
|
362
|
+
|
|
363
|
+
if (v.phase === 'recording') {
|
|
364
|
+
const hint = v.mode === 'dictation' ? 'Диктовка — текст дописывается в строку' : 'Запись голосового'
|
|
365
|
+
return React.createElement('div', { className: 'dvo-pill' },
|
|
366
|
+
React.createElement('button', { type: 'button', className: 'dvo-pbtn', title: 'Отмена', onClick: cancelCurrent }, xIcon()),
|
|
367
|
+
React.createElement('canvas', { className: 'dvo-wave', ref: canvasRef, width: 720, height: 40 }),
|
|
368
|
+
React.createElement('span', { className: 'dvo-status' }, hint),
|
|
369
|
+
React.createElement('button', { type: 'button', className: 'dvo-pbtn', title: 'Стоп', onClick: stopCurrent }, stopIcon()),
|
|
370
|
+
)
|
|
371
|
+
}
|
|
372
|
+
|
|
373
|
+
if (v.phase === 'processing') {
|
|
374
|
+
return React.createElement('div', { className: 'dvo-pill' },
|
|
375
|
+
React.createElement('span', { className: 'dvo-status' }, spinIcon(), 'Распознаю…'))
|
|
376
|
+
}
|
|
377
|
+
|
|
378
|
+
if (v.phase === 'pending') {
|
|
379
|
+
const left = Math.max(0, Math.ceil((voice.pending ? voice.pending.leftMs : 0) / 1000))
|
|
380
|
+
return React.createElement('div', { className: 'dvo-pill' },
|
|
381
|
+
React.createElement('span', { className: 'dvo-status' }, 'Отправляю агенту через'),
|
|
382
|
+
React.createElement('span', { className: 'dvo-count' }, left + ' с'),
|
|
383
|
+
React.createElement('button', { type: 'button', className: 'dvo-pbtn', title: 'Отменить отправку', onClick: keepPending }, xIcon()),
|
|
384
|
+
)
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
return React.createElement('div', { className: 'dvo-pill' },
|
|
388
|
+
React.createElement('span', { className: 'dvo-status dvo-err' }, warnIcon(), v.error),
|
|
389
|
+
React.createElement('button', { type: 'button', className: 'dvo-pbtn', title: 'Скрыть', onClick: () => voice.set({ phase: 'idle', error: '' }) }, xIcon()),
|
|
390
|
+
)
|
|
391
|
+
}
|
|
392
|
+
|
|
393
|
+
// --------------------------------------------------------------- slots
|
|
394
|
+
function registerComposer(ctx) {
|
|
395
|
+
ctx.slots.inject('conversation.input.right', () => ctx.slots.register(
|
|
396
|
+
{ name: 'conversation.input.right', id: '@goodandready/dsh-voice', order: 6, label: () => 'Голос' },
|
|
397
|
+
(props) => React.createElement(VoiceButtons, { input: props.input, inputActions: props.inputActions }),
|
|
398
|
+
))
|
|
399
|
+
ctx.slots.inject('conversation.input.dock', () => ctx.slots.register(
|
|
400
|
+
{ name: 'conversation.input.dock', id: 'dsh-voice-rec', order: 0, label: () => 'Запись голоса' },
|
|
401
|
+
(props) => React.createElement(RecordPill, { input: props.input, inputActions: props.inputActions, ctx: ctx }),
|
|
402
|
+
))
|
|
403
|
+
}
|
|
404
|
+
|
|
405
|
+
// ------------------------------------------------------- settings page
|
|
406
|
+
const PROVIDERS = ['deepgram', 'groq', 'hf', 'local-whisper']
|
|
407
|
+
const MODEL_HINT = {
|
|
408
|
+
deepgram: 'nova-2', groq: 'whisper-large-v3-turbo',
|
|
409
|
+
hf: 'openai/whisper-large-v3', 'local-whisper': 'задаётся при запуске сервера',
|
|
410
|
+
}
|
|
411
|
+
const LANGS = ['auto', 'ru', 'en', 'uk', 'de']
|
|
412
|
+
|
|
413
|
+
const SET_CSS =
|
|
414
|
+
'.dvs-wrap{display:flex;flex-direction:column;gap:22px;padding:4px 0;max-width:720px}' +
|
|
415
|
+
'.dvs-block{display:flex;flex-direction:column;gap:10px}' +
|
|
416
|
+
'.dvs-h{font-size:14px;font-weight:600;color:var(--dsw-alias-label-primary)}' +
|
|
417
|
+
'.dvs-sub{font-size:12px;color:var(--dsw-alias-label-secondary)}' +
|
|
418
|
+
'.dvs-row{display:flex;gap:8px;align-items:center}' +
|
|
419
|
+
'.dvs-row select,.dvs-row input{background:var(--dsw-alias-bg-layer-1);border:1px solid var(--dsw-alias-border-l1);color:var(--dsw-alias-label-primary);border-radius:6px;padding:6px 8px;font-size:13px}' +
|
|
420
|
+
'.dvs-row .dvs-model{flex:1}' +
|
|
421
|
+
'.dvs-field{display:flex;flex-direction:column;gap:4px;font-size:12px;color:var(--dsw-alias-label-secondary)}' +
|
|
422
|
+
'.dvs-field input,.dvs-field select{background:var(--dsw-alias-bg-layer-1);border:1px solid var(--dsw-alias-border-l1);color:var(--dsw-alias-label-primary);border-radius:6px;padding:6px 8px;font-size:13px}' +
|
|
423
|
+
'.dvs-mini{border:1px solid var(--dsw-alias-border-l2);background:transparent;color:var(--dsw-alias-label-primary);border-radius:6px;width:28px;height:28px;cursor:pointer;flex:none}' +
|
|
424
|
+
'.dvs-save{background:var(--dsw-alias-brand-primary);color:#fff;border:none;border-radius:6px;padding:7px 14px;font-size:13px;cursor:pointer}' +
|
|
425
|
+
'.dvs-ok{font-size:12px;color:var(--dsw-alias-state-success-primary)}' +
|
|
426
|
+
'.dvs-bad{font-size:12px;color:var(--dsw-alias-state-error-primary)}'
|
|
427
|
+
const setCssId = 'dsh-voice/settings.module.css'
|
|
428
|
+
if (typeof document !== 'undefined' && !document.querySelector('style[data-plugin-css="' + setCssId + '"]')) {
|
|
429
|
+
const tag = document.createElement('style')
|
|
430
|
+
tag.textContent = SET_CSS
|
|
431
|
+
tag.setAttribute('data-plugin', 'dsh-voice')
|
|
432
|
+
tag.dataset.pluginCss = setCssId
|
|
433
|
+
document.head.appendChild(tag)
|
|
434
|
+
}
|
|
435
|
+
|
|
436
|
+
// Редактор одной цепочки: строки «провайдер + модель» с перестановкой.
|
|
437
|
+
function ChainEditor(props) {
|
|
438
|
+
const rows = Array.isArray(props.value) ? props.value : []
|
|
439
|
+
const change = (i, patch) => {
|
|
440
|
+
const next = rows.map((r, k) => (k === i ? Object.assign({}, r, patch) : r))
|
|
441
|
+
props.onChange(next)
|
|
442
|
+
}
|
|
443
|
+
const move = (i, delta) => {
|
|
444
|
+
const j = i + delta
|
|
445
|
+
if (j < 0 || j >= rows.length) return
|
|
446
|
+
const next = rows.slice()
|
|
447
|
+
const tmp = next[i]; next[i] = next[j]; next[j] = tmp
|
|
448
|
+
props.onChange(next)
|
|
449
|
+
}
|
|
450
|
+
const remove = (i) => props.onChange(rows.filter((_, k) => k !== i))
|
|
451
|
+
const add = () => props.onChange(rows.concat([{ provider: 'local-whisper', model: '' }]))
|
|
452
|
+
|
|
453
|
+
return React.createElement('div', { className: 'dvs-block' },
|
|
454
|
+
rows.map((row, i) => React.createElement('div', { className: 'dvs-row', key: i },
|
|
455
|
+
React.createElement('select', {
|
|
456
|
+
value: row.provider, disabled: !props.writable,
|
|
457
|
+
onChange: (e) => change(i, { provider: e.target.value }),
|
|
458
|
+
}, PROVIDERS.map((p) => React.createElement('option', { key: p, value: p }, p))),
|
|
459
|
+
React.createElement('input', {
|
|
460
|
+
className: 'dvs-model', value: row.model || '', disabled: !props.writable,
|
|
461
|
+
placeholder: MODEL_HINT[row.provider] || '', onChange: (e) => change(i, { model: e.target.value }),
|
|
462
|
+
}),
|
|
463
|
+
React.createElement('button', { type: 'button', className: 'dvs-mini', title: 'Выше', disabled: !props.writable, onClick: () => move(i, -1) }, '↑'),
|
|
464
|
+
React.createElement('button', { type: 'button', className: 'dvs-mini', title: 'Ниже', disabled: !props.writable, onClick: () => move(i, 1) }, '↓'),
|
|
465
|
+
React.createElement('button', { type: 'button', className: 'dvs-mini', title: 'Убрать', disabled: !props.writable, onClick: () => remove(i) }, '×'),
|
|
466
|
+
)),
|
|
467
|
+
React.createElement('div', { className: 'dvs-row' },
|
|
468
|
+
React.createElement('button', { type: 'button', className: 'dvs-mini', title: 'Добавить провайдера', disabled: !props.writable, onClick: add }, '+'),
|
|
469
|
+
React.createElement('span', { className: 'dvs-sub' }, 'Порядок сверху вниз — порядок попыток'),
|
|
470
|
+
),
|
|
471
|
+
)
|
|
472
|
+
}
|
|
473
|
+
|
|
474
|
+
function VoiceSection(props) {
|
|
475
|
+
const ctx = props.ctx
|
|
476
|
+
const scope = ctx.settingsScope.bind({ namespace: NS })
|
|
477
|
+
const [snap, setSnap] = React.useState(null)
|
|
478
|
+
const [draft, setDraft] = React.useState(null)
|
|
479
|
+
const [saved, setSaved] = React.useState(false)
|
|
480
|
+
const [err, setErr] = React.useState('')
|
|
481
|
+
|
|
482
|
+
React.useEffect(() => {
|
|
483
|
+
let alive = true
|
|
484
|
+
const render = () => { if (alive) setSnap(scope.getSnapshot()) }
|
|
485
|
+
render()
|
|
486
|
+
const off = scope.subscribe(render)
|
|
487
|
+
return () => { alive = false; off() }
|
|
488
|
+
}, [])
|
|
489
|
+
|
|
490
|
+
const value = snap && snap.value ? snap.value : {}
|
|
491
|
+
const writable = snap ? snap.writable !== false : false
|
|
492
|
+
React.useEffect(() => { if (snap && draft === null) setDraft(JSON.parse(JSON.stringify(value))) }, [snap, draft, value])
|
|
493
|
+
// Клиент должен знать порог VAD и окно отмены — они живут в тех же настройках.
|
|
494
|
+
React.useEffect(() => {
|
|
495
|
+
if (!snap) return
|
|
496
|
+
voice.settings = {
|
|
497
|
+
vadSilenceMs: Number(value && value.dictation && value.dictation.vadSilenceMs) || 700,
|
|
498
|
+
autoSendMs: Number(value && value.message && value.message.autoSendMs) || 4000,
|
|
499
|
+
}
|
|
500
|
+
}, [snap])
|
|
501
|
+
|
|
502
|
+
if (!snap) return React.createElement('div', { className: 'dvs-wrap' }, 'Загрузка…')
|
|
503
|
+
|
|
504
|
+
const setIn = (mode, key, v) => setDraft((d) => {
|
|
505
|
+
const next = JSON.parse(JSON.stringify(d || {}))
|
|
506
|
+
next[mode] = next[mode] || {}
|
|
507
|
+
next[mode][key] = v
|
|
508
|
+
return next
|
|
509
|
+
})
|
|
510
|
+
const setTop = (key, v) => setDraft((d) => Object.assign({}, d || {}, { [key]: v }))
|
|
511
|
+
|
|
512
|
+
const save = async () => {
|
|
513
|
+
setErr(''); setSaved(false)
|
|
514
|
+
if (!draft) return
|
|
515
|
+
try {
|
|
516
|
+
for (const k of Object.keys(draft)) await scope.set(k, draft[k])
|
|
517
|
+
voice.settings = {
|
|
518
|
+
vadSilenceMs: Number(draft.dictation && draft.dictation.vadSilenceMs) || 700,
|
|
519
|
+
autoSendMs: Number(draft.message && draft.message.autoSendMs) || 4000,
|
|
520
|
+
}
|
|
521
|
+
setSaved(true); setTimeout(() => setSaved(false), 2000)
|
|
522
|
+
} catch (e) { setErr(String(e && e.message ? e.message : e)) }
|
|
523
|
+
}
|
|
524
|
+
|
|
525
|
+
const modeVal = (mode, key, fallback) => {
|
|
526
|
+
const m = draft && draft[mode]
|
|
527
|
+
return m && m[key] !== undefined ? m[key] : fallback
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
const langField = (mode) => React.createElement('label', { className: 'dvs-field' }, 'Язык',
|
|
531
|
+
React.createElement('select', {
|
|
532
|
+
value: modeVal(mode, 'language', 'ru'), disabled: !writable,
|
|
533
|
+
onChange: (e) => setIn(mode, 'language', e.target.value),
|
|
534
|
+
}, LANGS.map((o) => React.createElement('option', { key: o, value: o }, o))))
|
|
535
|
+
|
|
536
|
+
const numField = (mode, key, label, hint) => React.createElement('label', { className: 'dvs-field' }, label,
|
|
537
|
+
React.createElement('input', {
|
|
538
|
+
type: 'number', value: modeVal(mode, key, ''), disabled: !writable,
|
|
539
|
+
onChange: (e) => setIn(mode, key, Number(e.target.value)),
|
|
540
|
+
}),
|
|
541
|
+
React.createElement('span', { className: 'dvs-sub' }, hint))
|
|
542
|
+
|
|
543
|
+
const textField = (key, label, hint) => React.createElement('label', { className: 'dvs-field' }, label,
|
|
544
|
+
React.createElement('input', {
|
|
545
|
+
value: draft && draft[key] !== undefined ? draft[key] : '', disabled: !writable,
|
|
546
|
+
onChange: (e) => setTop(key, e.target.value),
|
|
547
|
+
}),
|
|
548
|
+
React.createElement('span', { className: 'dvs-sub' }, hint))
|
|
549
|
+
|
|
550
|
+
return React.createElement('div', { className: 'dvs-wrap' },
|
|
551
|
+
React.createElement('div', { className: 'dvs-block' },
|
|
552
|
+
React.createElement('div', { className: 'dvs-h' }, 'Голосовой набор'),
|
|
553
|
+
React.createElement('div', { className: 'dvs-sub' }, 'Речь режется по паузам, текст дописывается в строку ввода.'),
|
|
554
|
+
React.createElement(ChainEditor, {
|
|
555
|
+
value: draft && draft.dictation ? draft.dictation.chain : [], writable: writable,
|
|
556
|
+
onChange: (v) => setIn('dictation', 'chain', v),
|
|
557
|
+
}),
|
|
558
|
+
langField('dictation'),
|
|
559
|
+
numField('dictation', 'vadSilenceMs', 'Пауза до конца фразы, мс', 'Меньше — чаще куски и быстрее текст, но выше риск обрезать слово'),
|
|
560
|
+
),
|
|
561
|
+
React.createElement('div', { className: 'dvs-block' },
|
|
562
|
+
React.createElement('div', { className: 'dvs-h' }, 'Голосовое сообщение'),
|
|
563
|
+
React.createElement('div', { className: 'dvs-sub' }, 'Одна запись целиком, после распознавания уходит агенту.'),
|
|
564
|
+
React.createElement(ChainEditor, {
|
|
565
|
+
value: draft && draft.message ? draft.message.chain : [], writable: writable,
|
|
566
|
+
onChange: (v) => setIn('message', 'chain', v),
|
|
567
|
+
}),
|
|
568
|
+
langField('message'),
|
|
569
|
+
numField('message', 'autoSendMs', 'Окно отмены, мс', 'Сколько времени можно отменить автоматическую отправку'),
|
|
570
|
+
),
|
|
571
|
+
React.createElement('div', { className: 'dvs-block' },
|
|
572
|
+
React.createElement('div', { className: 'dvs-h' }, 'Общее'),
|
|
573
|
+
textField('whisperUrl', 'Локальный whisper: endpoint', 'POST /inference сервера whisper.cpp'),
|
|
574
|
+
textField('whisperBin', 'Локальный whisper: бинарь', 'используется при автозапуске'),
|
|
575
|
+
textField('whisperModel', 'Локальный whisper: модель', 'используется при автозапуске'),
|
|
576
|
+
React.createElement('label', { className: 'dvs-field' }, 'Автозапуск локального whisper',
|
|
577
|
+
React.createElement('input', {
|
|
578
|
+
type: 'checkbox', checked: !!(draft && draft.autoStart), disabled: !writable,
|
|
579
|
+
onChange: (e) => setTop('autoStart', e.target.checked),
|
|
580
|
+
})),
|
|
581
|
+
),
|
|
582
|
+
React.createElement('div', { className: 'dvs-row' },
|
|
583
|
+
React.createElement('button', { type: 'button', className: 'dvs-save', disabled: !writable, onClick: save }, 'Сохранить'),
|
|
584
|
+
saved ? React.createElement('span', { className: 'dvs-ok' }, 'Сохранено ✓') : null,
|
|
585
|
+
err ? React.createElement('span', { className: 'dvs-bad' }, err) : null,
|
|
586
|
+
),
|
|
587
|
+
)
|
|
588
|
+
}
|
|
589
|
+
|
|
590
|
+
function registerSettings(ctx) {
|
|
591
|
+
ctx.slots.inject('settings.section', () => ctx.slots.register(
|
|
592
|
+
{ name: 'settings.section', id: '@goodandready/dsh-voice', order: 30, label: () => 'Голос', inject: () => ({ ctx: ctx }) },
|
|
593
|
+
VoiceSection,
|
|
594
|
+
))
|
|
595
|
+
}
|
|
596
|
+
|
|
597
|
+
exports.inject = ['timer', 'slots', 'settingsScope']
|
|
598
|
+
exports.apply = function apply(ctx) {
|
|
599
|
+
registerComposer(ctx)
|
|
600
|
+
registerSettings(ctx)
|
|
601
|
+
}
|
|
602
|
+
return module.exports
|
|
603
|
+
},
|
|
604
|
+
})
|
package/lib/index.js
ADDED
|
@@ -0,0 +1,266 @@
|
|
|
1
|
+
// dsh-voice — хост-половина.
|
|
2
|
+
//
|
|
3
|
+
// Два режима ввода голосом, у каждого своя цепочка провайдеров:
|
|
4
|
+
// dictation — браузер режет речь по паузам и шлёт куски, текст дописывается
|
|
5
|
+
// в строку ввода;
|
|
6
|
+
// message — одна запись целиком, текст уходит агенту после окна отмены.
|
|
7
|
+
//
|
|
8
|
+
// Роуты:
|
|
9
|
+
// POST /dsh-voice/transcribe {dataBase64, mimeType, mode} -> {ok, text, provider, tookMs}
|
|
10
|
+
// GET /dsh-voice/status -> {ok, whisperRunning, modes}
|
|
11
|
+
//
|
|
12
|
+
// Ключи провайдеров читаются на хосте через ctx.credentials и в браузер не
|
|
13
|
+
// попадают.
|
|
14
|
+
|
|
15
|
+
import z from '@deepseek-ai/schemastery'
|
|
16
|
+
import { defineTool } from '@deepseek-ai/dsh-tools'
|
|
17
|
+
import { credentialRef } from '@deepseek-ai/dsh-credentials'
|
|
18
|
+
import { readFile, stat } from 'node:fs/promises'
|
|
19
|
+
import path from 'node:path'
|
|
20
|
+
import { runChain } from './chain.js'
|
|
21
|
+
import { makeProviders, PROVIDER_KEYS, DEFAULT_MODELS } from './providers.js'
|
|
22
|
+
import { toWav16k } from './wav.js'
|
|
23
|
+
|
|
24
|
+
export const name = 'dsh-voice'
|
|
25
|
+
export const inject = ['tools', 'credentials', 'webServer', 'shell']
|
|
26
|
+
|
|
27
|
+
const ChainEntry = z.object({
|
|
28
|
+
provider: z.string().default('local-whisper')
|
|
29
|
+
.description(`Provider key. One of: ${PROVIDER_KEYS.join(', ')}.`),
|
|
30
|
+
model: z.string().default('')
|
|
31
|
+
.description('Model override. Empty means the provider default.'),
|
|
32
|
+
})
|
|
33
|
+
|
|
34
|
+
export const Config = z.object({
|
|
35
|
+
dictation: z.object({
|
|
36
|
+
chain: z.array(ChainEntry)
|
|
37
|
+
.default([{ provider: 'deepgram', model: '' }, { provider: 'groq', model: '' }, { provider: 'local-whisper', model: '' }])
|
|
38
|
+
.description('Fallback chain for dictation. Speed matters more than accuracy here.'),
|
|
39
|
+
language: z.string().default('ru'),
|
|
40
|
+
vadSilenceMs: z.number().default(700)
|
|
41
|
+
.description('Silence longer than this ends a phrase and sends the chunk.'),
|
|
42
|
+
}).default({}),
|
|
43
|
+
message: z.object({
|
|
44
|
+
chain: z.array(ChainEntry)
|
|
45
|
+
.default([{ provider: 'groq', model: '' }, { provider: 'hf', model: '' }, { provider: 'local-whisper', model: '' }])
|
|
46
|
+
.description('Fallback chain for voice messages. Accuracy matters more than speed.'),
|
|
47
|
+
language: z.string().default('ru'),
|
|
48
|
+
autoSendMs: z.number().default(4000)
|
|
49
|
+
.description('Cancel window before the recognized text is sent to the agent.'),
|
|
50
|
+
}).default({}),
|
|
51
|
+
deepgramKeyEnv: z.string().default('DEEPGRAM_API_KEY'),
|
|
52
|
+
groqKeyEnv: z.string().default('GROQ_API_KEY'),
|
|
53
|
+
hfTokenEnv: z.string().default('HF_TOKEN'),
|
|
54
|
+
whisperUrl: z.string().default('http://127.0.0.1:8001/inference'),
|
|
55
|
+
whisperBin: z.string().default('whisper-server')
|
|
56
|
+
.description('whisper.cpp server binary, looked up in PATH unless an absolute path is given.'),
|
|
57
|
+
whisperModel: z.string().default('')
|
|
58
|
+
.description('Absolute path to the ggml model. Autostart stays off while this is empty; '
|
|
59
|
+
+ 'point it at your own model file to let the plugin launch whisper.cpp itself.'),
|
|
60
|
+
autoStart: z.boolean().default(true)
|
|
61
|
+
.description('Launch the local whisper.cpp server on activation if the port is free. '
|
|
62
|
+
+ 'Requires whisperModel to be set.'),
|
|
63
|
+
ffmpegBin: z.string().default('ffmpeg')
|
|
64
|
+
.description('ffmpeg used to convert browser webm/opus into the WAV that whisper.cpp requires.'),
|
|
65
|
+
timeoutMs: z.number().default(120000),
|
|
66
|
+
maxFileBytes: z.number().default(25 * 1024 * 1024),
|
|
67
|
+
})
|
|
68
|
+
|
|
69
|
+
const MIME_BY_EXT = {
|
|
70
|
+
'.wav': 'audio/wav', '.mp3': 'audio/mpeg', '.m4a': 'audio/mp4', '.mp4': 'audio/mp4',
|
|
71
|
+
'.ogg': 'audio/ogg', '.oga': 'audio/ogg', '.flac': 'audio/flac', '.webm': 'audio/webm', '.aac': 'audio/aac',
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
function writeJson(res, code, body) {
|
|
75
|
+
try {
|
|
76
|
+
res.writeHead(code, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' })
|
|
77
|
+
res.end(JSON.stringify(body))
|
|
78
|
+
} catch { /* сокет мог закрыться */ }
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
function readBody(req, maxBytes) {
|
|
82
|
+
return new Promise((resolve, reject) => {
|
|
83
|
+
const chunks = []
|
|
84
|
+
let size = 0
|
|
85
|
+
req.on('data', (c) => {
|
|
86
|
+
size += c.length
|
|
87
|
+
if (size > maxBytes) { reject(new Error('body too large')); req.destroy(); return }
|
|
88
|
+
chunks.push(c)
|
|
89
|
+
})
|
|
90
|
+
req.on('end', () => resolve(Buffer.concat(chunks)))
|
|
91
|
+
req.on('error', reject)
|
|
92
|
+
})
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
export function apply(ctx, config) {
|
|
96
|
+
let child = null
|
|
97
|
+
|
|
98
|
+
async function resolveKey(ref) {
|
|
99
|
+
try {
|
|
100
|
+
const resolved = await ctx.credentials.resolve(credentialRef(ref))
|
|
101
|
+
if (resolved && resolved.value) return resolved.value
|
|
102
|
+
} catch { /* падаем в окружение */ }
|
|
103
|
+
return process.env[ref] || ''
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
async function whisperAlive() {
|
|
107
|
+
try {
|
|
108
|
+
const controller = new AbortController()
|
|
109
|
+
const t = setTimeout(() => controller.abort(), 2000)
|
|
110
|
+
const res = await fetch(config.whisperUrl.split('/inference')[0] + '/', { signal: controller.signal })
|
|
111
|
+
clearTimeout(t)
|
|
112
|
+
return res.ok
|
|
113
|
+
} catch { return false }
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
async function startWhisper() {
|
|
117
|
+
if (!config.autoStart) return false
|
|
118
|
+
// Без пути к модели запускать нечего: пакет не знает, где она лежит у
|
|
119
|
+
// конкретного пользователя, и молча стартовать чужой бинарь не должен.
|
|
120
|
+
if (!config.whisperModel) return false
|
|
121
|
+
if (await whisperAlive()) return true
|
|
122
|
+
try {
|
|
123
|
+
const spec = ctx.shell.resolve({
|
|
124
|
+
command: `${JSON.stringify(config.whisperBin)} -m ${JSON.stringify(config.whisperModel)}`
|
|
125
|
+
+ ` --host 127.0.0.1 --port 8001 -t 8 -p 1 -l ${config.dictation.language}`,
|
|
126
|
+
timeoutMs: 0,
|
|
127
|
+
stdoutMaxBytes: 4 * 1024 * 1024,
|
|
128
|
+
})
|
|
129
|
+
child = ctx.shell.start(spec)
|
|
130
|
+
for (let i = 0; i < 20; i++) {
|
|
131
|
+
await new Promise((r) => setTimeout(r, 500))
|
|
132
|
+
if (await whisperAlive()) return true
|
|
133
|
+
}
|
|
134
|
+
return false
|
|
135
|
+
} catch { return false }
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
startWhisper().catch(() => {})
|
|
139
|
+
|
|
140
|
+
// Общий путь распознавания: собрать провайдеров по цепочке режима и пройти её.
|
|
141
|
+
async function transcribe(modeCfg, bytes, mime, signal) {
|
|
142
|
+
const models = {}
|
|
143
|
+
const order = []
|
|
144
|
+
for (const entry of Array.isArray(modeCfg.chain) ? modeCfg.chain : []) {
|
|
145
|
+
if (!PROVIDER_KEYS.includes(entry.provider)) continue
|
|
146
|
+
order.push(entry.provider)
|
|
147
|
+
models[entry.provider] = entry.model || DEFAULT_MODELS[entry.provider]
|
|
148
|
+
}
|
|
149
|
+
const providers = makeProviders(
|
|
150
|
+
{ resolveKey, fetchImpl: fetch, cfg: config, toWav: (b) => toWav16k(b, config.ffmpegBin) },
|
|
151
|
+
{ bytes, mime, lang: modeCfg.language, signal, models },
|
|
152
|
+
)
|
|
153
|
+
return runChain(order, providers)
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
ctx.effect(() => ctx.webServer.register({
|
|
157
|
+
kind: 'exact',
|
|
158
|
+
path: '/dsh-voice/status',
|
|
159
|
+
handler: async (req, res) => {
|
|
160
|
+
if (req.method !== 'GET') { writeJson(res, 405, { ok: false, error: { code: 'method', message: 'GET only' } }); return }
|
|
161
|
+
writeJson(res, 200, {
|
|
162
|
+
ok: true,
|
|
163
|
+
whisperRunning: await whisperAlive(),
|
|
164
|
+
modes: {
|
|
165
|
+
dictation: { chain: config.dictation.chain, language: config.dictation.language, vadSilenceMs: config.dictation.vadSilenceMs },
|
|
166
|
+
message: { chain: config.message.chain, language: config.message.language, autoSendMs: config.message.autoSendMs },
|
|
167
|
+
},
|
|
168
|
+
})
|
|
169
|
+
},
|
|
170
|
+
}), 'dsh-voice: /status route')
|
|
171
|
+
|
|
172
|
+
ctx.effect(() => ctx.webServer.register({
|
|
173
|
+
kind: 'exact',
|
|
174
|
+
path: '/dsh-voice/transcribe',
|
|
175
|
+
handler: async (req, res) => {
|
|
176
|
+
if (req.method !== 'POST') { writeJson(res, 405, { ok: false, error: { code: 'method', message: 'POST only' } }); return }
|
|
177
|
+
let raw
|
|
178
|
+
try {
|
|
179
|
+
raw = await readBody(req, config.maxFileBytes + 1024 * 1024)
|
|
180
|
+
} catch (e) {
|
|
181
|
+
writeJson(res, 400, { ok: false, error: { code: 'body', message: e.message } }); return
|
|
182
|
+
}
|
|
183
|
+
let payload
|
|
184
|
+
try { payload = JSON.parse(raw.toString('utf8') || '{}') } catch { payload = {} }
|
|
185
|
+
|
|
186
|
+
const dataBase64 = typeof payload.dataBase64 === 'string' ? payload.dataBase64 : ''
|
|
187
|
+
if (!dataBase64) { writeJson(res, 400, { ok: false, error: { code: 'no-audio', message: 'no audio data' } }); return }
|
|
188
|
+
const mime = typeof payload.mimeType === 'string' && payload.mimeType ? payload.mimeType : 'audio/webm'
|
|
189
|
+
const modeCfg = payload.mode === 'message' ? config.message : config.dictation
|
|
190
|
+
|
|
191
|
+
let bytes
|
|
192
|
+
try { bytes = Buffer.from(dataBase64, 'base64') } catch { bytes = null }
|
|
193
|
+
if (!bytes || bytes.length === 0) {
|
|
194
|
+
writeJson(res, 400, { ok: false, error: { code: 'decode', message: 'failed to decode audio' } }); return
|
|
195
|
+
}
|
|
196
|
+
if (bytes.length > config.maxFileBytes) {
|
|
197
|
+
writeJson(res, 413, { ok: false, error: { code: 'too-large', message: `audio is ${bytes.length} bytes, max ${config.maxFileBytes}` } }); return
|
|
198
|
+
}
|
|
199
|
+
|
|
200
|
+
// Локальный whisper в цепочке — поднимаем сервер заранее, иначе первый
|
|
201
|
+
// же чанк уйдёт в отказ, пока сервер стартует.
|
|
202
|
+
if ((modeCfg.chain || []).some((e) => e.provider === 'local-whisper') && !(await whisperAlive())) {
|
|
203
|
+
await startWhisper()
|
|
204
|
+
}
|
|
205
|
+
|
|
206
|
+
const controller = new AbortController()
|
|
207
|
+
const timer = setTimeout(() => controller.abort(), config.timeoutMs)
|
|
208
|
+
try {
|
|
209
|
+
const out = await transcribe(modeCfg, bytes, mime, controller.signal)
|
|
210
|
+
writeJson(res, 200, { ok: true, text: out.text, provider: out.provider, tookMs: out.tookMs })
|
|
211
|
+
} catch (e) {
|
|
212
|
+
writeJson(res, 502, { ok: false, error: { code: 'chain', message: String(e && e.message || e) } })
|
|
213
|
+
} finally {
|
|
214
|
+
clearTimeout(timer)
|
|
215
|
+
}
|
|
216
|
+
},
|
|
217
|
+
}), 'dsh-voice: /transcribe route')
|
|
218
|
+
|
|
219
|
+
ctx.tools.register(
|
|
220
|
+
defineTool({
|
|
221
|
+
name: 'transcribe_audio',
|
|
222
|
+
description:
|
|
223
|
+
'Recognize speech in an audio file and return the transcript as text. '
|
|
224
|
+
+ 'Uses the voice-message fallback chain from the dsh-voice settings, so a single '
|
|
225
|
+
+ 'provider outage or rate limit does not fail the request. '
|
|
226
|
+
+ 'Use for voice messages, recordings, interviews.',
|
|
227
|
+
parameters: {
|
|
228
|
+
file_path: { type: 'string', required: true, description: 'Absolute path to the audio file (wav, mp3, m4a, ogg, flac, webm).' },
|
|
229
|
+
language: { type: 'string', description: `Recognition language code. Default: ${config.message.language}.` },
|
|
230
|
+
},
|
|
231
|
+
output: {
|
|
232
|
+
schema: {
|
|
233
|
+
type: 'object',
|
|
234
|
+
additionalProperties: false,
|
|
235
|
+
properties: { provider: { type: 'string' }, text: { type: 'string' }, tookMs: { type: 'integer' } },
|
|
236
|
+
},
|
|
237
|
+
render(args, value) {
|
|
238
|
+
const body = value.text.length > 4000
|
|
239
|
+
? `${value.text.slice(0, 4000)}\n…[truncated ${value.text.length} chars]`
|
|
240
|
+
: value.text
|
|
241
|
+
return [{ type: 'text', text: `transcribe_audio (${value.provider}, ${value.tookMs}ms):\n${body}` }]
|
|
242
|
+
},
|
|
243
|
+
},
|
|
244
|
+
isConcurrencySafe: () => false,
|
|
245
|
+
timeoutMs: config.timeoutMs * 3 + 5000,
|
|
246
|
+
async execute(args, exec) {
|
|
247
|
+
const filePath = String(args.file_path || '').trim()
|
|
248
|
+
if (!filePath) throw new Error('transcribe_audio: file_path is required')
|
|
249
|
+
const info = await stat(filePath).catch(() => null)
|
|
250
|
+
if (!info) throw new Error(`transcribe_audio: file not found: ${filePath}`)
|
|
251
|
+
if (info.size > config.maxFileBytes) {
|
|
252
|
+
throw new Error(`transcribe_audio: file too large (${info.size} bytes, max ${config.maxFileBytes})`)
|
|
253
|
+
}
|
|
254
|
+
if (info.size < 100) throw new Error('transcribe_audio: file is empty or too small')
|
|
255
|
+
const mime = MIME_BY_EXT[path.extname(filePath).toLowerCase()] || 'audio/wav'
|
|
256
|
+
const bytes = await readFile(filePath)
|
|
257
|
+
const modeCfg = { ...config.message, language: String(args.language || config.message.language) }
|
|
258
|
+
return transcribe(modeCfg, bytes, mime, exec.signal)
|
|
259
|
+
},
|
|
260
|
+
}),
|
|
261
|
+
)
|
|
262
|
+
|
|
263
|
+
ctx.effect(() => () => {
|
|
264
|
+
if (child) { try { child.kill && child.kill() } catch { /* уже мёртв */ } }
|
|
265
|
+
}, 'dsh-voice: stop whisper child')
|
|
266
|
+
}
|
package/lib/providers.js
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
// Четыре провайдера распознавания речи. Чистые функции: сеть приходит
|
|
2
|
+
// параметром (fetchImpl), ключи — через resolveKey, поэтому всё проверяется
|
|
3
|
+
// без реальных запросов.
|
|
4
|
+
|
|
5
|
+
export const PROVIDER_KEYS = ['deepgram', 'groq', 'hf', 'local-whisper']
|
|
6
|
+
|
|
7
|
+
export const DEFAULT_MODELS = {
|
|
8
|
+
deepgram: 'nova-2',
|
|
9
|
+
groq: 'whisper-large-v3-turbo',
|
|
10
|
+
hf: 'openai/whisper-large-v3',
|
|
11
|
+
'local-whisper': '',
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
function pickModel(models, key) {
|
|
15
|
+
const chosen = models && typeof models[key] === 'string' ? models[key].trim() : ''
|
|
16
|
+
return chosen || DEFAULT_MODELS[key]
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
function fileName(mime) {
|
|
20
|
+
if (mime.includes('wav')) return 'audio.wav'
|
|
21
|
+
if (mime.includes('ogg')) return 'audio.ogg'
|
|
22
|
+
if (mime.includes('mp4')) return 'audio.m4a'
|
|
23
|
+
return 'audio.webm'
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
export function makeProviders(deps, req) {
|
|
27
|
+
const { resolveKey, fetchImpl, cfg } = deps
|
|
28
|
+
const { bytes, mime, lang, signal, models } = req
|
|
29
|
+
|
|
30
|
+
async function deepgram() {
|
|
31
|
+
const key = await resolveKey(cfg.deepgramKeyEnv)
|
|
32
|
+
if (!key) return { ok: false, provider: 'deepgram', reason: `no ${cfg.deepgramKeyEnv}` }
|
|
33
|
+
const model = pickModel(models, 'deepgram')
|
|
34
|
+
const url = `https://api.deepgram.com/v1/listen?model=${encodeURIComponent(model)}`
|
|
35
|
+
+ `&language=${encodeURIComponent(lang)}&smart_format=true`
|
|
36
|
+
const res = await fetchImpl(url, {
|
|
37
|
+
method: 'POST',
|
|
38
|
+
headers: { authorization: `Token ${key}`, 'content-type': mime },
|
|
39
|
+
body: bytes,
|
|
40
|
+
signal,
|
|
41
|
+
})
|
|
42
|
+
if (!res.ok) throw new Error(`Deepgram HTTP ${res.status}`)
|
|
43
|
+
const data = await res.json()
|
|
44
|
+
const text = (data?.results?.channels?.[0]?.alternatives?.[0]?.transcript || '').trim()
|
|
45
|
+
return { ok: text.length > 0, provider: 'deepgram', text, reason: text ? '' : 'empty transcript' }
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
async function groq() {
|
|
49
|
+
const key = await resolveKey(cfg.groqKeyEnv)
|
|
50
|
+
if (!key) return { ok: false, provider: 'groq', reason: `no ${cfg.groqKeyEnv}` }
|
|
51
|
+
const form = new FormData()
|
|
52
|
+
form.append('file', new Blob([bytes], { type: mime }), fileName(mime))
|
|
53
|
+
form.append('model', pickModel(models, 'groq'))
|
|
54
|
+
form.append('language', lang)
|
|
55
|
+
form.append('response_format', 'json')
|
|
56
|
+
const res = await fetchImpl('https://api.groq.com/openai/v1/audio/transcriptions', {
|
|
57
|
+
method: 'POST',
|
|
58
|
+
headers: { authorization: `Bearer ${key}` },
|
|
59
|
+
body: form,
|
|
60
|
+
signal,
|
|
61
|
+
})
|
|
62
|
+
if (!res.ok) throw new Error(`Groq HTTP ${res.status}`)
|
|
63
|
+
const data = await res.json()
|
|
64
|
+
const text = (data?.text || '').trim()
|
|
65
|
+
return { ok: text.length > 0, provider: 'groq', text, reason: text ? '' : 'empty transcript' }
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
async function hf() {
|
|
69
|
+
const token = await resolveKey(cfg.hfTokenEnv)
|
|
70
|
+
if (!token) return { ok: false, provider: 'hf', reason: `no ${cfg.hfTokenEnv}` }
|
|
71
|
+
const model = pickModel(models, 'hf')
|
|
72
|
+
const res = await fetchImpl(`https://router.huggingface.co/hf-inference/models/${model}`, {
|
|
73
|
+
method: 'POST',
|
|
74
|
+
headers: { authorization: `Bearer ${token}`, 'content-type': mime, 'x-wait-for-model': 'true' },
|
|
75
|
+
body: bytes,
|
|
76
|
+
signal,
|
|
77
|
+
})
|
|
78
|
+
if (!res.ok) throw new Error(`HF HTTP ${res.status}`)
|
|
79
|
+
const data = await res.json()
|
|
80
|
+
const text = (data?.text || '').trim()
|
|
81
|
+
return { ok: text.length > 0, provider: 'hf', text, reason: text ? '' : 'empty transcript' }
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
// whisper.cpp принимает только WAV — на webm/opus из браузера его сервер
|
|
85
|
+
// отвечает "Invalid request". Перегоняем, если формат не WAV.
|
|
86
|
+
async function localWhisper() {
|
|
87
|
+
let sendBytes = bytes
|
|
88
|
+
let sendMime = mime
|
|
89
|
+
if (!mime.includes('wav')) {
|
|
90
|
+
if (typeof deps.toWav !== 'function') {
|
|
91
|
+
return { ok: false, provider: 'local-whisper', reason: 'local whisper needs WAV, no converter configured' }
|
|
92
|
+
}
|
|
93
|
+
try {
|
|
94
|
+
sendBytes = await deps.toWav(bytes)
|
|
95
|
+
sendMime = 'audio/wav'
|
|
96
|
+
} catch (e) {
|
|
97
|
+
return { ok: false, provider: 'local-whisper', reason: `local whisper: ${String(e && e.message || e)}` }
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
const form = new FormData()
|
|
101
|
+
form.append('file', new Blob([sendBytes], { type: sendMime }), fileName(sendMime))
|
|
102
|
+
form.append('language', lang)
|
|
103
|
+
form.append('response_format', 'json')
|
|
104
|
+
const res = await fetchImpl(cfg.whisperUrl, { method: 'POST', body: form, signal })
|
|
105
|
+
// whisper.cpp отвечает 400 с JSON-телом на внутренних сбоях (например, не
|
|
106
|
+
// смог декодировать аудио) — читаем причину, а не бросаем исключение.
|
|
107
|
+
if (!res.ok) {
|
|
108
|
+
let detail = `HTTP ${res.status}`
|
|
109
|
+
try { const e = await res.json(); if (e?.error) detail = e.error } catch { /* тело не json */ }
|
|
110
|
+
return { ok: false, provider: 'local-whisper', reason: `local whisper: ${detail}` }
|
|
111
|
+
}
|
|
112
|
+
const data = await res.json()
|
|
113
|
+
const text = (data?.text || '').trim()
|
|
114
|
+
return { ok: text.length > 0, provider: 'local-whisper', text, reason: text ? '' : 'empty transcript' }
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
return { deepgram, groq, hf, 'local-whisper': localWhisper }
|
|
118
|
+
}
|
package/lib/wav.js
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
// Перегон произвольного аудио в WAV 16 кГц моно.
|
|
2
|
+
//
|
|
3
|
+
// Локальный whisper.cpp принимает только WAV: на webm/opus, который пишет
|
|
4
|
+
// браузер, его сервер отвечает "Invalid request". API-провайдеры webm едят
|
|
5
|
+
// как есть, поэтому конвертация нужна ровно для локальной ноги цепочки.
|
|
6
|
+
|
|
7
|
+
import { spawn } from 'node:child_process'
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* @param bytes {Buffer|Uint8Array} исходное аудио в любом контейнере
|
|
11
|
+
* @param ffmpegBin {string} путь к ffmpeg
|
|
12
|
+
* @returns {Promise<Buffer>} WAV 16 кГц моно
|
|
13
|
+
*/
|
|
14
|
+
export function toWav16k(bytes, ffmpegBin = 'ffmpeg') {
|
|
15
|
+
return new Promise((resolve, reject) => {
|
|
16
|
+
const proc = spawn(ffmpegBin, [
|
|
17
|
+
'-hide_banner', '-loglevel', 'error',
|
|
18
|
+
'-i', 'pipe:0',
|
|
19
|
+
'-ar', '16000', '-ac', '1',
|
|
20
|
+
'-f', 'wav', 'pipe:1',
|
|
21
|
+
])
|
|
22
|
+
const out = []
|
|
23
|
+
const err = []
|
|
24
|
+
proc.stdout.on('data', (c) => out.push(c))
|
|
25
|
+
proc.stderr.on('data', (c) => err.push(c))
|
|
26
|
+
proc.on('error', (e) => reject(new Error(`ffmpeg unavailable: ${e.message}`)))
|
|
27
|
+
proc.on('close', (code) => {
|
|
28
|
+
if (code !== 0) {
|
|
29
|
+
reject(new Error(`ffmpeg exit ${code}: ${Buffer.concat(err).toString('utf8').slice(0, 200)}`))
|
|
30
|
+
return
|
|
31
|
+
}
|
|
32
|
+
const wav = Buffer.concat(out)
|
|
33
|
+
if (wav.length < 64) { reject(new Error('ffmpeg produced empty output')); return }
|
|
34
|
+
resolve(wav)
|
|
35
|
+
})
|
|
36
|
+
proc.stdin.on('error', () => { /* ffmpeg закрыл вход раньше — код возврата всё расскажет */ })
|
|
37
|
+
proc.stdin.end(Buffer.from(bytes))
|
|
38
|
+
})
|
|
39
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@goodandready/dsh-voice",
|
|
3
|
+
"version": "0.3.0",
|
|
4
|
+
"description": "Voice input for DeepSeek Harness: dictation chunked by pauses and voice messages, each with its own provider fallback chain (Deepgram, Groq, HuggingFace, local whisper.cpp).",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"main": "./lib/index.js",
|
|
8
|
+
"exports": {
|
|
9
|
+
".": "./lib/index.js",
|
|
10
|
+
"./client": "./lib/client.js",
|
|
11
|
+
"./package.json": "./package.json",
|
|
12
|
+
"./cordis.patch.yml": "./cordis.patch.yml"
|
|
13
|
+
},
|
|
14
|
+
"files": [
|
|
15
|
+
"lib/",
|
|
16
|
+
"cordis.patch.yml",
|
|
17
|
+
"README.md",
|
|
18
|
+
"LICENSE"
|
|
19
|
+
],
|
|
20
|
+
"keywords": [
|
|
21
|
+
"dsh",
|
|
22
|
+
"dsh-plugin",
|
|
23
|
+
"deepseek-harness",
|
|
24
|
+
"voice",
|
|
25
|
+
"stt",
|
|
26
|
+
"dictation",
|
|
27
|
+
"whisper",
|
|
28
|
+
"deepgram",
|
|
29
|
+
"groq"
|
|
30
|
+
],
|
|
31
|
+
"repository": {
|
|
32
|
+
"type": "git",
|
|
33
|
+
"url": "https://github.com/GooDAnDReaDY/dsh-voice.git"
|
|
34
|
+
},
|
|
35
|
+
"homepage": "https://github.com/GooDAnDReaDY/dsh-voice",
|
|
36
|
+
"bugs": {
|
|
37
|
+
"url": "https://github.com/GooDAnDReaDY/dsh-voice/issues"
|
|
38
|
+
},
|
|
39
|
+
"scripts": {
|
|
40
|
+
"test": "node --test test/*.test.mjs"
|
|
41
|
+
},
|
|
42
|
+
"dsh": {
|
|
43
|
+
"bundle": {
|
|
44
|
+
"patch": "./cordis.patch.yml"
|
|
45
|
+
},
|
|
46
|
+
"client": {
|
|
47
|
+
"platform": "web",
|
|
48
|
+
"inject": [
|
|
49
|
+
"@deepseek-ai/dsh-client-runtime",
|
|
50
|
+
"@deepseek-ai/dsh-client-ui-slots"
|
|
51
|
+
]
|
|
52
|
+
}
|
|
53
|
+
},
|
|
54
|
+
"peerDependencies": {
|
|
55
|
+
"@deepseek-ai/cordis": "^4.0.1",
|
|
56
|
+
"@deepseek-ai/dsh-tools": "^0.1.0-rc.6",
|
|
57
|
+
"@deepseek-ai/dsh-credentials": "^0.1.0-rc.6",
|
|
58
|
+
"@deepseek-ai/dsh-host-webserver": "^0.1.0-rc.6",
|
|
59
|
+
"@deepseek-ai/schemastery": "^3.18.1"
|
|
60
|
+
}
|
|
61
|
+
}
|