dsh-voice-mode 0.7.7 → 0.7.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +158 -48
- package/README.md +129 -34
- package/assets/architecture.svg +1 -79
- package/assets/demo.gif +0 -0
- package/lib/client.js +709 -129
- package/lib/index.js +228 -45
- package/lib/sense-worker.mjs +2 -2
- package/lib/tts-vits-worker.cjs +1 -1
- package/package.json +3 -3
package/README.en.md
CHANGED
|
@@ -1,19 +1,24 @@
|
|
|
1
1
|
# dsh-voice-mode
|
|
2
2
|
|
|
3
|
-
[](../../LICENSE)
|
|
4
|
+
[](https://github.com/qishuilalala/dsh-voice-mode/releases)
|
|
5
|
+
[](https://www.npmjs.com/package/dsh-voice-mode)
|
|
6
|
+
[](../../docs/rules/STATE.md)
|
|
7
|
+
|
|
8
|
+
Full-duplex voice conversation mode for DeepSeek Harness (dsh): speak, get a
|
|
9
|
+
spoken answer. Streamed zipformer2 ASR → editable draft → auto send → the
|
|
10
|
+
final reply is read out sentence-by-sentence via Edge TTS, and your voice
|
|
11
|
+
interrupts playback and the running turn. No API key.
|
|
12
|
+
|
|
12
13
|
> 中文说明见 [README.md](./README.md)。
|
|
13
14
|
|
|
14
|
-
|
|
15
|
+

|
|
16
|
+
|
|
17
|
+
> **Version note (v0.7.9, 2026-09-18)**: **Wake-word pipeline overhaul** (Issue #10 + real-machine retest) — a harness driving the real engine pinpointed and fixed five flow defects: ① TTS echo polluting the standby segment while the agent reads (could not wake); ② only above-threshold frames uploaded, so trailing characters never flushed (wake word truncated); ③ discarding the whole segment on a wake hit (saying "wake word + command" in one breath sent only the tail); ④ the command hanging when the hit arrived after you stopped speaking; ⑤ pausing after the wake word closed the command window (command lost). **The wake word may now be said together with your command — it is stripped and never sent**, standby live-shows what it heard, and "wake word … pause … command" works. Otherwise as v0.7.7: Edge cloud TTS by default, local TTS (VITS / Kokoro) optional; silence split defaults to 1500 ms.
|
|
15
18
|
|
|
16
|
-
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## 🤝 Fork enhancements (this repo)
|
|
17
22
|
|
|
18
23
|
- **Edge cloud TTS by default; local TTS optional (privacy-first)**: local VITS (Chinese) and local Kokoro (zh+en, 103 voices; `int8` default ~109 MB / `fp32` ~311 MB for better quality; native `sherpa-onnx-node` addon, no WASM memory limits) run in an isolated child process;
|
|
19
24
|
- **103 Kokoro voices** (F0-measured gender labels, 4 favourite male voices pinned), browsed with a ◀▶ stepper;
|
|
@@ -22,24 +27,29 @@
|
|
|
22
27
|
- **Long segments**: continuous listening stitches consecutive segments into one message (internally chunked at 30 s and concatenated across chunks); 1500 ms silence split by default; hold keeps pauses from splitting;
|
|
23
28
|
- **Hardening**: session-existence check, loopback + Origin guards, per-endpoint rate limits, model SHA256 pinning, download-host allowlist.
|
|
24
29
|
|
|
25
|
-
> ⚠️ The screenshots
|
|
30
|
+
> ⚠️ The screenshots above (and `assets/demo.gif`) show the **upstream legacy single-button UI**; the current UI adds a mode-switch button next to the mic.
|
|
26
31
|
|
|
27
|
-
|
|
32
|
+
---
|
|
28
33
|
|
|
34
|
+
## ✨ Features
|
|
29
35
|
|
|
30
36
|
- **Voice mode**: toggle with the microphone button in the input toolbar or the global shortcut `Ctrl+Shift+V`; globally single-active (only one session is in voice mode at a time; switching sessions yields automatically)
|
|
31
37
|
- **Two interaction modes (switchable in settings, plus a mode-switch button beside the mic)**:
|
|
32
38
|
- `toggle` (default) continuous listening: RMS VAD segmentation → streaming zipformer2 ASR (words appear as you speak, live caption preview) → automatic sentence split after 1500 ms of silence into the draft, consecutive segments joined into one message, then auto-sent after ~1500 ms more of silence (≈3 s total); hold `Ctrl` to force an immediate send
|
|
33
39
|
- `hold` push-to-talk: short tap to enter/exit, **hold the mic button to talk, release to send** (swipe up to cancel, `Esc`/blur abandons the segment; pauses do not split while held, up to 10 min); hold `Ctrl` to record-by-keyboard, release to send
|
|
34
|
-
- **Wake word (optional, off by default)**: after setting `wakeWord`, entering voice mode starts in standby, and recognition only begins once the wake word is spoken (e.g. `你好小D`),
|
|
40
|
+
- **Wake word (optional, off by default)**: after setting `wakeWord`, entering voice mode starts in standby, and recognition only begins once the wake word is spoken (e.g. `你好小D`). **You may say it together with your command** ("wake word, check the weather") — the wake word is stripped from the transcript/caption and never sent; matching is fault-tolerant (homophone substitutions, a wrong first char, leading fillers); 3-4 characters recommended; in standby the status bar live-shows "say '<word>' · what it heard". Boundaries (important): **toggle mode only** (inactive with `hold` / manual barge-in); **it must be repeated after each completed utterance or barge-in** (if you only said the wake word, the engine keeps listening so you may pause before the command); **saying it while the agent is reading does not trigger** (barge-in is VAD-based and unrelated to the wake word)
|
|
41
|
+
- **Caption tiers (batch 3)**: `captionFontSize` 4 levels (0=12px / 1=14px / 2=18px / 3=24px) + `captionMaxWidth` 3 levels (0=50vw / 1=70vw / 2=90vw); adapts to narrow viewports
|
|
42
|
+
- **Yield semantics (batch 5 / ADR-0008)**: `backchannelYield` on by default (I10-exempt) — saying `嗯 / 对 / right` while the AI is reading auto-pauses for 1.5 s; genuine speech still triggers hard barge-in; lets the LLM yield the turn
|
|
35
43
|
- **Output pipeline**: only the final answer's `text-delta` is read (reasoning/tool calls are skipped), streamed sentence-by-sentence (Edge cloud by default; local VITS / Kokoro, int8/fp32, optional) with a live caption overlay at the bottom-right; tool calls trigger a beep; the full text is still written to the chat; in voice mode a spoken-format system prompt is injected (short natural sentences, no Markdown decoration), and the reader side strips markers as well for a smoother listening experience
|
|
36
|
-
- **Barge-in**: three sensitivity levels of voice-onset detection → local mute + host synth queue invalidation (epoch) + running turn cancellation (the half-finished part is kept and naturally flows into your new message)
|
|
44
|
+
- **Barge-in**: three sensitivity levels of voice-onset detection → local mute + host synth queue invalidation (epoch) + running turn cancellation (the half-finished part is kept and naturally flows into your new message). With a wake word set, barge-in **remains speak-to-interrupt** (the gate is the VAD, not the wake word; the engine returns to standby afterwards)
|
|
37
45
|
- **Lazy model download with progress**: the zipformer2 Chinese streaming model (~160 MB, `.part` resumable) is downloaded on first use with live progress in the status bar; `npm run prefetch` can pre-download it
|
|
38
46
|
- **Resilience**: mic-denied red hint, visible model-download failure, TTS unreachable status hint (auto retry), failed submit keeps the text in the draft, SSE auto-reconnect
|
|
39
|
-
- **Settings**: Settings → Plugins → voice-mode, with voice / rate / interrupt sensitivity / silence pause / idle timeout / model mirror / auto send / interaction mode / wake word; **voices are previewable** (the "试听/Preview" button synthesizes and plays the current voice at the current rate instantly, no need to enter voice mode; custom ShortNames are previewable too)
|
|
40
|
-
- **Idle exit**: auto-exit and mic release after
|
|
47
|
+
- **Settings**: Settings → Plugins → voice-mode, with voice / rate / interrupt sensitivity / silence pause / idle timeout / model mirror / auto send / interaction mode / wake word / ITN / caption tiers / yield semantics; **voices are previewable** (the "试听/Preview" button synthesizes and plays the current voice at the current rate instantly, no need to enter voice mode; custom ShortNames are previewable too)
|
|
48
|
+
- **Idle exit**: auto-exit and mic release after 5 minutes of inactivity (reading counts as activity; batch J 10→5)
|
|
49
|
+
|
|
50
|
+
---
|
|
41
51
|
|
|
42
|
-
## Interaction gestures
|
|
52
|
+
## ⌨️ Interaction gestures
|
|
43
53
|
|
|
44
54
|
| Gesture | Behaviour |
|
|
45
55
|
| --- | --- |
|
|
@@ -52,7 +62,9 @@
|
|
|
52
62
|
| Speak while AI is reading | Interrupt playback and cancel the running turn |
|
|
53
63
|
| Type in the input box | Auto-exit voice mode (draft is kept) |
|
|
54
64
|
|
|
55
|
-
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## 🚀 Quick Start (5 minutes)
|
|
56
68
|
|
|
57
69
|
**Requirements**: dsh web (Node ≥ 18), a modern browser (Chrome / Edge / Firefox, supporting `getUserMedia` and Web Audio).
|
|
58
70
|
|
|
@@ -83,7 +95,7 @@ npm run prefetch # run inside the plugin dir; writes to the platform ca
|
|
|
83
95
|
# or specify the cache location: node scripts/prefetch.mjs --cache-dir /where/ever/models
|
|
84
96
|
```
|
|
85
97
|
|
|
86
|
-
|
|
98
|
+
**First run**:
|
|
87
99
|
|
|
88
100
|
1. Click the mic button in the input toolbar (or press `Ctrl+Shift+V`) to enter voice mode; a status bar appears above the input box
|
|
89
101
|
2. Choose how to speak: just talk and let the ~1500 ms pause split and ~3 s of quiet auto-send (toggle); or hold the mic button and release to send (hold)
|
|
@@ -94,21 +106,49 @@ On first entry the recognition model is downloaded; the status bar shows `正在
|
|
|
94
106
|
|
|
95
107
|
If a wake word is configured, you land in standby first (the status bar prompts `说『唤醒词』开始`), and recognizing starts after you speak the wake word.
|
|
96
108
|
|
|
97
|
-
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## ⚙️ Settings (Settings → Plugins → Plugins config → 语音模式)
|
|
112
|
+
|
|
113
|
+
### 4 new settings (11 batches of comprehensive fixes)
|
|
114
|
+
|
|
115
|
+
| Key | Default | Description |
|
|
116
|
+
| --- | --- | --- |
|
|
117
|
+
| `senseITN` | `true` | Batch 2 P0: SenseVoice inverse text normalization (number / date / currency; on by default) |
|
|
118
|
+
| `captionFontSize` | `0` | Batch 3 P0: caption font tier 0=12px / 1=14px / 2=18px / 3=24px (default 0 is byte-equivalent to legacy) |
|
|
119
|
+
| `captionMaxWidth` | `1` | Batch 3 P0: caption width tier 0=50vw / 1=70vw / 2=90vw |
|
|
120
|
+
| `backchannelYield` | `true` | Batch 5 P1: yield semantics (ADR-0008); saying `嗯 / 对` while reading auto-pauses for 1.5 s; genuine speech still triggers hard barge-in. I10-exempt (default-on is a product decision); off = behavior identical to pre-change |
|
|
121
|
+
|
|
122
|
+
### 5 defaults micro-adjusted (batch J)
|
|
123
|
+
|
|
124
|
+
| Key | Old | New | Why |
|
|
125
|
+
| --- | --- | --- | --- |
|
|
126
|
+
| `rate` | 1.0 | **1.1** | Edge TTS defaults slightly slow; +10% improves perceived quality |
|
|
127
|
+
| `idleTimeoutMinutes` | 10 | **5** | More responsive idle exit (reading still counts as activity) |
|
|
128
|
+
| `interruptLevel` description | old wording | new wording | Make "3/2/1 frame confirmation" explicit |
|
|
129
|
+
|
|
130
|
+
> Field names unchanged → 100% backward compatible with existing `~/.dsh/settings.yaml`.
|
|
131
|
+
|
|
132
|
+
### Full 19-key settings table
|
|
98
133
|
|
|
99
134
|
| Key | Default | Description |
|
|
100
135
|
| --- | --- | --- |
|
|
101
136
|
| `ttsEngine` | `edge` | Read-aloud engine: `edge` Microsoft cloud (default, fast) / `vits` local Chinese / `kokoro` local zh+en; **applies live** |
|
|
102
137
|
| `kokoroModel` | `int8` | Kokoro precision: `int8` (default, 109 MB, CPU/low-bandwidth) / `fp32` (311 MB, better quality, GPU/large memory); same 103 voices; **applies live** |
|
|
103
138
|
| `voice` | per engine | Voice: 5 VITS speakers; 103 Kokoro voices (◀▶ stepper; 62/68/75/76 favourite males pinned); Edge ShortNames below. The inline "试听" button previews it at the current rate |
|
|
104
|
-
| `rate` | `1.
|
|
105
|
-
| `interruptLevel` | `0` | Barge-in sensitivity (host-side VAD frame detection + echo gate): 0 high threshold / 1 medium / 2 low |
|
|
139
|
+
| `rate` | `1.1` | Reading speed multiplier (0.5 slow ~ 2.0 fast), **applies live** (batch J 1.0→1.1) |
|
|
140
|
+
| `interruptLevel` | `0` | Barge-in sensitivity (host-side VAD frame detection + echo gate): 0 high threshold (3 frames) / 1 medium (2 frames) / 2 low (1 frame) |
|
|
106
141
|
| `silenceMs` | `1500` | Silence pause in ms that marks the end of a complete sentence |
|
|
107
|
-
| `idleTimeoutMinutes` | `
|
|
142
|
+
| `idleTimeoutMinutes` | `5` | Minutes of inactivity before auto-exiting voice mode (reading counts as activity; batch J 10→5) |
|
|
108
143
|
| `modelHost` | default | Model download host (use `https://hf-mirror.com` on mainland networks) |
|
|
109
144
|
| `autoSend` | `true` | Auto-send once quiet (consecutive segments join into one message); when off, text only goes to the draft (hold `Ctrl` / release in hold mode still sends) |
|
|
110
145
|
| `mode` | `toggle` | Interaction mode: `toggle` continuous listening + 1500 ms silence split; `hold` push-to-talk, release to send (short tap exits) |
|
|
111
|
-
| `wakeWord` | empty (off) | Wake word (e.g. `你好小D`): speak it after entering to activate
|
|
146
|
+
| `wakeWord` | empty (off) | Wake word (e.g. `你好小D`): speak it after entering to activate; empty = off. **May be said together with your command** — the wake word is stripped and never sent; fault-tolerant matching (edit distance ≤1 + a 3-char leading window absorbs homophones/fillers); **3-4 characters recommended** (single-char words get no tolerance; a 2-char word also absorbs any same-first-char 2-char word); must be repeated after each utterance split or barge-in; **toggle mode only**; not triggered while the agent is reading |
|
|
147
|
+
| `spokenFormat` | `true` | Spoken-format system prompt: inject "short natural sentences, no Markdown decoration" into voice-mode replies only, **applies live** |
|
|
148
|
+
| `senseITN` | `true` | Batch 2 P0: SenseVoice inverse text normalization (number / date / currency; on by default) |
|
|
149
|
+
| `captionFontSize` | `0` | Batch 3 P0: caption font tier 0=12px / 1=14px / 2=18px / 3=24px (default 0 is byte-equivalent to legacy) |
|
|
150
|
+
| `captionMaxWidth` | `1` | Batch 3 P0: caption width tier 0=50vw / 1=70vw / 2=90vw |
|
|
151
|
+
| `backchannelYield` | `true` | Batch 5 P1: yield semantics (ADR-0008); saying `嗯 / 对` while reading auto-pauses for 1.5 s; genuine speech still triggers hard barge-in. I10-exempt (default-on is a product decision); off = behavior identical to pre-change |
|
|
112
152
|
|
|
113
153
|
Effect timing: `voice`/`rate`/`ttsEngine`/`kokoroModel`/`spokenFormat` take effect **immediately** (TTS hot-swap); the rest apply on the next voice-mode entry. Defaults come from the plugin config (`base` layer) — they follow the config unless explicitly changed.
|
|
114
154
|
|
|
@@ -131,7 +171,9 @@ Effect timing: `voice`/`rate`/`ttsEngine`/`kokoroModel`/`spokenFormat` take effe
|
|
|
131
171
|
| `en-US-AriaNeural` | Aria · English · Female |
|
|
132
172
|
| `en-US-GuyNeural` | Guy · English · Male |
|
|
133
173
|
|
|
134
|
-
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## 🔧 Configuration (bundle patch / settings.yaml)
|
|
135
177
|
|
|
136
178
|
You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (the GUI card and RPC write to the same document layer):
|
|
137
179
|
|
|
@@ -143,10 +185,10 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
|
|
|
143
185
|
cacheDir: ~/.cache/dsh-voice-mode/models # overridable; platform default otherwise
|
|
144
186
|
# Defaults seeded for the settings (the settings panel overrides; the panel is authoritative):
|
|
145
187
|
voice: zh-CN-XiaoxiaoNeural
|
|
146
|
-
rate: 1.0
|
|
188
|
+
rate: 1.1 # batch J 1.0→1.1
|
|
147
189
|
interruptLevel: 0
|
|
148
190
|
silenceMs: 1500
|
|
149
|
-
idleTimeoutMinutes: 10
|
|
191
|
+
idleTimeoutMinutes: 5 # batch J 10→5
|
|
150
192
|
modelHost: https://huggingface.co
|
|
151
193
|
```
|
|
152
194
|
|
|
@@ -155,7 +197,9 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
|
|
|
155
197
|
> (`enabled/cacheDir` remain bundle-config-only).
|
|
156
198
|
> The plugin HTTP namespace is fixed to `/voice-mode` (matching the client bundle contract; not configurable).
|
|
157
199
|
|
|
158
|
-
|
|
200
|
+
---
|
|
201
|
+
|
|
202
|
+
## 🌐 API
|
|
159
203
|
|
|
160
204
|
| Route | Description |
|
|
161
205
|
| --- | --- |
|
|
@@ -164,10 +208,12 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
|
|
|
164
208
|
| `POST /voice-mode/asr` | Raw f32 LE 16k PCM payload → `{text}` (streaming zipformer2); returns `202 {loading}` until the model is ready; `?reset=1` discards the in-flight segment (used on wake-word hit) |
|
|
165
209
|
| `POST /voice-mode/cancel` | `{sessionId}` invalidates the TTS queue and drops the in-flight ASR segment |
|
|
166
210
|
| `POST /voice-mode/preview` | `{voice, rate?}` one-shot synthesis preview → `audio/mpeg` (400 missing voice / voice too long; 502 synthesis failure, e.g. invalid ShortName; 403 when the plugin's `enabled=false`). Does not require voice mode to be active; uses an isolated synthesis connection and does not affect the reading queue |
|
|
167
|
-
| `GET /voice-mode/config` | Client bootstrap parameters (silence threshold / sensitivity / voice and rate, etc.) |
|
|
211
|
+
| `GET /voice-mode/config` | Client bootstrap parameters (silence threshold / sensitivity / voice and rate, etc.) — includes 4 new ASR fields: `senseITN` / `senseVoice` / `captionFontSize` / `captionMaxWidth` / `backchannelYield` |
|
|
168
212
|
| `GET /voice-mode` | Health check `{ok, name, enabled, active}` |
|
|
169
213
|
|
|
170
|
-
|
|
214
|
+
---
|
|
215
|
+
|
|
216
|
+
## 💾 Model & cache
|
|
171
217
|
|
|
172
218
|
- Recognition model: `csukuangfj/sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30` (encoder ≈154 MB / decoder / joiner / tokens, ~160 MB total), running host-side via sherpa-onnx (Node WASM, Apache-2.0, natively cross-platform)
|
|
173
219
|
- Cache directory defaults by platform:
|
|
@@ -176,18 +222,43 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
|
|
|
176
222
|
- both overridable via `cacheDir`
|
|
177
223
|
- Downloads use `.part` resume; `huggingface.co` falls back to `hf-mirror.com` on failure (configurable via `modelHost`)
|
|
178
224
|
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
```
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
225
|
+
---
|
|
226
|
+
|
|
227
|
+
## 🏛️ How it works
|
|
228
|
+
|
|
229
|
+
```mermaid
|
|
230
|
+
flowchart LR
|
|
231
|
+
subgraph Client[Browser Client]
|
|
232
|
+
Mic[Microphone 16kHz<br/>AudioWorklet<br/>echoCancellation:true] --> VAD[Client-side VAD<br/>RMS segmentation]
|
|
233
|
+
VAD -->|partial 0.9s| UI[Statusbar + caption overlay]
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
subgraph Host[dsh.host]
|
|
237
|
+
ASR[zipformer2 streaming ASR<br/>host-side WASM]
|
|
238
|
+
SV[SenseVoice finalization<br/>+ ITN + punctuation]
|
|
239
|
+
Tap[llm/stream tap<br/>observe-only]
|
|
240
|
+
Seg[sentence segmenter]
|
|
241
|
+
Q[TtsQueue<br/>epoch barge-in]
|
|
242
|
+
TTS{Engine}
|
|
243
|
+
Edge[Edge cloud]
|
|
244
|
+
Vits[Local VITS WASM]
|
|
245
|
+
Kokoro[Local Kokoro<br/>native addon]
|
|
246
|
+
end
|
|
247
|
+
|
|
248
|
+
UI -->|audio f32 PCM<br/>POST /voice-mode/asr| ASR
|
|
249
|
+
ASR --> SV
|
|
250
|
+
SV --> Draft[composer draft<br/>autoSend]
|
|
251
|
+
Draft --> Tap
|
|
252
|
+
Tap --> Seg
|
|
253
|
+
Seg --> Q
|
|
254
|
+
Q --> TTS
|
|
255
|
+
TTS -->|edge| Edge
|
|
256
|
+
TTS -->|vits| Vits
|
|
257
|
+
TTS -->|kokoro| Kokoro
|
|
258
|
+
Edge -.->|SSE audio frame| UI
|
|
259
|
+
Vits -.->|SSE audio frame| UI
|
|
260
|
+
Kokoro -.->|SSE audio frame| UI
|
|
261
|
+
VAD -.->|wake-word / barge-in| Q
|
|
191
262
|
```
|
|
192
263
|
|
|
193
264
|
- Speech and reading only happen for the session pointed to by the global single-active pointer `activeVoiceSession`; other sessions pass through `llm/stream` with zero overhead (mode isolation)
|
|
@@ -196,7 +267,24 @@ input: mic ──RMS VAD (1500 ms silence split)──▶ POST /voice-mode/asr
|
|
|
196
267
|
- Local TTS (VITS / native Kokoro) runs in an isolated child process (fork, auto-restart); barge-in kills the in-flight synthesis instantly to free CPU
|
|
197
268
|
- The TTS queue is per-session with an epoch version: old frames are all invalidated after a barge-in, so it is truly silent
|
|
198
269
|
|
|
199
|
-
|
|
270
|
+
---
|
|
271
|
+
|
|
272
|
+
## 🔍 Comparison with dsh built-in voice mode
|
|
273
|
+
|
|
274
|
+
| Dimension | dsh built-in | dsh-voice-mode (this plugin) |
|
|
275
|
+
| --- | --- | --- |
|
|
276
|
+
| Recognition model | Cloud API (needs key) | **Local zipformer2 + SenseVoice** (zero key) |
|
|
277
|
+
| Multi-language | English-first | **SenseVoice auto-recognition + ITN** |
|
|
278
|
+
| TTS engine | Cloud TTS | **Edge cloud + local VITS/Kokoro** (three-way switch) |
|
|
279
|
+
| Barge-in detection | Basic VAD | **3 sensitivity levels + echo gate + yield semantics** |
|
|
280
|
+
| Hotword biasing | None | None (removed in v0.7.7; see version note) |
|
|
281
|
+
| Caption a11y | None | **4 font tiers + 3 width tiers + theme-following** |
|
|
282
|
+
| Wake word | None | **Lightweight streaming match + prefix filler whitelist** |
|
|
283
|
+
| dsh compatibility | — | **0.1.1-rc.2 → 0.1.5-rc.2 full range** |
|
|
284
|
+
|
|
285
|
+
---
|
|
286
|
+
|
|
287
|
+
## 🚧 Known limitations
|
|
200
288
|
|
|
201
289
|
- Barge-in relies on browser echo cancellation (`echoCancellation`); loud speaker volume may leak into the mic (no JS-level AEC)
|
|
202
290
|
- `Ctrl+Shift+V` overrides the browser's "paste as plain text" shortcut (normal `Ctrl+V` paste still works)
|
|
@@ -207,7 +295,9 @@ input: mic ──RMS VAD (1500 ms silence split)──▶ POST /voice-mode/asr
|
|
|
207
295
|
- The hero (new-session empty state) has no voice entry: voice mode is a session-level feature; enter a session first and use the mic button in the input toolbar
|
|
208
296
|
- The preview request timeout uses `AbortSignal.timeout` (Chrome 103+ / Firefox 100+ / Safari 16+); on older browsers clicking preview immediately shows a failure hint — an expected degradation
|
|
209
297
|
|
|
210
|
-
|
|
298
|
+
---
|
|
299
|
+
|
|
300
|
+
## 🛠️ Troubleshooting
|
|
211
301
|
|
|
212
302
|
| Symptom | Fix |
|
|
213
303
|
| --- | --- |
|
|
@@ -219,8 +309,26 @@ input: mic ──RMS VAD (1500 ms silence split)──▶ POST /voice-mode/asr
|
|
|
219
309
|
| Poor recognition | Get closer to the mic, reduce ambient noise; if echo remains, raise the interrupt sensitivity by one step |
|
|
220
310
|
| Hold mode has no effect | Make sure hold mode is active and you're in voice mode (button shows `按住说话`); the browser window must be in the foreground |
|
|
221
311
|
| Preview button reports synthesis failure | Edge TTS unreachable (overseas) or the ShortName doesn't exist: verify the name (`node scripts/list-voices.mjs` lists all) and retry later |
|
|
312
|
+
| Caption is hidden behind the input box | Default `captionMaxWidth=1` (70vw) + `captionFontSize=0` (12px) can overlap the bottom input on narrow viewports; raise the tier or click the caption's `×` to dismiss |
|
|
313
|
+
| Yield behavior is wrong (saying `嗯` doesn't pause / real speech gets hard-barge) | Short backchannel words (`嗯 / 对`) auto-pause 1.5 s then reading resumes; continuing to speak triggers hard barge-in; disable `backchannelYield` to restore pre-change behavior (ADR-0008) |
|
|
314
|
+
| Says `嗯` but no yield happens | Confirm `backchannelYield=true` (default on); in hold mode, continuing to talk within 1.5 s of release triggers hard barge-in |
|
|
315
|
+
| Auto-exits after 5 minutes idle (don't want) | Raise `idleTimeoutMinutes` (default 5 min, **reading counts as activity**) |
|
|
316
|
+
|
|
317
|
+
---
|
|
318
|
+
|
|
319
|
+
## 🛣️ Roadmap
|
|
222
320
|
|
|
223
|
-
|
|
321
|
+
Full backlog (43 P0-P3 items) at [`docs/competitive/backlog.md`](../../docs/competitive/backlog.md).
|
|
322
|
+
|
|
323
|
+
- ✅ **Done (v0.7.7)**: 11 batches of comprehensive fixes (caption tiers / yield semantics / model prewarm / defaults micro-adjust / dead-code cleanup / etc.)
|
|
324
|
+
- 🚧 **P0 (near-term)**: ADR-0003 client-side VAD / ADR-0006 first-level probe wired to manual / F1 emotion DSL full roll-out
|
|
325
|
+
- 📋 **P1 (mid-term)**: MCP `voice_*` toolset / card form draft validate / status-bar idle polish
|
|
326
|
+
- 💡 **P2 (far-term)**: Voice cloning (user-deferred) / ADR-0004 WebSocket transport
|
|
327
|
+
- ⏸️ **Deferred**: xAI fallback / C1 persona layer (user-deferred)
|
|
328
|
+
|
|
329
|
+
---
|
|
330
|
+
|
|
331
|
+
## 🛠️ Development
|
|
224
332
|
|
|
225
333
|
### Dependency discipline (important)
|
|
226
334
|
|
|
@@ -277,6 +385,8 @@ scripts/list-voices.mjs print all Edge TTS voices (source of the voice table)
|
|
|
277
385
|
|
|
278
386
|
Integration probes (`hold-e2e.js`, `spoken-prompt-rpc.sh`, `spoken-toggle-ui-check.js`) live in the repo root `test/`, outside this npm package.
|
|
279
387
|
|
|
280
|
-
|
|
388
|
+
---
|
|
389
|
+
|
|
390
|
+
## 📄 License
|
|
281
391
|
|
|
282
|
-
MIT
|
|
392
|
+
[MIT](../../LICENSE)
|
package/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# dsh-voice-mode
|
|
2
2
|
|
|
3
|
-
[](LICENSE)
|
|
4
|
+
[](https://github.com/qishuilalala/dsh-voice-mode/releases)
|
|
5
|
+
[](https://www.npmjs.com/package/dsh-voice-mode)
|
|
6
|
+
[](../../docs/rules/STATE.md)
|
|
7
7
|
|
|
8
8
|
DeepSeek Harness 语音双工对话模式:会话内一键进入 → 边说边出字的流式识别 → 停顿自动发送 → 最终答复按句流式朗读 + 实时字幕,开口即可打断(真 barge-in)。无需 API Key,识别模型在本地宿主端推理。
|
|
9
9
|
|
|
@@ -13,11 +13,13 @@ DeepSeek Harness 语音双工对话模式:会话内一键进入 → 边说边
|
|
|
13
13
|
|
|
14
14
|

|
|
15
15
|
|
|
16
|
-
> **版本说明(
|
|
16
|
+
> **版本说明(v0.7.9,2026-09-18)**:**唤醒词链路全面修复**(Issue #10 + 真机复测)——台架驱动真引擎逐项定位并修掉五处流程缺陷:① 朗读期 TTS 回声污染待机段(喊不醒);② 只上传超门限帧导致尾字不 flush(唤醒词只剩半截);③ 唤醒命中丢整段(连说「唤醒词+命令」只发出去尾部几个字);④ 命中晚于停口时命令悬挂不定稿;⑤ 只喊唤醒词后停顿会关掉命令窗口(命令丢失)。**现在唤醒词可与命令连说、词头自动剥离不进消息**,待机态实时显示「它听到了什么」,支持「唤醒词…停顿…命令」。其余同 v0.7.7:朗读默认 Edge 云端,本地 TTS(VITS / Kokoro)可选;静音断句默认 1500 毫秒。
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
---
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
## 🤝 Fork 增强(本仓库新增)
|
|
21
|
+
|
|
22
|
+
本仓库在上游 [haoku123/dsh-voice](https://github.com/haoku123/dsh-voice) 基础上加入了大量增强,核心如下(完整清单见 git 历史与 [docs/rules/STATE.md](../../docs/rules/STATE.md)):
|
|
21
23
|
|
|
22
24
|
- **朗读默认 Edge 云端;本地 TTS 可选(隐私优先)**:选本地则回复文本不出本机——
|
|
23
25
|
- 本地 VITS(`sherpa-onnx-vits-zh-ll`,纯中文,5 说话人);
|
|
@@ -32,22 +34,28 @@ DeepSeek Harness 语音双工对话模式:会话内一键进入 → 边说边
|
|
|
32
34
|
|
|
33
35
|
> ⚠️ 上文截图与 `assets/demo.gif` 为**上游旧版界面**(单按钮时期);当前界面在语音按钮旁多一颗「模式切换」按钮。
|
|
34
36
|
|
|
35
|
-
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## ✨ 功能
|
|
36
40
|
|
|
37
41
|
- **语音模式**:输入框工具排麦克风按钮或全局快捷键 `Ctrl+Shift+V` 进入/退出;全局单活(同一时刻仅一个会话处于语音模式,切换会话自动让出)
|
|
38
42
|
- **两种交互模式(输入框旁按钮或设置可切换,切换即持久化)**:
|
|
39
43
|
- `toggle`(默认)持续聆听:RMS VAD 分段 → zipformer2 流式识别(边说边出字,实时字幕预览)→ 静音约 1500 毫秒断句进草稿,连续多段拼成一条消息,再静音约 1500 毫秒(合计约 3 秒)自动发送;按住 `Ctrl` 强制立即发送
|
|
40
44
|
- `hold` 按住说话:短按进入/退出,**按住麦克风按钮说话、松手即发**(滑出取消、`Esc`/失焦放弃本段);按住期间停顿不断句(上限 10 分钟);`Ctrl` 按住即录、松开即发
|
|
41
|
-
- **唤醒词(可选,默认关)**:设置 `wakeWord` 后进入待机态,说出唤醒词才开始识别(如「你好小D
|
|
45
|
+
- **唤醒词(可选,默认关)**:设置 `wakeWord` 后进入待机态,说出唤醒词才开始识别(如「你好小D」)。**可直接与命令连说**(「你好小D,帮我查天气」)——唤醒词会在定稿/字幕里自动剥掉、不进消息;匹配带容错(同音字替换「小莫→小墨」、首字错、前置语气词「呃/喂/我说」),建议 3-4 字;待机态状态条实时显示「说『x』开始 · 它听到的转写」,唤醒成败可自查。边界(重要):**仅 `toggle` 模式生效**(`hold` / 手动打断下唤醒词静默失效);**说完整一句后回到待机态需重说唤醒词**(只喊唤醒词、还没说命令时保持聆听,可停顿想好再说);**朗读期说唤醒词不触发**(打断由 VAD 开口即触发,与唤醒词无关)
|
|
46
|
+
- **字幕档位**(批 3):`captionFontSize` 4 档(0=12px / 1=14px / 2=18px / 3=24px)+ `captionMaxWidth` 3 档(0=50vw / 1=70vw / 2=90vw),窄屏自适应
|
|
47
|
+
- **让位语义**(批 5 / ADR-0008):`backchannelYield` 默认开(I10 豁免)—— 朗读期用户插话「嗯/对」自动让位 1.5s,真要说走硬打断;让 LLM 主动让出话轮(人格层让位);关掉恢复改造前行为
|
|
42
48
|
- **输出链路**:只朗读最终答复的 `text-delta`(reasoning/工具调用不读),按句流式朗读(默认 Edge 云端;可切本地 VITS/Kokoro,中英混读选 Kokoro)+ 右下角实时字幕浮层;工具调用触发提示音;全文照常写入聊天记录;口语化提示词(设置 `spokenFormat`,默认开)让回复为自然短句、不带 Markdown 排版符号
|
|
43
|
-
- **开口打断(barge-in)**:服务端 Silero VAD 帧级检测 + 回声门控(echoGateDb)三档灵敏度 → 本地静音 + host 合成队列作废 +
|
|
49
|
+
- **开口打断(barge-in)**:服务端 Silero VAD 帧级检测 + 回声门控(echoGateDb)三档灵敏度 → 本地静音 + host 合成队列作废 + 正在运行的回合取消(保留半截并自然续入新消息);朗读中自动切超灵敏档。开启唤醒词后打断**仍是开口即打断**(打断门控是 VAD,不是唤醒词;打断后回待机态)
|
|
44
50
|
- **模型懒加载与进度**:首次使用自动下载识别/合成模型(`.part` 断点续传),状态条实时显示进度;可用 `npm run prefetch` 预下载
|
|
45
|
-
- **设置**:设置 → Plugins → 插件配置 → 语音模式(voice-mode
|
|
51
|
+
- **设置**:设置 → Plugins → 插件配置 → 语音模式(voice-mode),可调朗读引擎/音色/语速/打断灵敏度/静音停顿/空闲超时/模型镜像/自动发送/交互模式/唤醒词/口语化提示词/字幕档位/让位语义;**音色可试听**(按当前音色+语速即时合成预览,自定义 ShortName 亦可)
|
|
46
52
|
- **界面语言**:跟随浏览器语言(中文 / English;切换后刷新页面生效)
|
|
47
53
|
- **容错**:麦克风被拒红点提示、模型下载失败可见提示、TTS 连接失败状态条提示(自动退避重试)、提交失败文字留在草稿、SSE 断线自动重连
|
|
48
|
-
- **空闲退出**:10
|
|
54
|
+
- **空闲退出**:5 分钟无活动自动退出并释放麦克风(**正在朗读计为活动**,长朗读不会中途下线;批 J 已微调默认 10→5)
|
|
55
|
+
|
|
56
|
+
---
|
|
49
57
|
|
|
50
|
-
##
|
|
58
|
+
## 🚀 5 分钟上手(Quick Start)
|
|
51
59
|
|
|
52
60
|
```sh
|
|
53
61
|
dsh plugin --profile web add dsh-voice-mode
|
|
@@ -55,7 +63,15 @@ dsh plugin --profile web add dsh-voice-mode
|
|
|
55
63
|
|
|
56
64
|
bundle 插件安装后需重启 dsh 生效(Linux:`systemctl restart dsh`;其他平台重启 dsh 进程)。
|
|
57
65
|
|
|
58
|
-
|
|
66
|
+
**第一次用**:
|
|
67
|
+
|
|
68
|
+
1. 进入任一会话,按 `Ctrl+Shift+V`(或点输入区麦克风按钮)进入语音模式,状态条显示「聆听中…」;
|
|
69
|
+
2. 说一句完整的话(如「帮我看看今天的天气」)→ 实时字幕立即出现,停顿后自动发送;
|
|
70
|
+
3. AI 回复开始朗读时,**开口说话 → 朗读即刻停止,你的话被听见**(这就是 barge-in)。
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## ⌨️ 操作手势
|
|
59
75
|
|
|
60
76
|
| 手势 | 作用 |
|
|
61
77
|
| --- | --- |
|
|
@@ -68,22 +84,28 @@ bundle 插件安装后需重启 dsh 生效(Linux:`systemctl restart dsh`;
|
|
|
68
84
|
| 点状态条「退出」 | 退出语音模式 |
|
|
69
85
|
| 点字幕浮层「跳过」 | 跳过当前句朗读 |
|
|
70
86
|
|
|
71
|
-
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## ⚙️ 设置(设置 → Plugins → 插件配置 → 语音模式)
|
|
72
90
|
|
|
73
91
|
| 键 | 默认 | 说明 |
|
|
74
92
|
| --- | --- | --- |
|
|
75
93
|
| `ttsEngine` | `edge` | 朗读引擎:`edge` 微软云端(默认,快)/ `vits` 本地中文 / `kokoro` 本地中英;**即时生效** |
|
|
76
94
|
| `kokoroModel` | `int8` | Kokoro 模型精度:`int8`(默认,109MB,纯 CPU/低带宽推荐)/ `fp32`(311MB,音质更好,独显/大内存推荐);两档共用 103 音色,**即时生效** |
|
|
77
95
|
| `voice` | 按引擎 | 音色:VITS 五说话人;Kokoro 103 个(下拉+◀▶,62 深沉/68 浑厚/75 清亮/76 磁性置顶);Edge 进入时自动加载全量 322 个。行内「试听」可即时预览 |
|
|
78
|
-
| `rate` | `1.
|
|
79
|
-
| `interruptLevel` | `0` | 发声打断灵敏度(服务端 VAD 帧级检测 + 回声门控):0
|
|
96
|
+
| `rate` | `1.1` | 朗读语速倍率(0.5 慢速 ~ 2.0 快速),**即时生效**(批 J 1.0→1.1) |
|
|
97
|
+
| `interruptLevel` | `0` | 发声打断灵敏度(服务端 VAD 帧级检测 + 回声门控):0 高门槛(3 帧)/ 1 中(2 帧)/ 2 低(1 帧) |
|
|
80
98
|
| `silenceMs` | `1500` | 说完整一句的静音停顿毫秒数 |
|
|
81
|
-
| `idleTimeoutMinutes` | `
|
|
99
|
+
| `idleTimeoutMinutes` | `5` | 无活动自动退出语音模式的分钟数(朗读计为活动;批 J 10→5) |
|
|
82
100
|
| `modelHost` | 默认源 | 模型下载源(国内网络填 `https://hf-mirror.com`) |
|
|
83
101
|
| `autoSend` | `true` | 静音到点自动发送(连续多段拼成一条消息);关闭则只进草稿(按住 `Ctrl` / hold 松手仍会发送) |
|
|
84
102
|
| `mode` | `toggle` | 交互模式:`toggle` 持续聆听 + 1500ms 静音断句;`hold` 按住说话、松手发送(短按退出) |
|
|
85
|
-
| `wakeWord` | 空(关) | 唤醒词(如「你好小D」):进入后先说唤醒词激活,避免误触;空 =
|
|
103
|
+
| `wakeWord` | 空(关) | 唤醒词(如「你好小D」):进入后先说唤醒词激活,避免误触;空 = 关闭。**可与命令连说**,词头自动剥掉不进消息;匹配带容错(编辑距离 ≤1 + 前 3 字符前导窗口,吸收同音字/语气词);**建议 3-4 字**——单字词无容错、2 字词的容错会连带吸收所有同首字的 2 字词(如「小莫」也会唤醒「小张」),介意误唤醒用 3 字以上;每句断句或打断后回待机需重说;**仅 toggle 模式生效**(hold / 手动打断下不生效);朗读期说唤醒词不触发 |
|
|
86
104
|
| `spokenFormat` | `true` | 语音会话注入口语化提示词:开启后**仅当前语音会话**的回复被注入「口语化短句、不用 Markdown 排版符号」提示词(朗读更顺),**即时生效** |
|
|
105
|
+
| `senseITN` | `true` | 批 2 P0:SenseVoice 逆文本归一化(数字/日期/货币规范化;默认开) |
|
|
106
|
+
| `captionFontSize` | `0` | 批 3 P0:字幕字号档位 0=12px / 1=14px / 2=18px / 3=24px(默认 0 与现状字节等价) |
|
|
107
|
+
| `captionMaxWidth` | `1` | 批 3 P0:字幕宽度档位 0=50vw / 1=70vw / 2=90vw(视口 <686px 接近 480px,>686px 宽于 480px) |
|
|
108
|
+
| `backchannelYield` | `true` | 批 5 P1:让位语义(ADR-0008);朗读期说「嗯/对」自动让位 1.5s + 真要说走硬打断。I10 豁免(默认开是产品决策);关 = 行为等同改造前 |
|
|
87
109
|
|
|
88
110
|
生效范围:`voice`/`rate`/`ttsEngine`/`kokoroModel`/`spokenFormat` **立即生效**;其余设置下次进入语音模式时生效。设置项默认值由插件配置(`base` 层)提供。
|
|
89
111
|
|
|
@@ -111,22 +133,69 @@ bundle 插件安装后需重启 dsh 生效(Linux:`systemctl restart dsh`;
|
|
|
111
133
|
|
|
112
134
|
`voice-mode` 命名空间配置可直接写入 `~/.dsh/settings.yaml`;插件总开关 `enabled`(默认 `true`)、模型缓存目录 `cacheDir`、在安装配置中设置。
|
|
113
135
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## 🏛️ 工作原理(Architecture)
|
|
139
|
+
|
|
140
|
+
```mermaid
|
|
141
|
+
flowchart LR
|
|
142
|
+
subgraph Client[浏览器 Client]
|
|
143
|
+
Mic[麦克风 16kHz<br/>AudioWorklet<br/>echoCancellation:true] --> VAD[客户端 VAD<br/>RMS 分段]
|
|
144
|
+
VAD -->|partial 0.9s| UI[状态条 + 字幕浮层]
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
subgraph Host[宿主 dsh.host]
|
|
148
|
+
ASR[zipformer2 流式识别<br/>host 端 WASM]
|
|
149
|
+
SV[SenseVoice 定稿<br/>+ ITN + 标点]
|
|
150
|
+
Tap[llm/stream tap<br/>仅观察·不阻塞]
|
|
151
|
+
Seg[sentence segmenter]
|
|
152
|
+
Q[TtsQueue<br/>epoch 打断]
|
|
153
|
+
TTS{引擎}
|
|
154
|
+
Edge[Edge 云端]
|
|
155
|
+
Vits[本地 VITS WASM]
|
|
156
|
+
Kokoro[本地 Kokoro<br/>原生 addon]
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
UI -->|audio f32 PCM<br/>POST /voice-mode/asr| ASR
|
|
160
|
+
ASR --> SV
|
|
161
|
+
SV --> Draft[composer draft<br/>autoSend]
|
|
162
|
+
Draft --> Tap
|
|
163
|
+
Tap --> Seg
|
|
164
|
+
Seg --> Q
|
|
165
|
+
Q --> TTS
|
|
166
|
+
TTS -->|edge| Edge
|
|
167
|
+
TTS -->|vits| Vits
|
|
168
|
+
TTS -->|kokoro| Kokoro
|
|
169
|
+
Edge -.->|SSE audio frame| UI
|
|
170
|
+
Vits -.->|SSE audio frame| UI
|
|
171
|
+
Kokoro -.->|SSE audio frame| UI
|
|
172
|
+
VAD -.->|唤醒词/打断| Q
|
|
123
173
|
```
|
|
124
174
|
|
|
125
175
|
- 识别在 **host 端本地运行**(zipformer2 中文 int8 WASM + SenseVoice 定稿,模型懒下载),音频不上传第三方;识别定稿由 SenseVoice 补标点;
|
|
126
176
|
- 朗读默认 **Edge 云端**;本地 VITS 纯中文 / Kokoro 原生中英(跑在独立子进程、崩溃自愈)可选(隐私优先);
|
|
127
177
|
- 同一时间仅一个会话处于语音模式(全局单活);LLM 流被无损观察(不阻塞)。
|
|
128
178
|
|
|
129
|
-
|
|
179
|
+
详细架构决策:见 [`docs/adr/`](../../docs/adr/README.md) 8 个 ADR。
|
|
180
|
+
|
|
181
|
+
---
|
|
182
|
+
|
|
183
|
+
## 🔍 与 dsh 内置语音模式对比
|
|
184
|
+
|
|
185
|
+
| 维度 | dsh 内置 | dsh-voice-mode(本插件) |
|
|
186
|
+
| --- | --- | --- |
|
|
187
|
+
| 识别模型 | 云端 API(需 key) | **本地 zipformer2 + SenseVoice**(零 key) |
|
|
188
|
+
| 多语种 | 英文为主 | **SenseVoice 自动识别(auto)+ ITN** |
|
|
189
|
+
| 朗读引擎 | 云端 TTS | **Edge 云端 + 本地 VITS/Kokoro** 三选一 |
|
|
190
|
+
| 打断检测 | 基础 VAD | **三档灵敏度 + 回声门控 + 让位语义** |
|
|
191
|
+
| 热词偏置 | 无 | 无(已移除,详见 v0.7.7 文档说明) |
|
|
192
|
+
| 字幕 a11y | 无 | **4 档字号 + 3 档宽度 + 主题跟随** |
|
|
193
|
+
| 唤醒词 | 无 | **轻量流式匹配 + 前缀语气词白名单** |
|
|
194
|
+
| 兼容 dsh | — | **0.1.1-rc.2 → 0.1.5-rc.2 全版本** |
|
|
195
|
+
|
|
196
|
+
---
|
|
197
|
+
|
|
198
|
+
## 🚧 已知限制
|
|
130
199
|
|
|
131
200
|
- 发声打断依赖浏览器回声消除(`echoCancellation`);扬声器音量过大时可能漏声到麦克风
|
|
132
201
|
- `Ctrl+Shift+V` 会覆盖浏览器「粘贴纯文本」快捷键(普通粘贴仍可用 `Ctrl+V`)
|
|
@@ -143,7 +212,9 @@ bundle 插件安装后需重启 dsh 生效(Linux:`systemctl restart dsh`;
|
|
|
143
212
|
- iOS 后台/锁屏时识别与朗读暂停,回前台自动恢复(可能丢句);建议语音模式期间保持前台
|
|
144
213
|
- **安全说明**:插件 HTTP 面(`/voice-mode/*`)遵循宿主安全模型——请勿将 dsh 端口直接暴露公网;经反向代理发布时由代理层(如 basic auth)鉴权;插件侧对敏感操作保留会话归属校验
|
|
145
214
|
|
|
146
|
-
|
|
215
|
+
---
|
|
216
|
+
|
|
217
|
+
## 🛠️ 故障排查
|
|
147
218
|
|
|
148
219
|
| 现象 | 处理 |
|
|
149
220
|
| --- | --- |
|
|
@@ -152,9 +223,31 @@ bundle 插件安装后需重启 dsh 生效(Linux:`systemctl restart dsh`;
|
|
|
152
223
|
| 朗读无声音/无字幕 | 本地引擎首次合成需加载模型;若持续失败查看状态条提示(自动退避重试);确认页面前台且未静音 |
|
|
153
224
|
| 语音模式进不去 | 检查插件 `enabled`;多标签页时确认当前会话为活动会话 |
|
|
154
225
|
| 识别到但不是我要说的 | 环境噪声或唤醒词误判:降低音量、提高 `interruptLevel`(高门槛)或启用 `wakeWord` |
|
|
226
|
+
| 唤醒词唤不醒 | 先看待机态状态条的实时转写(它听到了什么):同音字 1 字内已自动容错;停顿约 1.5 秒(= 静音断句时长)再说,清掉待机段残留语音、从段首重新匹配;换一个转写稳定的词(建议 3-4 字) |
|
|
227
|
+
| 打断后说话没反应(开了唤醒词) | 唤醒词只管「开始识别」的门:打断/每句断句后回到待机态,**需重说唤醒词**再继续说;待机态状态条会显示「说『x』开始」提示 |
|
|
228
|
+
| 唤醒词完全没反应 | ① 确认交互模式是 `toggle`(`hold` 与手动打断下唤醒词不生效);② 确认 `bargeInMode`:`manual`(或 `detect` 在本机无原生回声消除时自动落 manual)会关闭常驻聆听 → 唤醒词失效,改回 `auto` 或按住麦克风/Ctrl;③ 朗读期说唤醒词不触发(先等 AI 说完);④ 看待机态实时转写,确认它听到了什么(同音字 1 字内已容错,建议 3-4 字词) |
|
|
155
229
|
| 按住说话松手后没反应 | 确认交互模式为「按住说」且按住期间按钮高亮;松手后识别定稿约 1 秒内进入草稿 |
|
|
230
|
+
| 打不断(朗读中开口无反应) | 调高 `interruptLevel`(更敏感档 = 0 或 1)或检查麦克风权限;若 VAD 持续不触发可临时切到「手动打断」(`mode: hold` / 唤醒词定时延后);**不要**调 `echoGateDb`——真机 3.1 分钟朗读期里 Silero 0/777 帧把回声判成语音,原生 AEC 生效时此阈值从未被执行(详见 ADR-0006) |
|
|
231
|
+
| 字幕被输入框挡住 | 默认 `captionMaxWidth=1`(70vw)+ `captionFontSize=0`(12px)在窄屏可能与底部输入框重叠;调高档位或点字幕浮层「×」收起 |
|
|
232
|
+
| 让位行为异常(朗读期说「嗯」不停 / 真话被打断) | 「嗯/对」类短词触发让位 1.5s 后继续朗读;继续说真话会走硬打断;不要时关 `backchannelYield` 即可恢复改造前行为(ADR-0008) |
|
|
233
|
+
| 朗读期说「嗯」没让位 | 确认 `backchannelYield=true`(默认开);hold 模式松手后让位 1.5s 内继续说话会变硬打断 |
|
|
234
|
+
| 空闲 5 分钟自动退出(不想退) | 调高 `idleTimeoutMinutes`(默认 5 分钟,**朗读计为活动**) |
|
|
235
|
+
|
|
236
|
+
---
|
|
156
237
|
|
|
157
|
-
##
|
|
238
|
+
## 🛣️ 路线图(Roadmap)
|
|
239
|
+
|
|
240
|
+
完整 backlog(43 项 P0-P3)见 [`docs/competitive/backlog.md`](../../docs/competitive/backlog.md)。
|
|
241
|
+
|
|
242
|
+
- ✅ **已完成(v0.7.7)**:11 批次周全修复(字幕档位 / 让位语义 / 模型预热 / 默认值微调 / 死代码清理等)
|
|
243
|
+
- 🚧 **P0(近期)**:ADR-0003 VAD 下沉 / ADR-0006 第一级探测接通 manual / F1 emotion DSL 全量上线
|
|
244
|
+
- 📋 **P1(中期)**:MCP `voice_*` 工具集 / 卡片表单 draft validate / 状态条 idle 优化
|
|
245
|
+
- 💡 **P2(远期)**:声音克隆(用户已决定推迟)/ ADR-0004 WebSocket transport
|
|
246
|
+
- ⏸️ **已推迟**:xAI fallback / C1 人格层(用户已决定推迟)
|
|
247
|
+
|
|
248
|
+
---
|
|
249
|
+
|
|
250
|
+
## 🛠️ 开发
|
|
158
251
|
|
|
159
252
|
```sh
|
|
160
253
|
pnpm install && pnpm build # esbuild:lib/index.js(host)+ lib/client.js(browser)
|
|
@@ -178,8 +271,10 @@ src/client.tsx client:麦克风按钮 + 模式切换按钮 + 状态条 +
|
|
|
178
271
|
src/strings.ts client:中英文案字典(navigator.language)
|
|
179
272
|
```
|
|
180
273
|
|
|
181
|
-
|
|
274
|
+
---
|
|
275
|
+
|
|
276
|
+
## 📄 License
|
|
182
277
|
|
|
183
|
-
[MIT](LICENSE)
|
|
278
|
+
[MIT](../../LICENSE)
|
|
184
279
|
|
|
185
|
-
> 部分实现借鉴 [haoku123/dsh-voice](https://github.com/haoku123/dsh-voice)(派生声明见子包 LICENSE)。
|
|
280
|
+
> 部分实现借鉴 [haoku123/dsh-voice](https://github.com/haoku123/dsh-voice)(派生声明见子包 LICENSE)。
|