dsh-voice-mode 0.2.3 → 0.3.0

package/README.en.md

@@ -20,14 +20,13 @@
 
 - **Voice mode**: toggle with the microphone button in the input toolbar or the global shortcut `Ctrl+Shift+V`; globally single-active (only one session is in voice mode at a time; switching sessions yields automatically)
 - **Two interaction modes (switchable in settings)**:
-  - `toggle` (default) continuous listening: RMS VAD segmentation → streaming zipformer2 ASR (words appear as you speak, live caption preview) → automatic sentence split and send after 2 s of silence; hold `Ctrl` to force an immediate send
+  - `toggle` (default) continuous listening: local RMS gate opens segments → streaming zipformer2 ASR (words appear as you speak, live caption preview) → automatic sentence split and send after 700 ms of silence (<250 ms utterances are treated as noise; endpoints are decided by the host-side Silero VAD, with a client-side silence timer as fallback); hold `Ctrl` to force an immediate send
   - `hold` push-to-talk: short tap to enter/exit, **hold the mic button to talk, release to send** (swipe up to cancel, `Esc`/blur abandons the segment); hold `Ctrl` to record-by-keyboard, release to send
-- **Wake word (optional, off by default)**: after setting `wakeWord`, entering voice mode starts in standby, and recognition only begins once the wake word is spoken (e.g. `你好小D`), preventing accidental triggers
-- **Output pipeline**: only the final answer's `text-delta` is read (reasoning/tool calls are skipped), streamed sentence-by-sentence via Edge TTS with a live caption overlay at the bottom-right; tool calls trigger a beep; the full text is still written to the chat; in voice mode a spoken-format system prompt is injected (short natural sentences, no Markdown decoration), and the reader side strips markers as well for a smoother listening experience
-- **Barge-in**: three sensitivity levels of voice-onset detection → local mute + host synth queue invalidation (epoch) + running turn cancellation (the half-finished part is kept and naturally flows into your new message)
-- **Lazy model download with progress**: the zipformer2 Chinese streaming model (~160 MB, `.part` resumable) is downloaded on first use with live progress in the status bar; `npm run prefetch` can pre-download it
+- **Output pipeline**: only the final answer's `text-delta` is read (reasoning/tool calls are skipped), streamed sentence-by-sentence via Edge TTS with a live caption overlay at the bottom-right; the full text is still written to the chat; in voice mode a spoken-format system prompt is injected (short natural sentences, no Markdown decoration), and the reader side strips markers as well for a smoother listening experience
+- **Barge-in**: **NLMS acoustic echo cancellation (P3, cross-correlation bulk-delay estimate + 64 ms short filter)** using the page's own TTS playback as the reference, then **host-side Silero VAD frame-level detection (isSpeech, sent down with partial responses) drives the interrupt** — three sensitivity levels map to speech-confirmation time (0 high barrier ~0.3 s / 1 medium 0.2 s / 2 low 0.1 s); with `bargeInMode: auto` speaking interrupts automatically, with `manual` (recommended for speakers) hold the mic button / Ctrl to interrupt explicitly → local mute + host synth queue invalidation (epoch) + running turn cancellation (the half-finished part is kept and naturally flows into your new message)
+- **Lazy model download with progress**: the zipformer2 streaming model (~160 MB), Silero VAD (~2 MB) and the SenseVoice finalization model (~228 MB, first finalize; can be disabled) are downloaded on first use — all `.part` resumable with hf-mirror fallback; live progress in the status bar; `npm run prefetch` pre-downloads them
 - **Resilience**: mic-denied red hint, visible model-download failure, TTS unreachable status hint (auto retry), failed submit keeps the text in the draft, SSE auto-reconnect
-- **Settings**: Settings → Plugins → voice-mode, with voice / rate / interrupt sensitivity / silence pause / idle timeout / model mirror / auto send / interaction mode / wake word; **voices are previewable** (the "试听/Preview" button synthesizes and plays the current voice at the current rate instantly, no need to enter voice mode; custom ShortNames are previewable too)
+- **Settings**: Settings → Plugins → voice-mode, with voice / rate / interrupt sensitivity / silence pause / idle timeout / model mirror / auto send / interaction mode; **voices are previewable** (the "试听/Preview" button synthesizes and plays the current voice at the current rate instantly, no need to enter voice mode; custom ShortNames are previewable too)
 - **Idle exit**: auto-exit and mic release after 10 minutes of inactivity
 
 ## Interaction gestures
@@ -35,11 +34,10 @@
 | Gesture | Behaviour |
 | --- | --- |
 | Click the mic button / `Ctrl+Shift+V` | Enter / exit voice mode |
-| Just speak, pause 2 s (toggle) | Auto sentence split and send |
+| Just speak (toggle) | Silence-based auto sentence split and send (host VAD endpoint) |
 | Hold `Ctrl` (toggle, ≥250 ms speech) | Force-send the current segment immediately |
 | **Hold the mic button (hold)** | Hold to talk, release to send; swipe up / `Esc` / blur abandons the segment; <250 ms tap exits the mode |
 | Hold `Ctrl` (hold, ≥600 ms) | Keyboard push-to-talk, release to send |
-| Speak the wake word first (if configured) | Activate from standby into listening (then recognition and sending begin) |
 | Speak while AI is reading | Interrupt playback and cancel the running turn |
 | Type in the input box | Auto-exit voice mode (draft is kept) |
 
@@ -54,7 +52,7 @@ dsh plugin --profile web add dsh-voice-mode
 npx -y @deepseek-ai/dsh plugin --profile web add dsh-voice-mode
 
 # Option 2: local tarball
-dsh plugin --profile web add ./dsh-voice-mode-0.1.0.tgz
+dsh plugin --profile web add ./dsh-voice-mode-0.3.0.tgz
 
 # Option 3: from source
 git clone https://github.com/qishuilalala/dsh-voice-mode.git
@@ -72,34 +70,34 @@ dsh plugin --profile web add .
 ```sh
 npm run prefetch          # run inside the plugin dir; writes to the platform cache dir
 # or specify the cache location: node scripts/prefetch.mjs --cache-dir /where/ever/models
+node scripts/bench-asr.mjs --dir <test-set-dir>   # P4-2 offline CER/latency/size comparison (zipformer int8 vs xlarge / small-CTC / paraformer; needs 16k mono PCM wav + same-name .txt)
 ```
 
 ## Usage
 
 1. Click the mic button in the input toolbar (or press `Ctrl+Shift+V`) to enter voice mode; a status bar appears above the input box
-2. Choose how to speak: just talk and let the 2 s pause auto-send (toggle); or hold the mic button and release to send (hold)
+2. Choose how to speak: just talk and let the silence-based end-point auto-send (toggle); or hold the mic button and release to send (hold)
 3. The AI answer is read sentence-by-sentence with a caption overlay at the bottom-right; click "Skip" or just start speaking to interrupt
 4. Click "Exit" in the status bar (or press `Ctrl+Shift+V` again) to leave voice mode
 
 On first entry the recognition model is downloaded; the status bar shows `正在加载模型… <file> <percent>%`.
 
-If a wake word is configured, you land in standby first (the status bar prompts `说『唤醒词』开始`), and recognizing starts after you speak the wake word.
-
 ## Settings (Settings → Plugins → Plugins config → 语音模式)
 
 | Key | Default | Description |
 | --- | --- | --- |
 | `voice` | `zh-CN-XiaoxiaoNeural` | Edge TTS voice (see the common voices table below), **applies live**; the inline "试听" button previews it at the current rate (both listed voices and custom ShortNames are previewable; failures show a visible hint) |
 | `rate` | `1.0` | Reading speed multiplier (0.5 slow ～ 2.0 fast), **applies live** |
-| `interruptLevel` | `0` | Barge-in sensitivity: 0 high threshold / 1 medium / 2 low |
-| `silenceMs` | `2000` | Silence pause in ms that marks the end of a complete sentence |
+| `interruptLevel` | `0` | Barge-in sensitivity: 0 high barrier / 1 medium / 2 low (~0.3/0.2/0.1 s speech confirmation) |
+| `silenceMs` | `700` | Silence pause in ms that marks the end of a complete sentence (at least 250 ms of speech required) |
 | `idleTimeoutMinutes` | `10` | Minutes of inactivity before auto-exiting voice mode |
 | `modelHost` | default | ASR model download host (use `https://hf-mirror.com` on mainland networks) |
 | `autoSend` | `true` | Auto-send after a finalized transcript; when off, text only goes to the draft (hold `Ctrl` / release in hold mode still sends) |
-| `mode` | `toggle` | Interaction mode: `toggle` continuous listening + 2 s silence split; `hold` push-to-talk, release to send (short tap exits) |
-| `wakeWord` | empty (off) | Wake word (e.g. `你好小D`): speak it after entering to activate, avoiding accidental triggers; empty = off |
+| `mode` | `toggle` | Interaction mode: `toggle` continuous listening + silence-based end-point detection; `hold` push-to-talk, release to send (short tap exits) |
+| `spokenFormat` | `false` | Inject a spoken-format system prompt into voice replies (colloquial short sentences, no Markdown decoration; **applies live**) |
+| `senseVoice` | `true` | Re-transcribe the finalized utterance with SenseVoice (punctuation + ITN, more accurate); turning it off skips the 228 MB model and keeps streaming only |
 
-Effect timing: `voice`/`rate` take effect **immediately** (TTS hot-swap); the rest apply on the next voice-mode entry. Defaults come from the plugin config (`base` layer) — they follow the config unless explicitly changed.
+Effect timing: `voice`/`rate`/`spokenFormat`/`senseVoice` take effect **immediately**; the rest apply on the next voice-mode entry. Defaults come from the plugin config (`base` layer) — they follow the config unless explicitly changed.
 
 ### Common voices (full list: `node scripts/list-voices.mjs`)
 
@@ -134,7 +132,7 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
     voice: zh-CN-XiaoxiaoNeural
     rate: 1.0
     interruptLevel: 0
-    silenceMs: 2000
+    silenceMs: 700
     idleTimeoutMinutes: 10
     modelHost: https://huggingface.co
 ```
@@ -148,9 +146,9 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
 
 | Route | Description |
 | --- | --- |
-| `GET /voice-mode/stream` | SSE: `event: audio` (`{sessionId, seq, text, audio(base64 MP3)}`), `event: mode` (global single-active ownership), `event: tool` (beep), `event: asr-progress / asr-ready / asr-error / tts-error` |
+| `GET /voice-mode/stream` | SSE: `event: audio` (`{sessionId, sentenceId, chunkId, final, text?(only on the final frame), audio(base64 MP3 chunk)}` — P1-1 chunked forwarding, the client reassembles each sentence before playback), `event: latency` (`{sessionId, stage}`: first-llm-token / first-sentence-text, P1-5), `event: turn` (`{sessionId, state}`: idle / listening / finalizing / agent-speaking, P2-4), `event: mode` (global single-active ownership), `event: asr-progress / asr-ready / asr-error / tts-error` |
 | `POST /voice-mode/toggle` | `{sessionId, on}` enter/exit voice mode (globally single-active) |
-| `POST /voice-mode/asr` | Raw f32 LE 16k PCM payload → `{text}` (streaming zipformer2); returns `202 {loading}` until the model is ready; `?reset=1` discards the in-flight segment (used on wake-word hit) |
+| `POST /voice-mode/asr` | f32 LE 16k PCM payload → `{text, endpoint?}` (streaming zipformer2; `endpoint: true` = host-side Silero VAD end-point detection, P2-1); on `final=1` the utterance is also re-transcribed with SenseVoice (int8, punctuation + ITN, P4-1) — falls back to the streaming result when unavailable; returns `202 {loading}` until the model is ready; `?reset=1` discards the in-flight segment. `?offset=N` (P1-4): this packet is the segment slice starting at sample N — the client uploads only newly-appended samples (P1-4 incremental upload); omit for full-packets (backward compatible) |
 | `POST /voice-mode/cancel` | `{sessionId}` invalidates the TTS queue and drops the in-flight ASR segment |
 | `POST /voice-mode/preview` | `{voice, rate?}` one-shot synthesis preview → `audio/mpeg` (400 missing voice / voice too long; 502 synthesis failure, e.g. invalid ShortName; 403 when the plugin's `enabled=false`). Does not require voice mode to be active; uses an isolated synthesis connection and does not affect the reading queue |
 | `GET /voice-mode/config` | Client bootstrap parameters (silence threshold / sensitivity / voice and rate, etc.) |
@@ -158,7 +156,7 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
 
 ## Model & cache
 
-- Recognition model: `csukuangfj/sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30` (encoder ≈154 MB / decoder / joiner / tokens, ~160 MB total), running host-side via sherpa-onnx (Node WASM, Apache-2.0, natively cross-platform)
+- Recognition models (all host-side via sherpa-onnx Node WASM, Apache-2.0): streaming `csukuangfj/sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30` (~160 MB, encoder ≈154 MB / decoder / joiner / tokens); VAD `csukuangfj/vad/silero_vad.onnx` (~2 MB, P2); SenseVoice `csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17/model.int8.onnx` (~228 MB, P4 finalize, opt-out via settings)
 - Cache directory defaults by platform:
   - **Windows**: `%LOCALAPPDATA%\dsh-voice-mode\models`
   - **macOS / Linux**: `~/.cache/dsh-voice-mode/models`
@@ -170,7 +168,7 @@ You can also edit the `voice-mode:` section of `~/.dsh/settings.yaml` directly (
 ![architecture](https://raw.githubusercontent.com/qishuilalala/dsh-voice-mode/HEAD/plugin/dsh-voice-mode/assets/architecture.svg)
 
 ```
-input:  mic ──RMS VAD (2s silence split)──▶ POST /voice-mode/asr (f32 PCM, 16k, incremental)
+input:  mic ──local RMS gate──▶ POST /voice-mode/asr (f32 PCM, 16k, incremental; host Silero VAD endpoints + isSpeech interrupt)
                                             │ zipformer2 streaming ASR (host-side WASM)
                                             ▼
         composer draft ──autoSend──▶ model stream ──llm/stream tap (active voice session only)
@@ -186,14 +184,15 @@ input:  mic ──RMS VAD (2s silence split)──▶ POST /voice-mode/asr (f32
 
 ## Known limitations
 
-- Barge-in relies on browser echo cancellation (`echoCancellation`); loud speaker volume may leak into the mic (no JS-level AEC)
+- Barge-in now includes NLMS acoustic echo cancellation (P3, reference = this page's TTS playback, cross-correlation bulk-delay estimate + 64 ms short filter) plus host-side Silero VAD frame-level detection; browser `echoCancellation` is only a fallback. Under extreme speaker volume/distance the suppression needs on-device calibration (`echoGateDb` etc.)
 - `Ctrl+Shift+V` overrides the browser's "paste as plain text" shortcut (normal `Ctrl+V` paste still works)
 - The recognition model prioritizes Simplified Chinese; recognition quality is affected by ambient noise
 - Browser autoplay policy: reading requires prior user interaction on the page (clicking the mic satisfies it); if the browser blocks playback and the status bar shows no hint, make sure the page is foregrounded and not muted
-- **The wake word is a lightweight implementation** (text matching on the streaming transcript, not a dedicated KWS engine): it may lag or misfire in noisy environments; the wake word itself never enters the chat (the buffer is dropped on hit)
 - In hold mode, switching windows/tabs while holding **abandons the segment** (prevents continuous recording); come back and hold again
 - The hero (new-session empty state) has no voice entry: voice mode is a session-level feature; enter a session first and use the mic button in the input toolbar
 - The preview request timeout uses `AbortSignal.timeout` (Chrome 103+ / Firefox 100+ / Safari 16+); on older browsers clicking preview immediately shows a failure hint — an expected degradation
+- **Security**: the plugin HTTP surface (`/voice-mode/*`) follows the host security model — do not expose the dsh port to the public internet; when publishing behind a reverse proxy, add auth at the proxy layer (e.g. basic auth). `/asr` validates the active voice session (403 otherwise); `/toggle` `/cancel` `/preview` are unauthenticated availability surfaces (they can only flip voice-mode state / stop TTS; a local malicious page could trigger that, with no privilege escalation or eavesdropping — reads are CORS-restricted)
+- **Safari / iOS**: requires HTTPS or localhost (secure context for the mic); first entry needs mic permission (iOS: Settings → Safari → Microphone); recognition/reading pause in the background on iOS and resume on return (may drop a sentence) — keep the page in the foreground during voice mode
 
 ## Troubleshooting
 
@@ -237,7 +236,7 @@ changing deps, run `npm pack --dry-run` and `pnpm test` as regression.
 
 ```sh
 pnpm install && pnpm build    # esbuild: lib/index.js (host) + lib/client.js (browser)
-pnpm test                     # segmenter/wakeword unit tests + pre-release self-check (no network)
+pnpm test                     # unit tests (segmenter/aec + pre-release self-check, no network; tests import src/*.ts directly — needs Node ≥22.18, type stripping)
 node test/hold-e2e.js         # hold-mode acceptance (standalone browser, /asr route interception)
 systemctl restart dsh         # Linux; restart the dsh process on other platforms
 ```
@@ -245,25 +244,34 @@ systemctl restart dsh         # Linux; restart the dsh process on other platform
 > Note: dsh installs the plugin as a pnpm `file:` link (directory copy); after `node build.mjs` you
 > must copy `lib/client.js` to `<profile>/node_modules/dsh-voice-mode/lib/` and restart dsh before
 > the browser picks up the new bundle.
+>
+> Dev-mode latency telemetry (P1-5): run `localStorage.setItem('dsh-voice-mode.telemetry', '1')` in the
+> browser console, refresh, and enter voice mode — the status bar shows per-stage latencies of the
+> speech-end → first-audio chain (end → endpoint → submit → 1st token → 1st sentence → 1st chunk →
+> 1st audio, plus the total), for P1 acceptance measurement. Remove the key to disable (off by default, zero collection).
 
 ### Structure
 
 ```
-src/index.ts      host: single-active pointer, llm/stream tap, SSE, settings registration
-src/asr-host.ts   host: zipformer2 streaming ASR + lazy model download (.part resume)
-src/tts-queue.ts  host: per-session TTS queue + epoch barge-in
+src/index.ts      host: single-active pointer, llm/stream tap, SSE, settings registration, turn state machine
+src/asr-host.ts   host: zipformer2 streaming ASR + Silero VAD endpoint + SenseVoice finalize + lazy model download (.part resume)
+src/tts-queue.ts  host: per-session TTS queue + chunked forwarding + epoch barge-in
 src/segmenter.ts  host: sentence segmentation (markdown stripping + terminating punctuation)
-src/client.tsx    client: mic button + status bar + reading overlay + barge-in
-src/asr.ts        client: getUserMedia + RMS VAD + partial polling
+src/asr.ts        client: getUserMedia + AEC-injected capture + VAD segmentation + incremental upload + endpoint handling
+src/aec.ts        client: NLMS acoustic echo cancellation (pure module, unit-tested)
+src/resample.ts   client: linear resampling (capture / echo reference)
+src/client.tsx    client: mic button + status bar + overlay + playback engine (Web Audio queue) + barge-in
+src/settings-form.tsx client: settings card (Plugins → plugin config)
+scripts/bench-asr.mjs   offline CER/latency/size comparison of streaming ASR models
 scripts/prefetch.mjs  model pre-download (cross-platform cache dir + resume)
 test/segmenter.test.mjs  sentence segmentation unit tests
-test/wakeword.test.mjs    wake-word matching unit tests
+test/aec.test.mjs         NLMS echo cancellation numeric tests (synthetic)
 test/verify-client.mjs   pre-release self-check (bundle manifest/exports/shape)
 test/hold-e2e.js          hold-mode end-to-end acceptance (standalone browser)
 scripts/list-voices.mjs   print all Edge TTS voices (source of the voice table)
 ```
 
-Integration probes (`hold-e2e.js`, `spoken-prompt-rpc.sh`, `spoken-toggle-ui-check.js`) live in the repo root `test/`, outside this npm package.
+Integration probes (`hold-e2e.js`, `capture-e2e.js`, `asr-e2e.js`, `output-e2e.js`, …) live in the repo root `test/`, outside this npm package; `capture-e2e.js` self-checks the capture→partial→final loop with a fake mic.
 
 ## License
 

package/README.md

@@ -15,14 +15,14 @@ DeepSeek Harness 语音双工对话模式：会话内一键进入 → 边说边
 
 - **语音模式**：输入框工具排麦克风按钮或全局快捷键 `Ctrl+Shift+V` 进入/退出；全局单活（同一时刻仅一个会话处于语音模式，切换会话自动让出）
 - **两种交互模式（设置可切换）**：
-  - `toggle`（默认）持续聆听：RMS VAD 分段 → zipformer2 流式识别（边说边出字，实时字幕预览）→ 静音 2 秒自动断句进草稿并自动发送；按住 `Ctrl` 强制立即发送
+  - `toggle`（默认）持续聆听：本地 RMS 门控开段 → zipformer2 流式识别（边说边出字，实时字幕预览）→ 静音 700ms 自动断句进草稿并自动发送（不足 250ms 语音视为噪声不判句；端点判定由 host 侧 Silero VAD 神经网络完成，客户端静音计时兜底）；按住 `Ctrl` 强制立即发送
   - `hold` 按住说话：短按进入/退出，**按住麦克风按钮说话、松手即发**（滑出取消、`Esc`/失焦放弃本段）；`Ctrl` 按住即录、松开即发
-- **唤醒词（可选，默认关）**：设置 `wakeWord` 后进入待机态，说出唤醒词才开始识别（如「你好小D」）
-- **输出链路**：只朗读最终答复的 `text-delta`（reasoning/工具调用不读），按句流式 Edge TTS 朗读 + 右下角实时字幕浮层；工具调用触发提示音；全文照常写入聊天记录；可选口语化提示词（设置 `spokenFormat`，默认关）让回复为自然短句、不带 Markdown 排版符号，朗读侧再做一轮标记剥离
-- **开口打断（barge-in）**：三档灵敏度发声前沿检测 → 本地静音 + host 合成队列作废 + 正在运行的回合取消（保留半截并自然续入新消息）
-- **模型懒加载与进度**：首次使用自动下载 zipformer2 中文流式模型（约 160MB，`.part` 断点续传），状态条实时显示进度；可用 `npm run prefetch` 预下载
-- **设置**：设置 → Plugins → 插件配置 → 语音模式（voice-mode），可调音色/语速/打断灵敏度/静音停顿/空闲超时/模型镜像/自动发送/交互模式/唤醒词/口语化提示词；**音色可试听**（按当前音色+语速即时合成预览，自定义 ShortName 亦可）
-- **界面语言**：跟随浏览器语言（中文 / English；切换后刷新页面生效）
+- **输出链路**：只朗读最终答复的 `text-delta`（reasoning/工具调用不读），按句流式 Edge TTS 朗读 + 右下角实时字幕浮层；全文照常写入聊天记录；可选口语化提示词（设置 `spokenFormat`，默认关）让回复为自然短句、不带 Markdown 排版符号，朗读侧再做一轮标记剥离
+- **开口打断（barge-in）**：**NLMS 回声消除（P3，互相关 bulk delay 估计 + 64ms 短滤波器）**以本页 TTS 播放为参考消掉外放回声后，**host 端 Silero VAD 帧级检测（isSpeech）随 partial 下行驱动打断**——三档灵敏度对应发声确认时长（0 高门槛约 0.3s / 1 中 0.2s / 2 低 0.1s）；`bargeInMode` 选 `auto` 时开口自动打断，选 `manual`（外放推荐）时改按住麦克风/Ctrl 显式打断 → 本地静音 + host 合成队列作废 + 正在运行的回合取消（保留半截并自然续入新消息）
+- **模型懒加载与进度**：首次使用自动下载 zipformer2 流式模型（~160MB）、Silero VAD（~2MB）与 SenseVoice 定稿模型（~228MB，首次定稿时，可关），全部 `.part` 断点续传、hf-mirror 回退；状态条实时显示进度；`npm run prefetch` 可预下载流式 ASR 模型（VAD/SenseVoice 首次使用时自动下载）
+- **SenseVoice 定稿解码在独立 worker 线程执行**：228MB 离线模型的载入与整段解码不占用宿主主线程事件循环（解码期间其它请求/SSE 保持响应），10s 超时兜底真实可触发，失败/超时/关闭均自动降级 zipformer 定稿，worker 崩溃后下一句自动重建
+- **设置**：设置 → Plugins → 插件配置 → 语音模式（voice-mode），可调音色/语速/打断灵敏度/打断方式/静音停顿/空闲超时/模型镜像/自动发送/交互模式/口语化提示词；**音色可试听**（按当前音色+语速即时合成预览，自定义 ShortName 亦可）
+- **界面语言**：跟随 dsh 语言设置（网页 `<html lang>`，未设置时回退浏览器语言；切换后刷新页面生效）
 - **容错**：麦克风被拒红点提示、模型下载失败可见提示、TTS 连接失败状态条提示（自动退避重试）、提交失败文字留在草稿、SSE 断线自动重连
 - **空闲退出**：10 分钟无活动自动退出并释放麦克风
 
@@ -36,12 +36,20 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
 
 ## 操作手势
 
+**快速真机验收（第一步，约 3 分钟）**：
+
+1. 进入任一会话，`Ctrl+Shift+V` 或点输入栏麦克风按钮进入语音模式（状态条显示「聆听中…」）；
+2. 说一句完整的话（如「帮我看看今天天气」）→ 状态条实时字幕应在 ≤400ms 出现、停顿后自动发送（或按住 `Ctrl` 立即定稿）；
+3. AI 回复开始朗读时开口打断 → 朗读即刻停止、你说的话进入识别；此时查看 设置 → 语音模式 →「语音模型」区块应显示三模型状态（第一次用会显示下载进度）。
+
+完整验收清单（P1-P4 指标/回归/平台）见 `.scratch/voice-duplex-upgrade/acceptance-checklist.md`。
+
+
 | 手势 | 作用 |
 | --- | --- |
 | `Ctrl+Shift+V` | 进入 / 退出语音模式 |
-| 直接说话 | `toggle`：边说边出字，停顿 2 秒自动发送；按住 `Ctrl` 强制立即发送 |
+| 直接说话 | `toggle`：边说边出字，停顿 700ms 自动发送；按住 `Ctrl` 强制立即发送 |
 | 按住麦克风按钮 | `hold`：松手发送；短按退出；滑出 / `Esc` / 失焦放弃本段 |
-| 说唤醒词 | 待机态激活识别（配置后） |
 | AI 朗读时开口说话 | 打断朗读并取消当前回合 |
 | 点状态条「退出」 | 退出语音模式 |
 | 点字幕浮层「跳过」 | 跳过当前句朗读 |
@@ -52,16 +60,26 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
 | --- | --- | --- |
 | `voice` | `zh-CN-XiaoxiaoNeural` | Edge TTS 音色（见下方常用音色表），**即时生效**；行内「试听」按钮可即时预览 |
 | `rate` | `1.0` | 朗读语速倍率（0.5 慢速 ～ 2.0 快速），**即时生效** |
-| `interruptLevel` | `0` | 发声打断灵敏度：0 高门槛 / 1 中 / 2 低 |
-| `silenceMs` | `2000` | 说完整一句的静音停顿毫秒数 |
+| `interruptLevel` | `0` | 发声打断灵敏度：0 高门槛 / 1 中 / 2 低（发声确认约 0.3/0.2/0.1 秒） |
+| `bargeInMode` | `auto` | 打断方式：`auto` 自动打断（开口即打断，耳机/安静环境）；`manual` 手动打断（**外放推荐**——外放回声会误触发自动打断导致自打断静音，改按住麦克风/Ctrl 显式打断，永不自打断） |
+| `echoGateDb` | `6` | 回声门控阈值（dB）：自动打断要求残差高于回声地板此值；外放仍误打断调大（8~10），太难打断调小（3~4） |
+| `shortcut` | `Ctrl+Shift+V` | 进入/退出语音模式的快捷键（形如 Ctrl+Shift+V；留空禁用快捷键，只用麦克风按钮） |
+| `silenceMs` | `700` | 说完整一句的静音停顿毫秒数（至少 250ms 语音才判句） |
 | `idleTimeoutMinutes` | `10` | 无活动自动退出语音模式的分钟数 |
 | `modelHost` | 默认源 | ASR 模型下载源（国内网络填 `https://hf-mirror.com`） |
 | `autoSend` | `true` | 识别定稿后自动发送；关闭则只进草稿（按住 `Ctrl` / hold 松手仍会发送） |
-| `mode` | `toggle` | 交互模式：`toggle` 持续聆听 + 2s 静音断句；`hold` 按住说话、松手发送（短按退出） |
-| `wakeWord` | 空（关） | 唤醒词（如「你好小D」）：进入后先说唤醒词激活，避免误触；空 = 关闭 |
+| `autoResume` | `false` | 切换回上次语音会话时自动恢复语音模式（需麦克风权限已授予；省去每次切换会话后重新点麦克风） |
+| `mode` | `toggle` | 交互模式：`toggle` 持续聆听 + 静音自动断句；`hold` 按住说话、松手发送（短按退出） |
 | `spokenFormat` | `false` | 语音会话注入口语化提示词：开启后**仅当前语音会话**的回复被注入「口语化短句、不用 Markdown 排版符号」提示词（朗读更顺），**即时生效** |
+| `senseVoice` | `true` | 定稿用 SenseVoice 重译（带标点 + 数字归一化，识别更准）；关闭可省 228MB 模型、只走流式识别 |
 
-生效范围：`voice`/`rate`/`spokenFormat` **立即生效**；其余设置下次进入语音模式时生效。设置项默认值由插件配置（`base` 层）提供。
+生效范围：`voice`/`rate`/`spokenFormat`/`senseVoice` **立即生效**；其余（`silenceMs`/`interruptLevel`/`bargeInMode`/`idleTimeoutMinutes`/`modelHost`/`autoSend`/`autoResume`/`mode`）下次进入语音模式时生效。前六个键的平台默认由插件配置（`base` 层）提供，其余由 schema 提供。
+
+### 开发工具
+
+- `node scripts/list-voices.mjs`：列出 Edge TTS 音色
+- `node scripts/prefetch.mjs`：预下载流式 ASR zipformer 模型（VAD/SenseVoice 首次使用时自动下载）
+- `node scripts/bench-asr.mjs --dir <测试集目录>`（P4-2）：离线 CER/段延迟/体积对照（现役 zipformer int8 vs xlarge / small-CTC / 在线 Paraformer），测试集 = 16k 单声道 PCM `.wav` + 同名 `.txt` 参考文本
 
 ### 常用音色（完整清单见 `node scripts/list-voices.mjs`）
 
@@ -85,8 +103,8 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
 ## 工作原理
 
 ```
-麦克风(16kHz, AEC) ─▶ 浏览器 VAD 分段 ─▶ HOST zipformer2 流式识别(本地 WASM)
-                                        │
+麦克风(16kHz, AEC) ─▶ 本地 RMS 门控 ─▶ HOST zipformer2 流式识别(本地 WASM)
+                                   │         Silero VAD 端点断句 + isSpeech 打断判定
 用户说话 ◀── 打断 ◀── 音箱 ◀── Edge TTS 逐句合成 ◀── 分句(text-delta 过滤)
 ```
 
@@ -96,11 +114,10 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
 
 ## 已知限制
 
-- 发声打断依赖浏览器回声消除（`echoCancellation`）；扬声器音量过大时可能漏声到麦克风
+- 打断已内置 NLMS 声学回声消除（P3，参考=本页 TTS 播放，互相关 bulk delay 估计 + 64ms 短滤波器）+ host 端 Silero VAD 帧级检测；浏览器 `echoCancellation` 仅兜底；外放极端音量/距离下抑制量需真机标定（`echoGateDb` 等参数）
 - `Ctrl+Shift+V` 会覆盖浏览器「粘贴纯文本」快捷键（普通粘贴仍可用 `Ctrl+V`）
 - 识别模型为简体中文优先；识别质量受环境噪声影响
 - 浏览器自动播放策略：朗读需要页面已有用户交互（点击麦克风即满足）；「试听」依赖 `AbortSignal.timeout`（Safari 16+ / Chrome 103+ / Firefox 100+；老浏览器点击试听会立即提示失败，属预期降级）
-- **唤醒词为轻量实现**（流式文本匹配，非专用 KWS 引擎）：嘈杂环境可能延迟或误激活；唤醒词本身不会进入聊天
 - hold 模式按住时切换窗口/标签页会**放弃本段**（防持续收音）
 - hero（新会话空态）无语音入口：请先进入会话使用麦克风按钮
 - `spokenFormat` 提示词经官方 `system-prompt/assemble` 瀑布注入；若当前会话使用**完整提示词**配置（persona `complete: true` 的 agent preset），提示词不注入（官方 complete 契约优先）
@@ -108,7 +125,7 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
   - 需 **HTTPS 或 localhost**（iOS/macOS Safari 强制安全上下文；`http://` 局域网 IP 下麦克风不可用）
   - 首次进入需授权麦克风；被拒后到「设置 → Safari → 麦克风」开启（iOS）
   - iOS 后台/锁屏时识别与朗读暂停，回前台自动恢复（可能丢句）；建议语音模式期间保持前台
-- **安全说明**：插件 HTTP 面（`/voice-mode/*`）遵循宿主安全模型——请勿将 dsh 端口直接暴露公网；经反向代理发布时由代理层（如 basic auth）鉴权；插件侧对敏感操作保留会话归属校验
+- **安全说明**：插件 HTTP 面（`/voice-mode/*`）遵循宿主安全模型——请勿将 dsh 端口直接暴露公网；经反向代理发布时由代理层（如 basic auth）鉴权；`/asr` 有活跃会话归属校验（403），`/toggle` `/cancel` `/preview` 为无鉴权可用性面（仅影响语音模式状态/TTS，本机恶意网页可触发让出/停播，无提权或窃听；读侧受 CORS 限制）
 
 ## 故障排查
 
@@ -118,26 +135,31 @@ bundle 插件安装后需重启 dsh 生效（Linux：`systemctl restart dsh`；
 | 状态条「正在加载模型… x%」卡住 | 检查网络；模型大（160MB）可先 `npm run prefetch`；国内网络 `modelHost` 配 `https://hf-mirror.com` |
 | 朗读无声音/无字幕 | 查看状态条「朗读连接失败：正在重试…」（Edge TTS 网络问题，自动退避重试）；确认页面前台且未静音 |
 | 语音模式进不去 | 检查插件 `enabled`；多标签页时确认当前会话为活动会话 |
-| 识别到但不是我要说的 | 环境噪声或唤醒词误判：降低音量、提高 `interruptLevel`（高门槛）或启用 `wakeWord` |
+| 识别到但不是我要说的 | 环境噪声：降低音量或提高 `interruptLevel`（高门槛） |
 
 ## 开发
 
 ```sh
 pnpm install && pnpm build    # esbuild：lib/index.js（host）+ lib/client.js（browser）
-pnpm test                     # segmenter/wakeword 单测 + 发布前自检（无需网络）
+pnpm test                     # 单测（segmenter/aec + 发布前自检，无需网络；测试直接 import src/*.ts，需 Node ≥22.18，即 type stripping）
 systemctl restart dsh         # 本机加载新 host 代码；其他平台重启 dsh 进程
 ```
 
-> 注意：dsh 安装的是 pnpm `file:` 链接（目录拷贝），改完 `node build.mjs` 后需把 `lib/client.js` 同步到 `<profile>/node_modules/dsh-voice-mode/lib/` 再刷新页面（`lib/index.js` 与工作区为同一文件自动同步）。集成探测脚本（`test/hold-e2e.js`、`test/spoken-prompt-rpc.sh`、`test/spoken-toggle-ui-check.js`）位于仓库根 `test/`，不在 npm 包内。
+> 注意：dsh 安装的是 pnpm `file:` 链接（目录拷贝），改完 `node build.mjs` 后需把 `lib/client.js` 同步到 `<profile>/node_modules/dsh-voice-mode/lib/` 再刷新页面（`lib/index.js` 与工作区为同一文件自动同步）。集成探测脚本（`test/hold-e2e.js`、`test/capture-e2e.js`、`test/asr-e2e.js`、`test/output-e2e.js`、`test/make-real-voice.mjs` + `test/sherpa-direct.js` + `test/realvoice-e2e.js`）位于仓库根 `test/`，不在 npm 包内；`capture-e2e.js` 用 fake 麦克风做「采集→partial→final」闭环自检，`make-real-voice.mjs` 合成真实中文语音、`realvoice-e2e.js` 用 Chrome file-fake-audio 走**真实语音→浏览器→host ASR→文本**全链路（实测词级准确）。
+
+> 开发模式延迟埋点（P1-5）：浏览器控制台执行 `localStorage.setItem('dsh-voice-mode.telemetry', '1')` 后刷新页面，进入语音模式时状态条会实时显示「说完 → 端点 → 定稿 → 首Token → 首句 → 首chunk → 首音」各段耗时与合计（说完→首音），供 P1 延迟验收测量；`localStorage.removeItem('dsh-voice-mode.telemetry')` 关闭（默认关闭，零采集）。
 
 ```
-src/index.ts      host：单活指针、llm/stream tap、SSE、settings 注册、口语化提示词注入
-src/asr-host.ts   host：zipformer2 流式识别 + 模型懒下载（.part 断点续传）
-src/asr.ts        client：音频采集、VAD 分段、增量识别、唤醒词
-src/client.tsx    client：麦克风按钮 + 状态条 + 字幕浮层 + 打断
-src/tts-queue.ts  host：逐会话 TTS 队列 + epoch 打断机制
+src/index.ts      host：单活指针、llm/stream tap、SSE、settings 注册、turn 回合状态机、口语化提示词注入
+src/asr-host.ts   host：zipformer2 流式识别 + Silero VAD 端点 + SenseVoice 定稿 + 模型懒下载（.part 断点续传）
+src/tts-queue.ts  host：逐会话 TTS 队列 + 逐 chunk 转发 + epoch 打断机制
 src/segmenter.ts  host：句子切分（markdown 剥离 + 终止标点）
-src/strings.ts    client：中英文案字典（navigator.language）
+src/asr.ts        client：音频采集、NLMS AEC 注入、VAD 分段、增量识别、端点处理
+src/aec.ts        client：NLMS 声学回声消除（纯模块，可单测）
+src/resample.ts   client：线性重采样（采集/回声参考共用）
+src/client.tsx    client：麦克风按钮 + 状态条 + 字幕浮层 + 播放引擎（Web Audio 队列）+ 打断
+src/settings-form.tsx client：设置卡片（Plugins → 插件配置）
+src/strings.ts    client：中英文案字典（以 dsh 语言设置 <html lang> 为准）
 ```
 
 ## License

package/assets/architecture.svg

@@ -55,7 +55,7 @@
 
   <rect class="node-blue" x="225" y="137" width="175" height="82"/>
   <text class="node-title" x="312.5" y="160">RMS VAD</text>
-  <text class="node-detail" x="312.5" y="179">持续聆听 2s 静音断句 ·</text>
+  <text class="node-detail" x="312.5" y="179">持续聆听 · VAD 自动断句 ·</text>
   <text class="node-detail" x="312.5" y="197"><tspan class="mono">hold</tspan> 模式按住说话绕过 VAD</text>
 
   <rect class="node-blue" x="430" y="127" width="205" height="102"/>