dsh-voice 0.8.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/README.en.md +17 -15
- package/README.md +17 -15
- package/docs/USAGE.en.md +39 -16
- package/docs/USAGE.md +41 -18
- package/lib/client.js +5067 -0
- package/lib/config.d.ts +7 -0
- package/lib/config.js +21 -0
- package/lib/edge-tts.d.ts +4 -0
- package/lib/edge-tts.js +38 -1
- package/lib/host-config.js +7 -0
- package/lib/index.d.ts +3 -1
- package/lib/index.js +4 -1
- package/lib/openai-tts.d.ts +19 -6
- package/lib/openai-tts.js +163 -2
- package/lib/stt.d.ts +19 -0
- package/lib/stt.js +157 -9
- package/lib/tools.d.ts +4 -2
- package/lib/tools.js +380 -70
- package/lib/tts-text.d.ts +46 -3
- package/lib/tts-text.js +345 -16
- package/lib/web.d.ts +66 -0
- package/lib/web.js +677 -0
- package/locale/en.json +1 -1
- package/locale/zh.json +1 -1
- package/package.json +10 -5
- package/skills/voice-workflow/SKILL.md +51 -19
- package/lib/update-client.js +0 -745
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
# 更新记录
|
|
2
2
|
|
|
3
|
+
## 0.9.0 (2026-11-18)
|
|
4
|
+
|
|
5
|
+
- 新增原生 5 主题 Voice Studio 侧边栏工作台(原生清透 / 米白纸感 / Claude 暖书卷 / 8-Bit 像素 / 晴空薄荷)与 Coachmark 新手引导:支持在线试听 300+ 音色与 10 套人设、0.8x–1.5x 倍速播放、浏览器麦克风速记转写、自定义角色与发音词典管理,并集成 `@stardustlc/dsh-updates` 底部更新控件。
|
|
6
|
+
- Add native 5-theme Voice Studio sidebar workbench (Native Airy / Warm Cream / Claude Warmth / Bright 8-Bit / Airy Mint) with interactive Coachmark onboarding: in-panel audition of 300+ voices & 10 personas with 0.8x–1.5x playback speed, browser `MediaRecorder` mic memo transcription, custom voice aliases & pronunciation lexicon management, and `@stardustlc/dsh-updates` footer controls.
|
|
7
|
+
- 精准 MP3 帧与 Xing/VBRI VBR 时长解析(`inspectMp3Buffer`)与匹配采样率静音帧生成(`createSilenceMp3`):`voice_tts` 返回精确数值 `durationMs` 与 `durationSec`,助力精准卡点配音。
|
|
8
|
+
- Exact MP3 frame & Xing/VBRI VBR duration inspector (`inspectMp3Buffer`) and matched-sample-rate silence frames (`createSilenceMp3`): `voice_tts` now returns accurate numeric `durationMs` and `durationSec` for tight voiceover timing.
|
|
9
|
+
- 发音词典(`pronunciations`):Edge 引擎自动生成单趟 SSML `<sub alias>`(字幕保留 `Nginx` 原文,朗读按 `Engine X` 发音),OpenAI/SiliconFlow 引擎执行非递归纯文本替读,并支持宿主原生热更新配置。
|
|
10
|
+
- Pronunciation lexicon (`pronunciations`): emits single-pass SSML `<sub alias>` on Edge (retaining original terms like `Nginx` in subtitles while speaking `Engine X`), performs non-recursive plain-text replacement on OpenAI/SiliconFlow, and supports volatile live-reloaded DSH settings.
|
|
11
|
+
- 反向 `.srt` / `.vtt` 字幕时间轴卡点配音(`subtitleInput`):按视频字幕时间窗自动计算 `scaleRatePercent` 加速重合成并精确填充前导与句间静音。
|
|
12
|
+
- Reverse `.srt` / `.vtt` timeline-synced video dubbing (`subtitleInput`): automatically computes per-cue `scaleRatePercent` speedup re-synthesis when over-window and pads exact leading/inter-cue silence.
|
|
13
|
+
- 播客 BGM 自动闪避混音(`bgmFile` / `bgmVolume` / `bgmDucking`)、说话人区分(`voice_stt` `diarize: true` + 双声道分离)与 SiliconFlow SHA-256 参考音频 URI 缓存上传(`/uploads/audio/voice` 分段仅上传一次)。
|
|
14
|
+
- Podcast BGM auto-ducking (`bgmFile` / `bgmVolume` / `bgmDucking` via FFmpeg `sidechaincompress`), speaker diarization (`voice_stt` `diarize: true` with stereo channel splitting), and SiliconFlow SHA-256 reference voice URI upload caching (`/uploads/audio/voice` uploaded once per reference clip).
|
|
15
|
+
|
|
3
16
|
## 0.8.1 (2026-11-18)
|
|
4
17
|
|
|
5
18
|
- 升级共享更新组件至 `@stardustlc/dsh-updates@0.2.4`:对话页新增一键热更新提醒卡片,自动检查周期缩短至 6 小时,手动热更新后可顺手开启包含兼容功能小版本的空闲自动更新。
|
package/README.en.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
**Turn text into natural speech and recordings into clean transcripts.**
|
|
6
6
|
|
|
7
|
-
Voice plugin for DeepSeek Harness (`>=0.2.2-rc.3`):
|
|
7
|
+
Voice Studio workbench plugin for DeepSeek Harness (`>=0.2.2-rc.3`): includes a **native 5-theme sidebar Voice Studio workbench** (Native Airy / Warm Cream / Claude Warmth / Bright 8-Bit / Airy Mint with interactive Coachmark tour) alongside zero-key Edge neural TTS (300+ live online voices, 45+ all-age/dialect built-ins, 10 personas) and OpenAI / SiliconFlow / custom engines (emotion acting, Kokoro blending, SHA-256 cached zero-shot cloning). Supports **exact MP3 duration feedback (`durationMs` / `durationSec`)**, **pronunciation lexicon (`pronunciations`)**, **reverse `.srt`/`.vtt` video dubbing (`subtitleInput`)**, **podcast BGM auto-ducking & `-16 LUFS` mastering**, and **speaker-diarized (`diarize`)** multi-engine STT strictly confined to your session workspace.
|
|
8
8
|
|
|
9
9
|
[](https://www.npmjs.com/package/dsh-voice) [](https://www.npmjs.com/package/dsh-voice) [](https://www.npmjs.com/package/dsh-voice)
|
|
10
10
|
|
|
@@ -19,10 +19,10 @@ Voice plugin for DeepSeek Harness (`>=0.2.2-rc.3`): provides zero-key Edge neura
|
|
|
19
19
|
|
|
20
20
|
## Highlights
|
|
21
21
|
|
|
22
|
-
- **
|
|
23
|
-
- **
|
|
24
|
-
- **
|
|
25
|
-
- **
|
|
22
|
+
- **Native 5-Theme Voice Studio Sidebar Workbench**: Switch live between 5 synced themes (**Native Airy / Warm Cream / Claude Warmth / Bright 8-Bit / Airy Mint**) with an interactive Coachmark onboarding tour. Audition 300+ Edge voices and 10 `persona` presets in-panel (`0.8x–1.5x` playback + duration badge), dictate with your browser microphone, and manage `customVoices` aliases and `pronunciations` rules visually.
|
|
23
|
+
- **Exact MP3 Duration, Pronunciation Lexicon & Reverse `.srt`/`.vtt` Dubbing**: Inspects CBR/VBR and multi-engine MP3 streams to return exact `durationMs` / `durationSec` with matched-sample-rate silence frames (for hitting exact 60s short-video voiceover targets). `pronunciations` compiles to SSML `<sub alias>` on Edge (subtitles keep `Nginx` while speech says `Engine X`) and plain non-recursive replacement on OpenAI/SiliconFlow. `subtitleInput` scales per-cue speech rate and pads millisecond silences to stay locked to `.srt`/`.vtt` video timelines.
|
|
24
|
+
- **Podcast BGM Auto-Ducking, Emotion Acting & SHA-256 Cached Cloning**: Pass `bgmFile`, `bgmVolume`, and `bgmDucking` alongside `segments` for FFmpeg `sidechaincompress` music ducking, smooth fade-out, and optional `-16 LUFS` loudness mastering. Use `instructions` for acting direction (`<|endofprompt|>` + `[laughter]`/`[breath]` on CosyVoice2, native on `gpt-4o-mini-tts`, Kokoro voice blending), while SiliconFlow caches uploaded `referenceAudio` URIs by SHA-256 hash instead of re-uploading megabytes per segment.
|
|
25
|
+
- **Long-form File Synthesis, Browser Auditions, Speaker Diarization & Proxies**: `inputFile` reads `.md`/`.txt` with automatic Markdown cleanup (up to 100k chars); `voice_preview` writes `voice_previews/index.html`; `voice_stt` supports `groq`/`openai`/`siliconflow`/keyless loopback `custom`, `autoCompress: true` for >25 MB audio, and `diarize: true` (FFmpeg stereo channel split or `[Speaker A]` turn labeling); native `http://`, `https://`, `socks5://`, `socks5h://` proxy support.
|
|
26
26
|
|
|
27
27
|
## Install
|
|
28
28
|
|
|
@@ -32,25 +32,27 @@ Open **Plugins** in the DSH desktop app and install `dsh-voice`, or use the CLI:
|
|
|
32
32
|
dsh plugin --profile desktop add dsh-voice
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
For the web profile, replace `desktop` with `web`. Restart DSH once after installing. Requires DSH `>=0.2.2-rc.3`. The plugin bundles the `voice-workflow` skill so your assistant automatically follows best practices for voice discovery, personas, cloning,
|
|
35
|
+
For the web profile, replace `desktop` with `web`. Restart DSH once after installing. Requires DSH `>=0.2.2-rc.3`. The plugin bundles the `voice-workflow` skill so your assistant automatically follows best practices for voice discovery, personas, cloning, subtitle dubbing, BGM podcasts, and diarized transcription.
|
|
36
36
|
|
|
37
37
|
<a id="start-using-it"></a>
|
|
38
38
|
## Get Started in 3 Steps
|
|
39
39
|
|
|
40
|
-
1. **
|
|
41
|
-
2. **Optional TTS engines (acting/cloning) or ASR keys (loopback Whisper is keyless)**: Open **DSH Settings → Plugins → Voice (`dsh-voice`)** — switch `ttsEngine` to `siliconflow`, `openai`, or `custom` for emotion `instructions` or `referenceAudio` cloning (
|
|
42
|
-
3. **Preview, synthesize, or transcribe inside your workspace**: Keep scripts, reference clips, and recordings inside your session workspace; generated MP3s, `.txt` transcripts, `.srt`/`.vtt` subtitles, and preview pages stay safely inside the workspace and auto-number instead of overwriting existing files.
|
|
40
|
+
1. **Open the Voice Studio sidebar workbench (zero API keys for TTS)**: Follow the Coachmark tour in the 5-theme sidebar workbench to audition 300+ Edge voices and 10 built-in `persona` presets, manage custom characters and pronunciation rules, or have the assistant narrate Markdown files, produce podcasts, and dub `.srt` timelines immediately.
|
|
41
|
+
2. **Optional TTS engines (acting/cloning) or ASR keys (loopback Whisper is keyless)**: Open **DSH Settings → Plugins → Voice (`dsh-voice`)** — switch `ttsEngine` to `siliconflow`, `openai`, or `custom` for emotion `instructions` or `referenceAudio` cloning (live on save). For browser mic dictation or transcription, pick `asrEngine` (`groq`, `openai`, `siliconflow`, or `custom`) and add `asrApiKey` (loopback `127.0.0.1`/`localhost` needs no key).
|
|
42
|
+
3. **Preview, synthesize, or transcribe inside your workspace**: Keep scripts, `.srt`/`.vtt` timelines, BGM clips, reference clips, and recordings inside your session workspace; generated MP3s, `.txt` transcripts, `.srt`/`.vtt` subtitles, and preview pages stay safely inside the workspace and auto-number instead of overwriting existing files.
|
|
43
43
|
|
|
44
44
|
## Copy-Ready Prompts for Everyday Work
|
|
45
45
|
|
|
46
46
|
- **Discover online voices & personas and audition in the browser**
|
|
47
47
|
> Fetch online voices tagged `podcast` and generate previews in `voice_previews/index.html` comparing them with `late-night-radio` and `bilingual-tech-female` so I can pick one.
|
|
48
|
-
- **Turn a Markdown report into a
|
|
49
|
-
> Read `weekly-report.md` in this workspace and turn it into a two-host dialogue using `podcast-host-female` and `podcast-host-male` with `
|
|
48
|
+
- **Turn a Markdown report into a two-host podcast with BGM auto-ducking**
|
|
49
|
+
> Read `weekly-report.md` in this workspace and turn it into a two-host dialogue using `podcast-host-female` and `podcast-host-male`, mix `assets/bgm.mp3` with `bgmDucking: true` and `normalizeLoudness: true`, and save `podcast.mp3` and `podcast.srt`.
|
|
50
|
+
- **Reverse-dub a `.srt` timeline with custom pronunciation rules**
|
|
51
|
+
> Dub `video.srt` in this workspace using `bilingual-tech-male`, pronouncing `Nginx` as `Engine X` while keeping `Nginx` in subtitles, save `dubbed.mp3` and `dubbed.srt`, and tell me the exact duration.
|
|
50
52
|
- **Zero-shot clone or emotion-directed voiceover (SiliconFlow / custom engine)**
|
|
51
53
|
> Clone the voice from `sample_ref.wav` (transcript: "Hello, I am Alex, nice to meet you") in this workspace and read this welcome script with a warm, cheerful tone, saving `welcome.mp3`.
|
|
52
|
-
- **Transcribe a large meeting recording with
|
|
53
|
-
> Transcribe `meeting.m4a` in this workspace (auto-compress if over 25 MB), save `meeting.txt` and `meeting.srt`, and list the action items.
|
|
54
|
+
- **Transcribe a large meeting recording with speaker diarization & subtitles**
|
|
55
|
+
> Transcribe `meeting.m4a` in this workspace with `diarize: true` (auto-compress if over 25 MB), save `meeting.txt` and `meeting.srt`, and list the action items.
|
|
54
56
|
|
|
55
57
|
## Quick Troubleshooting Checklist
|
|
56
58
|
|
|
@@ -58,8 +60,8 @@ For the web profile, replace `desktop` with `web`. Restart DSH once after instal
|
|
|
58
60
|
| :--- | :--- | :--- |
|
|
59
61
|
| Health check says `asrApiKey` is missing, but you only want TTS | ASR key is only needed for audio-to-text transcription | **Keep using TTS directly** — health check reports `ttsReady` / `sttReady` separately and supports `scope: "tts"` |
|
|
60
62
|
| `referenceAudio` fails saying cloning engine is required | Default `edge` engine does not support zero-shot reference cloning | Switch `ttsEngine` to `siliconflow` or `custom` in **DSH Settings → Plugins → Voice** (applies live) |
|
|
61
|
-
| `normalizeLoudness` fails saying `ffmpeg` is missing | `-16 LUFS`
|
|
62
|
-
| "Must stay inside the session workspace" error | Script, recording, reference clip, or output resolves outside workspace | Copy the file into the current DSH workspace folder first |
|
|
63
|
+
| `normalizeLoudness` or `bgmFile` fails saying `ffmpeg` is missing | `-16 LUFS` loudness mastering and BGM ducking use local `ffmpeg` | Install `ffmpeg` locally (or pre-process with `dsh-ffmpeg`), or omit `normalizeLoudness`/`bgmFile` |
|
|
64
|
+
| "Must stay inside the session workspace" error | Script, subtitle, BGM, recording, reference clip, or output resolves outside workspace | Copy the file into the current DSH workspace folder first |
|
|
63
65
|
| Network / connection error | Corporate firewall or proxy required to reach Edge TTS / ASR | Set `proxyUrl` (`http://`, `https://`, `socks5://`, or `socks5h://`) in **DSH Settings → Plugins → Voice** (applies live) |
|
|
64
66
|
| Audio exceeds 25 MB limit | Standard Whisper/ASR upload limit is 25 MB | Ask the assistant to enable `autoCompress: true` (uses local `ffmpeg`) or pre-process with `dsh-ffmpeg` |
|
|
65
67
|
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
**把文字变成自然人声,把会议录音转成干净文字。**
|
|
6
6
|
|
|
7
|
-
DeepSeek Harness(`>=0.2.2-rc.3
|
|
7
|
+
DeepSeek Harness(`>=0.2.2-rc.3`)语音工作台插件:原生提供 **5 套主题可切换的侧边栏语音工作台**(原生清透 / 米白纸感 / Claude 暖书卷 / 8-Bit 像素 / 晴空薄荷,附 Coachmark 新手引导),支持零密钥 Edge 神经网络语音合成(300+ 在线音色与 10 套人设)与 OpenAI / SiliconFlow / 自定义备用引擎(情绪演技、Kokoro 混音、SHA-256 缓存零样本克隆),具备 **精确 MP3 时长反馈(`durationMs` / `durationSec`)**、**发音词典替读(`pronunciations`)**、**`.srt` / `.vtt` 字幕反向卡点配音(`subtitleInput`)**、**播客 BGM 自动闪避与 `-16 LUFS` 响度均衡**,以及支持 **说话人区分(`diarize`)** 的多引擎语音转文字(STT),严格将读写限制在当前会话工作区内。
|
|
8
8
|
|
|
9
9
|
[](https://www.npmjs.com/package/dsh-voice) [](https://www.npmjs.com/package/dsh-voice) [](https://www.npmjs.com/package/dsh-voice)
|
|
10
10
|
|
|
@@ -19,10 +19,10 @@ DeepSeek Harness(`>=0.2.2-rc.3`)语音插件:内置零密钥 Edge 神经
|
|
|
19
19
|
|
|
20
20
|
## 核心亮点
|
|
21
21
|
|
|
22
|
-
-
|
|
23
|
-
-
|
|
24
|
-
-
|
|
25
|
-
-
|
|
22
|
+
- **原生 5 套主题 Voice Studio 侧边栏工作台**:支持 **原生清透 / 米白纸感 / Claude 暖书卷 / 8-Bit 像素 / 晴空薄荷** 5 套实时同步主题与 Coachmark 新手引导;在侧边栏即可在线试听 300+ Edge 音色与 10 套内置 `persona`(支持 `0.8x–1.5x` 倍速与实时时长显示)、使用浏览器麦克风语音速记转写,并可视化管理 `customVoices` 自定义角色与 `pronunciations` 发音词典。
|
|
23
|
+
- **精确 MP3 时长反馈、发音词典与 `.srt` / `.vtt` 字幕反向卡点配音**:跨 CBR/VBR 与多引擎流精确计算 `durationMs` / `durationSec` 并生成匹配采样率的静音帧(轻松卡准 60 秒短视频配音);`pronunciations` 在 Edge 自动编译为 SSML `<sub alias>`(字幕保留 `Nginx` 原拼写、语音读作 `Engine X`),在 OpenAI/SiliconFlow 自动做非递归替读;`subtitleInput` 支持按既有 `.srt` / `.vtt` 时间轴自动逐条测速加速并补齐毫秒静音间隙对齐画面。
|
|
24
|
+
- **播客 BGM 自动闪避、情绪演技与 SHA-256 缓存零样本克隆**:`segments` 编排多主持人对话时可传入 `bgmFile` / `bgmVolume` / `bgmDucking`(FFmpeg `sidechaincompress` 人声闪避 + 结尾平滑淡出 + 可选 `-16 LUFS` 响度均衡);`instructions` 透传演技指令(支持 CosyVoice2 `<|endofprompt|>`、`[laughter]`/`[breath]` 与 Kokoro 混音);SiliconFlow 克隆按参考音频 SHA-256 哈希自动缓存 `speech:...` URI,多段播客无需重复上传 MB 级参考音频。
|
|
25
|
+
- **长文直读、网页试听、说话人区分转写与代理**:`inputFile` 直读工作区 `.md` / `.txt` 并自动清理 Markdown(最高 10 万字自动按句分片);`voice_preview` 生成 `voice_previews/index.html` 交互式试听页;`voice_stt` 支持 `groq` / `openai` / `siliconflow` / `custom`(本地回环免密)、`autoCompress`(>25MB 自动压缩切片)与 `diarize: true`(双声道分离或说话人轮次标记 `[说话人 A]`);代理原生支持 `http://`、`https://`、`socks5://` 与 `socks5h://`。
|
|
26
26
|
|
|
27
27
|
## 安装
|
|
28
28
|
|
|
@@ -32,25 +32,27 @@ DeepSeek Harness(`>=0.2.2-rc.3`)语音插件:内置零密钥 Edge 神经
|
|
|
32
32
|
dsh plugin --profile desktop add dsh-voice
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
网页版把 `desktop` 改为 `web`;安装后重启一次 DSH。宿主要求 `>=0.2.2-rc.3`。插件附带 `voice-workflow`
|
|
35
|
+
网页版把 `desktop` 改为 `web`;安装后重启一次 DSH。宿主要求 `>=0.2.2-rc.3`。插件附带 `voice-workflow` 技能包,会在语音合成、人设配音、声音克隆、字幕卡点、播客混音与转写任务中自动指引助手采用最佳实践。
|
|
36
36
|
|
|
37
37
|
<a id="开始使用"></a>
|
|
38
38
|
## 最快 3 步用起来
|
|
39
39
|
|
|
40
|
-
1.
|
|
41
|
-
2. **按需配置备用 TTS(情绪/克隆)或 ASR 密钥(本地回环免密)**:打开 DSH「**设置 → 插件 → 语音(dsh-voice
|
|
42
|
-
3. **试听选声 →
|
|
40
|
+
1. **打开侧边栏 Voice Studio 工作台,零密钥即可开箱试听与配音**:装好插件后无需填任何密钥,即可在 5 主题侧边栏工作台跟随 Coachmark 引导试听 300+ Edge 音色与 10 种内置 `persona`、管理自定义角色与发音词典,并直接让助手直读 Markdown 长文、合成多角色播客或按 `.srt` 字幕卡点配音。
|
|
41
|
+
2. **按需配置备用 TTS(情绪/克隆)或 ASR 密钥(本地回环免密)**:打开 DSH「**设置 → 插件 → 语音(dsh-voice)**」或在工作台操作:如需情绪指令或零样本声音克隆,可切换 `ttsEngine` 为 `siliconflow` / `openai` / `custom`(保存即生效);如需使用浏览器麦克风速记或转写录音,选择 `asrEngine`(`groq` / `openai` / `siliconflow` / `custom`)并填入 `asrApiKey`(本地 `127.0.0.1` / `localhost` Whisper 免密)。
|
|
42
|
+
3. **试听选声 → 合成或转写交付**:把待转写录音、背景音乐、字幕时间轴、参考克隆音频或文稿放到当前会话工作区,生成的 MP3、`.txt` 纪要、`.srt` / `.vtt` 字幕与试听页面都会安全落在工作区,默认同名自动编号不误盖。
|
|
43
43
|
|
|
44
44
|
## 平时怎么对助手说(可直接复制的自然语言示例)
|
|
45
45
|
|
|
46
46
|
- **联网发现音色/人设并在浏览器试听页里对比**
|
|
47
47
|
> 联网查一下带 `podcast` 标签的中文和英文音色,连同 `late-night-radio`、`bilingual-tech-female` 人设一起生成短试听,用 `voice_previews/index.html` 让我挑选。
|
|
48
|
-
- **直接把 Markdown
|
|
49
|
-
> 直接读取工作区里的 `weekly-report.md`,用 `podcast-host-female` 和 `podcast-host-male`
|
|
48
|
+
- **直接把 Markdown 长文变成带 BGM 自动闪避的双人播客**
|
|
49
|
+
> 直接读取工作区里的 `weekly-report.md`,用 `podcast-host-female` 和 `podcast-host-male` 编排成双人播客,配上 `assets/bgm.mp3` 开启 BGM 自动闪避和 `-16 LUFS` 响度均衡,输出 `podcast.mp3` 和 `podcast.srt`。
|
|
50
|
+
- **按现有 `.srt` 字幕反向卡点配音 + 发音词典替读**
|
|
51
|
+
> 按工作区里的 `video.srt` 时间轴用 `bilingual-tech-male` 反向卡点配音(把 `Nginx` 替读成 `Engine X`,字幕仍保留 `Nginx`),导出 `dubbed.mp3` 和 `dubbed.srt`,并告诉我精确时长。
|
|
50
52
|
- **用工作区 5–10 秒参考音频做零样本克隆或情绪配音(SiliconFlow / 自定义引擎)**
|
|
51
53
|
> 用工作区里的 `sample_ref.wav`(台词是「你好,我是小智,很高兴认识大家」)作为参考声音,用轻快带笑意的语气朗读这段欢迎词,导出 `welcome.mp3`。
|
|
52
|
-
-
|
|
53
|
-
> 把当前目录里的 `meeting.m4a` 转写成中文(如果超过 25MB
|
|
54
|
+
- **把大体积会议录音转成带说话人区分的纪要与字幕**
|
|
55
|
+
> 把当前目录里的 `meeting.m4a` 转写成中文(如果超过 25MB 自动压缩分段,开启 `diarize: true` 区分说话人),同时输出 `meeting.txt` 和 `meeting.srt`,再提炼待办事项清单。
|
|
54
56
|
|
|
55
57
|
## 常见卡点自检
|
|
56
58
|
|
|
@@ -58,8 +60,8 @@ dsh plugin --profile desktop add dsh-voice
|
|
|
58
60
|
| :--- | :--- | :--- |
|
|
59
61
|
| 自检提示「缺 ASR 密钥」但只想做朗读配音 | ASR 密钥仅录音转文字需要,不影响朗读 | **直接正常合成语音即可**(自检可按 `tts` 范围单独查看 `ttsReady`) |
|
|
60
62
|
| 传入 `referenceAudio` 提示引擎不支持克隆 | 默认 `edge` 引擎不支持零样本参考音频克隆 | 在 DSH「设置 → 插件 → 语音」将 `ttsEngine` 切为 `siliconflow` 或 `custom`(保存即生效) |
|
|
61
|
-
| 开启 `normalizeLoudness` 报错缺工具 | `-16 LUFS`
|
|
62
|
-
| 提示「必须位于会话工作区内」 |
|
|
63
|
+
| 开启 `normalizeLoudness` 或 `bgmFile` 报错缺工具 | `-16 LUFS` 响度均衡与 BGM 自动闪避混音需要本机 `ffmpeg` | 本机安装 `ffmpeg`(或先用 `dsh-ffmpeg` 处理),未安装时先不开启该开关 |
|
|
64
|
+
| 提示「必须位于会话工作区内」 | 文稿、字幕、BGM、录音、参考音频或输出路径指到了工作区外面 | 先把文件复制进当前会话工作区文件夹再让助手处理 |
|
|
63
65
|
| 合成或转写报网络连接失败 | 受限网络无法直连在线语音 / ASR 服务 | 在 DSH「设置 → 插件 → 语音」填入本地代理 `proxyUrl`(支持 `http://` / `https://` / `socks5://` / `socks5h://`,保存即生效) |
|
|
64
66
|
| 音频超过 25MB 无法转写 | 在线 ASR 接口单文件体积上限为 25MB | 让助手开启 `autoCompress: true`(需本机有 `ffmpeg`)自动压缩/切片转写,或先用 `dsh-ffmpeg` 处理 |
|
|
65
67
|
|
package/docs/USAGE.en.md
CHANGED
|
@@ -4,12 +4,13 @@
|
|
|
4
4
|
|
|
5
5
|
## Get Started in 3 Steps
|
|
6
6
|
|
|
7
|
-
### Step 1: Know What Works Out of the Box
|
|
7
|
+
### Step 1: Know What Works Out of the Box, 5-Theme Workbench & Optional Setup
|
|
8
8
|
|
|
9
|
-
- **
|
|
9
|
+
- **Native 5-theme Voice Studio sidebar workbench**: Open Voice Studio in the DSH sidebar with 5 synced themes (**Native Airy / Warm Cream / Claude Warmth / Bright 8-Bit / Airy Mint**) and an interactive Coachmark onboarding tour. Audition 300+ online voices and 10 personas (`0.8x–1.5x` speed + duration badge), dictate with your browser microphone (`MediaRecorder`), and manage `customVoices` and `pronunciations` rules without editing YAML.
|
|
10
|
+
- **Text-to-speech (`voice_tts`, `voice_list`, `voice_preview`) works with zero config**: Uses Edge neural voices by default with zero API keys — discover 300+ live online Edge voices or 45+ all-age/dialect built-ins, apply 10 ready-made `persona` presets, reverse-dub `.srt`/`.vtt` timelines (`subtitleInput`), apply `pronunciations`, inspect exact `durationMs`/`durationSec`, generate an interactive browser audition page, narrate long workspace files, compose multi-speaker podcasts, and export `.srt`/`.vtt` subtitles. Switch `ttsEngine` to `siliconflow`, `openai`, or `custom` in Settings when you need emotion `instructions`, Kokoro voice blending, or zero-shot `referenceAudio` cloning.
|
|
10
11
|
- **Speech-to-text (`voice_stt`) needs an ASR key for cloud endpoints (loopback Whisper is keyless)**:
|
|
11
12
|
- **Recommended**: Open **DSH → Settings → Plugins → Voice (`dsh-voice`)**, pick `asrEngine` (`groq` by default, `openai`, `siliconflow`, or `custom`), and paste `asrApiKey` (masked and **applied live on save without restarting**; when `custom` points to `127.0.0.1` or `localhost`, no key is required);
|
|
12
|
-
- **Custom voice aliases (`customVoices`)**: Save reusable named characters
|
|
13
|
+
- **Custom voice aliases (`customVoices`) & pronunciation lexicon (`pronunciations`)**: Save reusable named characters and pronunciation overrides (e.g. `Nginx` → `Engine X`) in DSH Settings or Voice Studio (live-reloaded without restarting);
|
|
13
14
|
- **CLI / Environment variable**: Set `DSH_VOICE_ASR_KEY` (and optionally `DSH_VOICE_TTS_KEY`) or configure `cordis.patch.yml`:
|
|
14
15
|
|
|
15
16
|
```yaml
|
|
@@ -23,6 +24,11 @@
|
|
|
23
24
|
# ttsBaseUrl: https://api.siliconflow.cn/v1 # optional for non-edge engines
|
|
24
25
|
# ttsModel: FunAudioLLM/CosyVoice2-0.5B
|
|
25
26
|
# ttsApiKey: sk-... # prefer DSH Settings or DSH_VOICE_TTS_KEY
|
|
27
|
+
# pronunciations:
|
|
28
|
+
# - word: Nginx
|
|
29
|
+
# alias: Engine X
|
|
30
|
+
# - word: SQL
|
|
31
|
+
# alias: S Q L
|
|
26
32
|
# customVoices:
|
|
27
33
|
# - name: narrator-wang
|
|
28
34
|
# persona: documentary-narrator
|
|
@@ -38,9 +44,9 @@
|
|
|
38
44
|
# proxyUrl: socks5://127.0.0.1:1080 # supports http://, https://, socks5://, socks5h://
|
|
39
45
|
```
|
|
40
46
|
|
|
41
|
-
### Step 2: Keep Scripts, Reference Clips, Audio & Outputs Inside Your Session Workspace
|
|
47
|
+
### Step 2: Keep Scripts, Subtitles, BGM, Reference Clips, Audio & Outputs Inside Your Session Workspace
|
|
42
48
|
|
|
43
|
-
All `voice_tts` input files (`inputFile`), zero-shot cloning clips (`referenceAudio`, ≤10 MB), input audio (`audio`), and generated MP3 / `.txt` / `.srt` / `.vtt` / preview files must resolve under the active DSH session workspace (`session.header.cwd`) to protect host files. Copy outside files into the workspace first.
|
|
49
|
+
All `voice_tts` input files (`inputFile`), reverse-dubbing subtitle files (`subtitleInput`, ≤200 cues), podcast background music (`bgmFile`), zero-shot cloning clips (`referenceAudio`, ≤10 MB), input audio (`audio`), and generated MP3 / `.txt` / `.srt` / `.vtt` / preview files must resolve under the active DSH session workspace (`session.header.cwd`) to protect host files. Copy outside files into the workspace first.
|
|
44
50
|
|
|
45
51
|
### Step 3: Discover Voices/Personas, Audition, Then Synthesize or Transcribe
|
|
46
52
|
|
|
@@ -54,12 +60,14 @@ Unsure which voice fits best? Call `voice_list({ online: true, locale, gender, t
|
|
|
54
60
|
> Run `voice_list` with `online: true` for tag `podcast`, then preview a few alongside `late-night-radio` and `anime-boy` in `voice_previews/index.html` so I can pick my favorite.
|
|
55
61
|
- **Narrate a Markdown file directly with a persona and synchronized subtitles**
|
|
56
62
|
> Synthesize `guide.md` in this workspace using `documentary-narrator` into `guide.mp3` and export `guide.srt`.
|
|
57
|
-
- **Create a loudness-balanced two-host podcast MP3**
|
|
58
|
-
> Turn this release note into a two-host podcast using `podcast-host-female` and `podcast-host-male` with `normalizeLoudness: true`, saving `episode.mp3` and `episode.srt`.
|
|
63
|
+
- **Create a loudness-balanced two-host podcast MP3 with BGM auto-ducking**
|
|
64
|
+
> Turn this release note into a two-host podcast using `podcast-host-female` and `podcast-host-male`, mix `assets/bgm.mp3` with `bgmDucking: true` and `normalizeLoudness: true`, saving `episode.mp3` and `episode.srt`.
|
|
65
|
+
- **Reverse-dub an existing `.srt`/`.vtt` timeline with custom pronunciation rules**
|
|
66
|
+
> Dub `video.srt` in this workspace using `bilingual-tech-male`, pronouncing `Nginx` as `Engine X` (keep `Nginx` in subtitles), save `dubbed.mp3` and `dubbed.srt`, and report exact `durationSec`.
|
|
59
67
|
- **Emotion-directed acting or 5–10s zero-shot voice cloning (SiliconFlow / OpenAI / custom engine)**
|
|
60
68
|
> Use `ref.wav` (transcript: "Hello, what a great day to start!") in this workspace for zero-shot cloning with cheerful acting `instructions` and inline `[laughter]`, saving `greeting.mp3`.
|
|
61
|
-
- **Transcribe a large recording with auto-compression and subtitles**
|
|
62
|
-
> Transcribe `interview.m4a` in this workspace (enable `autoCompress` if over 25 MB; prompt hint: DeepSeek Harness, Cordis), and save `interview.txt` and `interview.srt`.
|
|
69
|
+
- **Transcribe a large recording with speaker diarization, auto-compression, and subtitles**
|
|
70
|
+
> Transcribe `interview.m4a` in this workspace with `diarize: true` (enable `autoCompress` if over 25 MB; prompt hint: DeepSeek Harness, Cordis), and save `interview.txt` and `interview.srt`.
|
|
63
71
|
- **Scoped offline configuration self-check**
|
|
64
72
|
> Run a voice health check to verify `ttsReady`, `sttReady`, voice/persona/alias status, prosody settings, proxy, and ASR key status.
|
|
65
73
|
|
|
@@ -73,28 +81,45 @@ Run `voice_health` first (offline check with zero network traffic; supports `sco
|
|
|
73
81
|
| :--- | :--- | :--- |
|
|
74
82
|
| Health check says `asrApiKey` is missing | `asrApiKey` and `DSH_VOICE_ASR_KEY` are both empty | Safe to ignore if you only need TTS (`ttsReady` is `true`), or pass `scope: "tts"`; for local Whisper, set `asrEngine: custom` with `127.0.0.1`/`localhost` (no key needed) |
|
|
75
83
|
| `referenceAudio` fails saying cloning engine is required | `ttsEngine` is still `edge` (Edge does not support zero-shot reference cloning) | Switch `ttsEngine` to `siliconflow` (`CosyVoice2-0.5B`) or `custom` (e.g. local GPT-SoVITS / IndexTTS) in **DSH Settings → Plugins → Voice** |
|
|
76
|
-
| `normalizeLoudness` fails saying `ffmpeg` is missing | `-16 LUFS`
|
|
84
|
+
| `normalizeLoudness` or `bgmFile` fails saying `ffmpeg` is missing | `-16 LUFS` loudness mastering and `sidechaincompress` BGM ducking require local `ffmpeg` | Install `ffmpeg` locally (or pre-process with `dsh-ffmpeg`), or omit `normalizeLoudness`/`bgmFile` |
|
|
77
85
|
| Invalid voice ID, rate, or pitch error | `ttsVoice` is not a valid Edge ID, built-in `persona`, or `customVoices` alias, or `ttsRate`/`ttsPitch` format is invalid | Run `voice_list` to pick a valid ID or persona, and fix `ttsVoice` / `ttsRate` / `ttsPitch` in Settings |
|
|
78
86
|
| Network error on TTS or STT | Direct connection blocked by corporate network | Enter `proxyUrl` (`http://`, `https://`, `socks5://`, or `socks5h://`) in **DSH Settings → Plugins → Voice** (applies live) |
|
|
79
|
-
| Path outside session workspace error | Script, reference clip, audio, or output resolves outside workspace | Copy the file into the current session workspace folder first |
|
|
87
|
+
| Path outside session workspace error | Script, subtitle, BGM, reference clip, audio, or output resolves outside workspace | Copy the file into the current session workspace folder first |
|
|
80
88
|
| Audio exceeds 25 MB limit | Standard Whisper / ASR multipart cap is 25 MB | Pass `autoCompress: true` to `voice_stt` (uses local `ffmpeg` to compress/segment automatically) or pre-process with `dsh-ffmpeg` |
|
|
81
89
|
| ASR or TTS refuses a 3xx redirect | Redirects are blocked so audio/keys never leak to another host | Set `asrBaseUrl` / `ttsBaseUrl` directly to the final non-redirecting `/v1` endpoint |
|
|
82
90
|
| Existing output file was not overwritten | `overwrite: false` (default) suffixes `_1`, `_2` safely | Use the exact path returned by the tool, or enable `overwrite` in Settings |
|
|
83
91
|
|
|
84
92
|
---
|
|
85
93
|
|
|
86
|
-
## Tool & Voice
|
|
94
|
+
## Tool & Voice Studio Reference
|
|
87
95
|
|
|
88
96
|
### Registered Tools (All 5)
|
|
89
97
|
|
|
90
98
|
| Tool | Purpose | Key Parameters & Notes |
|
|
91
99
|
| :--- | :--- | :--- |
|
|
92
|
-
| `voice_tts` | Synthesize MP3 & subtitles from text, files, or dialogues | Supports `text`, `inputFile` (≤100k chars auto-chunked), `cleanMarkdown` (default `true` for `.md`), `segments` (≤50 turns, per-turn `voice`/`persona`/`rate`/`pitch`/`pauseMs`/`instructions`/`referenceAudio`/`referenceText`), `
|
|
93
|
-
| `voice_stt` | Transcribe audio to text & subtitles (zero-copy upload) | `audio` required; supports `autoCompress` (uses local `ffmpeg` for >25 MB), `timestamps`, `subtitleOutput` (`.srt`/`.vtt`), `model`, `language`, `prompt`, `output`, `engine` |
|
|
100
|
+
| `voice_tts` | Synthesize MP3 & subtitles from text, files, timelines, or dialogues | Supports `text`, `inputFile` (≤100k chars auto-chunked), `cleanMarkdown` (default `true` for `.md`), `subtitleInput` (`.srt`/`.vtt` reverse timeline dubbing, ≤200 cues), `segments` (≤50 turns, per-turn `voice`/`persona`/`rate`/`pitch`/`pauseMs`/`instructions`/`referenceAudio`/`referenceText`), `pronunciations` (`[{ word, alias }]` or `{ word: alias }`), `bgmFile` / `bgmVolume` (`0.02–1.0`, default `0.18`) / `bgmDucking` (default `true`), `normalizeLoudness` (`-16 LUFS`), `persona`, `instructions`, `referenceAudio` (≤10 MB), `referenceText`, `subtitleOutput` (`.srt`/`.vtt`), `voice`, `rate`, `pitch`, `output`; returns exact `durationMs` and `durationSec` |
|
|
101
|
+
| `voice_stt` | Transcribe audio to text & subtitles (zero-copy upload) | `audio` required; supports `diarize` (`true` enables FFmpeg stereo channel split or `[Speaker]` turn diarization with `speakers` output), `autoCompress` (uses local `ffmpeg` for >25 MB), `timestamps`, `subtitleOutput` (`.srt`/`.vtt`), `model`, `language`, `prompt`, `output`, `engine` |
|
|
94
102
|
| `voice_list` | Discover online/built-in voices, personas & custom aliases | Optional `online` (`true` fetches & caches 300+ live Microsoft Edge voices for 24h, falling back to 45+ curated built-ins), `locale`, `gender` (`Female`/`Male`), `tag` (e.g. `podcast`, `multilingual`, `dialect`, `child`, `teen`, `mature`, `elder`, `anime`, `documentary`), `includePersonas` (default `true`) |
|
|
95
103
|
| `voice_preview` | Bounded-concurrency (≤3) batch previews + HTML page | Optional `voices` (≤8; accepts raw voice IDs, built-in `persona` IDs/labels, or `customVoices` aliases), `text` (≤200 chars), `outputDir`, `htmlIndex` (default `true`, writes `voice_previews/index.html`) |
|
|
96
104
|
| `voice_health` | Offline configuration self-check | Optional `scope` (`all`/`tts`/`stt`), reports `ttsReady` / `sttReady` across engines, voice/persona/`customVoices` aliases, prosody, proxy, and keys |
|
|
97
105
|
|
|
106
|
+
### `0.9.0` Voice Studio & Pro Audio Capabilities
|
|
107
|
+
|
|
108
|
+
1. **Native 5-Theme Voice Studio Sidebar Workbench**:
|
|
109
|
+
- Features 5 live-synced themes (**Native Airy / Warm Cream / Claude Warmth / Bright 8-Bit / Airy Mint**) and an interactive spotlight Coachmark onboarding tour;
|
|
110
|
+
- Includes 4 workbench tabs: **Voices & Personas** (live audition 300+ voices & 10 personas, `0.8x–1.5x` playback, set default), **Studio & Prompts** (live duration badge + 1-click prompt templates), **Mic & Custom Voices** (browser `MediaRecorder` dictation -> `transcribeMic`, plus `customVoices` editor), and **Lexicon & Health** (`pronunciations` rules + `ttsReady`/`sttReady` badges).
|
|
111
|
+
2. **Exact MP3 Duration Feedback (`durationMs` / `durationSec`) & Matched-Sample-Rate Silence Frames**:
|
|
112
|
+
- Inspects CBR/VBR and multi-engine MP3 frame headers to compute exact output duration and generates matched-sample-rate silence frames for pauses and timeline padding so agents can reliably target exact durations (such as 60s short-video voiceovers).
|
|
113
|
+
3. **Pronunciation Lexicon (`pronunciations`)**:
|
|
114
|
+
- Compiles to SSML `<sub alias>` on Edge (exported `.srt`/`.vtt` subtitles preserve `Nginx` while speech says `Engine X`), and applies longest-first non-recursive replacement on OpenAI/SiliconFlow/custom engines while keeping original spellings in fallback subtitle cues.
|
|
115
|
+
4. **Reverse `.srt` / `.vtt` Video Dubbing (`subtitleInput`)**:
|
|
116
|
+
- Parses `.srt`/`.vtt` cues, measures synthesized cue duration, re-synthesizes cues exceeding their time window by >8% with scaled speech rate (clamped `-30%` to `+85%`), and inserts millisecond silence gaps to stay aligned with the video timeline.
|
|
117
|
+
5. **Podcast BGM Auto-Ducking (`bgmFile`, `bgmVolume`, `bgmDucking`)**:
|
|
118
|
+
- Uses local `ffmpeg` to loop workspace BGM, duck music under speech via `sidechaincompress`, add a smooth fade-out at the end, and optionally apply `-16 LUFS` loudness normalization (`normalizeLoudness: true`).
|
|
119
|
+
6. **Speaker Diarization (`diarize: true`) & SiliconFlow SHA-256 Cached Cloning URI**:
|
|
120
|
+
- `voice_stt({ diarize: true })` splits stereo channels via `ffmpeg` when available or parses/infers speaker turns from `verbose_json` to output `[说话人 A]` paragraphs, `speakers` list, and speaker-prefixed `.srt`/`.vtt` cues;
|
|
121
|
+
- SiliconFlow zero-shot cloning uploads `referenceAudio` once per SHA-256 hash to `/uploads/audio/voice` and caches the returned `speech:...` URI across podcast segments.
|
|
122
|
+
|
|
98
123
|
### `0.8.0` 4-Layer Voice Diversity Overview
|
|
99
124
|
|
|
100
125
|
1. **300+ Live Edge Voice Discovery & 45+ All-Age/Dialect Built-ins**:
|
|
@@ -113,8 +138,6 @@ Run `voice_health` first (offline check with zero network traffic; supports `sco
|
|
|
113
138
|
4. **Zero-Shot Voice Cloning (`referenceAudio` + `referenceText`) & `customVoices` Aliases**:
|
|
114
139
|
- Clone any voice from a 5–10s clean workspace clip (≤10 MB) on `ttsEngine: 'siliconflow'` or `'custom'`;
|
|
115
140
|
- Save reusable named presets in DSH Settings `customVoices` (live-reloaded without restart) for `voice_tts`, `segments[]`, and `voice_preview`.
|
|
116
|
-
5. **Podcast Loudness Mastering (`normalizeLoudness: true`)**:
|
|
117
|
-
- Runs local `ffmpeg` (`loudnorm=I=-16:TP=-1.5:LRA=11`) after synthesis/concatenation for even `-16 LUFS` broadcast loudness across voices.
|
|
118
141
|
|
|
119
142
|
### Engine Presets & Workspace Guardrails
|
|
120
143
|
|
package/docs/USAGE.md
CHANGED
|
@@ -4,12 +4,13 @@
|
|
|
4
4
|
|
|
5
5
|
## 最快 3 步用起来
|
|
6
6
|
|
|
7
|
-
### 第 1
|
|
7
|
+
### 第 1 步:了解零配置范围、5 主题工作台与可选配置
|
|
8
8
|
|
|
9
|
-
-
|
|
9
|
+
- **原生 5 主题 Voice Studio 侧边栏工作台**:在 DSH 侧边栏打开语音工作台,支持 **原生清透 / 米白纸感 / Claude 暖书卷 / 8-Bit 像素 / 晴空薄荷** 5 套同步主题与交互式 Coachmark 新手引导;可在面板内直接搜索并试听 300+ 在线音色与 10 套人设(支持 `0.8x–1.5x` 倍速与实时时长显示)、使用浏览器麦克风录制并一键转写速记,以及可视化管理 `customVoices` 与 `pronunciations` 发音词典。
|
|
10
|
+
- **文字转语音(`voice_tts` / `voice_list` / `voice_preview`)零配置即用**:默认使用 Edge 神经语音,无需任何 API 密钥即可联网发现 300+ 实时音色或使用 45+ 全年龄/方言内置音色、调用 10 大内置 `persona` 人设、按 `subtitleInput`(`.srt`/`.vtt`)反向卡点配音、使用 `pronunciations` 替读、获取精确 `durationMs` / `durationSec` 时长、生成网页试听页、直读长文并合成 MP3 与 `.srt`/`.vtt` 字幕;如需情绪指令(`instructions`)、Kokoro 混音或零样本参考克隆(`referenceAudio`),可在设置中切换 `ttsEngine` 为 `siliconflow`、`openai` 或 `custom`。
|
|
10
11
|
- **录音转文字(`voice_stt`)按需填密钥(本地回环免密)**:
|
|
11
12
|
- **推荐方式**:打开 **DSH → 设置 → 插件 → 语音(`dsh-voice`)**,选择 `asrEngine`(默认 `groq`,可选 `openai`、`siliconflow` 或 `custom`),填入 `asrApiKey`(自动脱敏,**保存立即生效、无需重启**;当 `custom` 指向 `127.0.0.1` / `localhost` 本地 Whisper 服务时无需密钥);
|
|
12
|
-
- **自定义音色别名(`customVoices`)**:可在 DSH
|
|
13
|
+
- **自定义音色别名(`customVoices`)与发音词典(`pronunciations`)**:可在 DSH 设置或工作台保存命名角色与发音替读规则(如 `Nginx` → `Engine X`),保存即热更新生效;
|
|
13
14
|
- **命令行 / 环境变量方式**:设置环境变量 `DSH_VOICE_ASR_KEY`(及可选 `DSH_VOICE_TTS_KEY`),或在 `cordis.patch.yml` 中配置:
|
|
14
15
|
|
|
15
16
|
```yaml
|
|
@@ -23,6 +24,11 @@
|
|
|
23
24
|
# ttsBaseUrl: https://api.siliconflow.cn/v1 # 非 edge 引擎时可自定义
|
|
24
25
|
# ttsModel: FunAudioLLM/CosyVoice2-0.5B
|
|
25
26
|
# ttsApiKey: sk-... # 推荐在 DSH 设置页填写或用 DSH_VOICE_TTS_KEY
|
|
27
|
+
# pronunciations:
|
|
28
|
+
# - word: Nginx
|
|
29
|
+
# alias: Engine X
|
|
30
|
+
# - word: SQL
|
|
31
|
+
# alias: S Q L
|
|
26
32
|
# customVoices:
|
|
27
33
|
# - name: 老王
|
|
28
34
|
# persona: documentary-narrator
|
|
@@ -38,9 +44,9 @@
|
|
|
38
44
|
# proxyUrl: socks5://127.0.0.1:1080 # 支持 http://、https://、socks5://、socks5h://
|
|
39
45
|
```
|
|
40
46
|
|
|
41
|
-
### 第 2
|
|
47
|
+
### 第 2 步:在工作区内准备文稿、字幕、BGM、参考音频或录音
|
|
42
48
|
|
|
43
|
-
为了防止路径越界读取隐私文件,所有 `voice_tts` 直读文稿(`inputFile`)、零样本克隆参考音频(`referenceAudio`,≤10MB)、待转写音频(`audio`)以及生成的 MP3 / `.txt` / `.srt` / `.vtt` / 试听页都严格限制在**当前会话工作区**内;外部文件请先复制进工作区目录。
|
|
49
|
+
为了防止路径越界读取隐私文件,所有 `voice_tts` 直读文稿(`inputFile`)、反向卡点字幕(`subtitleInput`,≤200 条)、播客背景音乐(`bgmFile`)、零样本克隆参考音频(`referenceAudio`,≤10MB)、待转写音频(`audio`)以及生成的 MP3 / `.txt` / `.srt` / `.vtt` / 试听页都严格限制在**当前会话工作区**内;外部文件请先复制进工作区目录。
|
|
44
50
|
|
|
45
51
|
### 第 3 步:发现音色/人设 → 试听对比 → 合成或转写
|
|
46
52
|
|
|
@@ -54,12 +60,14 @@
|
|
|
54
60
|
> 调用 `voice_list` 联网按 `podcast` 标签筛选中英文音色,挑几个连同 `late-night-radio` 和 `anime-boy` 人设一起生成短试听,打开 `voice_previews/index.html` 让我选。
|
|
55
61
|
- **直接朗读 Markdown 长文并导出同步字幕**
|
|
56
62
|
> 用 `documentary-narrator` 人设直接读取工作区 `guide.md` 合成 `guide.mp3`,同时导出 `guide.srt`。
|
|
57
|
-
-
|
|
58
|
-
> 把这篇更新整理成双人播客,女主持用 `podcast-host-female`,男主持用 `podcast-host-male
|
|
63
|
+
- **制作带 BGM 自动闪避与响度均衡的双人对话播客 MP3**
|
|
64
|
+
> 把这篇更新整理成双人播客,女主持用 `podcast-host-female`,男主持用 `podcast-host-male`,搭配工作区 `assets/bgm.mp3` 开启 `bgmDucking: true` 和 `normalizeLoudness: true`,导出 `episode.mp3` 和 `episode.srt`。
|
|
65
|
+
- **按 `.srt` / `.vtt` 字幕时间轴反向卡点配音 + 发音词典替读**
|
|
66
|
+
> 用 `bilingual-tech-male` 按工作区 `video.srt` 时间轴反向配音,把 `Nginx` 替读为 `Engine X`(字幕保持原拼写 `Nginx`),保存 `dubbed.mp3` 和 `dubbed.srt` 并告诉我精确的 `durationSec` 时长。
|
|
59
67
|
- **情绪演技配音或 5–10 秒零样本声音克隆(SiliconFlow / OpenAI / 自定义引擎)**
|
|
60
68
|
> 用工作区的 `ref.wav`(对应台词「今天天气真好,我们出发吧」)零样本克隆声音,加上 `[laughter]` 和轻快热情的 `instructions` 语气提示,合成 `greeting.mp3`。
|
|
61
|
-
- **大体积录音转写 + 专有名词纠偏 + 字幕导出**
|
|
62
|
-
> 把工作区里的 `interview.m4a` 转写成中文(如果超过 25MB
|
|
69
|
+
- **大体积录音转写 + 说话人区分 + 专有名词纠偏 + 字幕导出**
|
|
70
|
+
> 把工作区里的 `interview.m4a` 转写成中文(如果超过 25MB 自动压缩分段,开启 `diarize: true` 区分说话人,提示词加上:DeepSeek Harness、Cordis),存成 `interview.txt` 和 `interview.srt`。
|
|
63
71
|
- **按范围执行离线配置自检**
|
|
64
72
|
> 跑一下语音插件自检(分别看看 `ttsReady` 和 `sttReady`),检查默认音色/人设/自定义别名、语速音调格式、代理和 ASR 配置状态。
|
|
65
73
|
|
|
@@ -72,30 +80,47 @@
|
|
|
72
80
|
| 现象 | 常见原因 | 一步修复 |
|
|
73
81
|
| :--- | :--- | :--- |
|
|
74
82
|
| 自检提示「缺 ASR 密钥」 | 尚未填写 `asrApiKey` 且未设 `DSH_VOICE_ASR_KEY` | 若只需朗读配音可忽略(`ttsReady` 为 `true` 时 `voice_tts` 正常可用),或传 `scope: "tts"`;本地 Whisper 可设 `asrEngine: custom` 指向 `127.0.0.1` / `localhost` 免密运行 |
|
|
75
|
-
| 传入 `referenceAudio`
|
|
76
|
-
| 开启 `normalizeLoudness` 报错缺 `ffmpeg` | `-16 LUFS`
|
|
83
|
+
| 传入 `referenceAudio` 提示不支持克隆 | 当前 `ttsEngine` 仍为 `edge`(Edge 不支持参考音频克隆) | 在 **DSH「设置 → 插件 → 语音」**将 `ttsEngine` 切为 `siliconflow`(`CosyVoice2-0.5B`)或 `custom`(如本地 GPT-SoVITS / IndexTTS) |
|
|
84
|
+
| 开启 `normalizeLoudness` 或 `bgmFile` 报错缺 `ffmpeg` | `-16 LUFS` 广电响度均衡与 `sidechaincompress` BGM 闪避混音依赖本机 `ffmpeg` | 本机安装 `ffmpeg`(或先用 `dsh-ffmpeg` 处理),或暂时关闭该开关 |
|
|
77
85
|
| 提示音色 ID、语速或音调不合法 | `ttsVoice` 既非合法 Edge ID 也不在内置 `persona`/`customVoices` 中,或 `ttsRate`/`ttsPitch` 格式不对 | 用 `voice_list` 查看合法 ID 与人设,在设置页修正 `ttsVoice`、`ttsRate`、`ttsPitch` |
|
|
78
86
|
| `voice_tts` 或 `voice_stt` 报网络失败 | 当前网络无法直连 Edge 朗读服务或外网 ASR 接口 | 在 **DSH「设置 → 插件 → 语音」**填写 `proxyUrl`(支持 `http://`、`https://`、`socks5://`、`socks5h://`,保存即生效) |
|
|
79
|
-
| 提示文件必须位于会话工作区内 | 传入了工作区外部路径或越界符号链接 |
|
|
87
|
+
| 提示文件必须位于会话工作区内 | 传入了工作区外部路径或越界符号链接 | 将文稿、字幕、BGM、参考音频或录音复制到当前会话工作区文件夹内再执行 |
|
|
80
88
|
| 音频超过 25MB 报超限 | OpenAI / Groq 兼容 ASR 单文件上限 25MB | 调用 `voice_stt` 时传入 `autoCompress: true`(调用本机 `ffmpeg` 自动转码压缩/分段),或先用 `dsh-ffmpeg` 处理 |
|
|
81
89
|
| ASR 返回 3xx 重定向被拦截 | 为防音频与密钥被转发到第三方主机,插件拒绝自动跟随重定向 | 将 `asrBaseUrl` 或 `ttsBaseUrl` 直接改为服务商最终无跳转的 `/v1` 地址 |
|
|
82
90
|
| 同名文件没有覆盖旧文件 | 默认 `overwrite: false` 防误删,自动追加 `_1`、`_2` 序号 | 查看工具返回的实际落盘路径,或在设置中显式开启 `overwrite` |
|
|
83
91
|
|
|
84
92
|
---
|
|
85
93
|
|
|
86
|
-
##
|
|
94
|
+
## 工具与语音工作流参考
|
|
87
95
|
|
|
88
96
|
### 注册工具清单(共 5 个)
|
|
89
97
|
|
|
90
98
|
| 工具 | 作用 | 关键参数与说明 |
|
|
91
99
|
| :--- | :--- | :--- |
|
|
92
|
-
| `voice_tts` |
|
|
93
|
-
| `voice_stt` | 音频转文字与字幕(零拷贝流式上传) | `audio` 必填;支持 `autoCompress`(>25MB 用本机 `ffmpeg` 自动压缩分段)、`timestamps`、`subtitleOutput`(`.srt`/`.vtt`)、`model`、`language`、`prompt`、`output`、`engine` |
|
|
100
|
+
| `voice_tts` | 文字/文稿/字幕/播客合成 MP3 与字幕 | 支持 `text`、`inputFile`(≤10 万字自动分片拼接)、`cleanMarkdown`(`.md` 默认开启)、`subtitleInput`(`.srt`/`.vtt` 反向卡点配音,≤200 条)、`segments`(≤50 段,每段支持 `voice`/`persona`/`rate`/`pitch`/`pauseMs`/`instructions`/`referenceAudio`/`referenceText`)、`pronunciations`(`[{ word, alias }]` 或 `{ word: alias }`)、`bgmFile` / `bgmVolume`(`0.02–1.0`,默认 `0.18`)/ `bgmDucking`(默认 `true`)、`normalizeLoudness`(`-16 LUFS`)、`persona`、`instructions`、`referenceAudio`(≤10MB)、`referenceText`、`subtitleOutput`(`.srt`/`.vtt`)、`voice`、`rate`、`pitch`、`output`;返回精确 `durationMs` 与 `durationSec` |
|
|
101
|
+
| `voice_stt` | 音频转文字与字幕(零拷贝流式上传) | `audio` 必填;支持 `diarize`(`true` 时按双声道分离或解析/推断说话人切换并输出 `[说话人 A]` 标记与 `speakers`)、`autoCompress`(>25MB 用本机 `ffmpeg` 自动压缩分段)、`timestamps`、`subtitleOutput`(`.srt`/`.vtt`)、`model`、`language`、`prompt`、`output`、`engine` |
|
|
94
102
|
| `voice_list` | 在线/离线发现音色、内置人设与自定义别名 | 可选 `online`(`true` 拉取并 24h 缓存微软 300+ 实时音色,断网自动回退 45+ 精选内置)、`locale`、`gender`(`Female`/`Male`)、`tag`(如 `podcast`/`multilingual`/`dialect`/`child`/`teen`/`mature`/`elder`/`anime`/`documentary`)、`includePersonas`(默认 `true`) |
|
|
95
103
|
| `voice_preview` | 有界并发(≤3)批量试听并生成网页试听页 | 可选 `voices`(≤8,支持原始音色 ID、10 大内置 `persona` ID/名称及 `customVoices` 别名)、`text`(≤200 字)、`outputDir`、`htmlIndex`(默认 `true`,生成 `voice_previews/index.html`) |
|
|
96
104
|
| `voice_health` | 本地配置自检(不触网) | 可选 `scope`(`all`/`tts`/`stt`),返回 `ttsReady` / `sttReady` 并校验引擎、音色/人设/`customVoices` 别名、语速音调、代理与密钥 |
|
|
97
105
|
|
|
98
|
-
### `0.
|
|
106
|
+
### `0.9.0` 语音工作台与专业音频能力一览
|
|
107
|
+
|
|
108
|
+
1. **原生 5 主题 Voice Studio 侧边栏工作台**:
|
|
109
|
+
- 支持 **原生清透 / 米白纸感 / Claude 暖书卷 / 8-Bit 像素 / 晴空薄荷** 5 套实时同步主题与 Coachmark 聚光灯新手引导;
|
|
110
|
+
- 提供「音色与人设(在线 300+ 试听与一键设为默认)」、「快速试配与播客指令(实时时长反馈与一键复制模板)」、「麦克风速记与角色管理(浏览器 `MediaRecorder` 录音转写 + `customVoices` 管理)」、「发音词典与自检(`pronunciations` 增删与 `ttsReady`/`sttReady` 状态)」4 个标签页。
|
|
111
|
+
2. **精确 MP3 时长反馈(`durationMs` / `durationSec`)与匹配采样率静音帧**:
|
|
112
|
+
- 跨 CBR/VBR 与多引擎流精确解析帧头时长,并按前序分片采样率生成匹配静音帧(确保段间停顿与字幕卡点零杂音、无缝拼接),方便助手根据 `durationSec` 卡准 60 秒短视频配音目标。
|
|
113
|
+
3. **发音词典(`pronunciations`)**:
|
|
114
|
+
- Edge 引擎编译为 SSML `<sub alias>`(导出 `.srt`/`.vtt` 字幕保留 `Nginx` 原词,人声念作 `Engine X`);OpenAI / SiliconFlow / 自定义引擎按词长降序单遍非递归替换朗读文本(同时在兜底字幕中保留原词)。
|
|
115
|
+
4. **`.srt` / `.vtt` 字幕反向卡点配音(`subtitleInput`)**:
|
|
116
|
+
- 逐条解析字幕时间窗,若单条语音超出时间窗 8% 以上则自动按比例计算加速语速(限幅 `-30%` 至 `+85%`)重配一次,并按毫秒精确插入静音帧对齐原视频画面时间轴。
|
|
117
|
+
5. **播客 BGM 自动闪避(`bgmFile` / `bgmVolume` / `bgmDucking`)**:
|
|
118
|
+
- 使用本机 `ffmpeg` 循环铺底工作区背景音乐,通过 `sidechaincompress` 在人声出现时自动压低背景音乐音量,并在结尾自动平滑淡出(超过 3 秒时添加 1.2s `afade`),可选链式叠加 `-16 LUFS` 广电响度均衡。
|
|
119
|
+
6. **说话人区分(`diarize: true`)与 SiliconFlow SHA-256 缓存声音克隆**:
|
|
120
|
+
- `voice_stt` 开启 `diarize: true` 后,双声道录音在有 `ffmpeg` 时自动拆分为左右声道(`说话人 A` / `说话人 B`)分别转写再按时间归并,单声道录音则读取服务端 `speaker` 字段或按 1.2 秒停顿推断说话人轮次;
|
|
121
|
+
- SiliconFlow 零样本克隆自动按音频内容 SHA-256 哈希向 `/uploads/audio/voice` 上传一次并缓存 `speech:...` URI,多段播客无需重复传输 MB 级音频。
|
|
122
|
+
|
|
123
|
+
### `0.8.0` 语音多样性能力回顾
|
|
99
124
|
|
|
100
125
|
1. **全量发现与全年龄/方言内置(45+ 内置 / 300+ 在线)**:
|
|
101
126
|
- `voice_list({ online: true })` 联网拉取微软 Edge 300+ 全量神经音色并内存缓存 24 小时,离线自动回退至 45+ 精选内置音色:
|
|
@@ -113,8 +138,6 @@
|
|
|
113
138
|
4. **零样本参考克隆(`referenceAudio` + `referenceText`)与 `customVoices` 别名**:
|
|
114
139
|
- 在 `siliconflow` 或 `custom` 引擎下传入工作区内 5–10 秒干净语音(≤10MB)及对应参考台词即可零样本克隆;
|
|
115
140
|
- 在 DSH 设置 `customVoices` 配置一次即可按中文或英文别名在 `voice_tts`、`segments[]` 与 `voice_preview` 中热更新复用。
|
|
116
|
-
5. **播客广电级响度均衡(`normalizeLoudness: true`)**:
|
|
117
|
-
- 合成或拼接完成后调用本机 `ffmpeg` 执行 `-16 LUFS`(`loudnorm=I=-16:TP=-1.5:LRA=11`)标准化,消除多角色或跨引擎音色之间的音量落差。
|
|
118
141
|
|
|
119
142
|
### 引擎预设与安全细节
|
|
120
143
|
|