@kmchai/cortico-world-voice 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +20 -23
- package/describe.json +5 -5
- package/package.json +4 -4
- package/src/ENV_PROMPT.md +3 -3
- package/src/config.ts +0 -24
- package/src/definition.ts +1 -1
- package/src/models.ts +1 -1
- package/src/speech/text.ts +1 -1
- package/src/tools.ts +2 -2
- package/src/triggers.ts +31 -105
- package/src/types.ts +0 -8
- package/src/world.ts +7 -38
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,19 @@
|
|
|
1
|
+
## 0.4.0
|
|
2
|
+
|
|
3
|
+
- 删除概率触发:不再有 `autoSpeak`、`autoProbability`、`autoFromText`、`autoTools`、`autoEvents`,也不再轮询事件或从工具参数自动找正文。
|
|
4
|
+
- 语音只在三种显式路径出声:`voice_speak` 工具、最近用户消息命中 `explicitKeywords`、正文里的 `<tts>…</tts>`。`voice_style` 仍影响后续语音。
|
|
5
|
+
- 环境提示词、工具说明、README 与配置页同步去掉自动朗读与概率文案;旧配置里的被删字段会被忽略。
|
|
6
|
+
- 这是破坏性行为变更:升级后普通回复不再随机朗读。需要出声的调用方应改用 `voice_speak`,或在消息里明确要求语音。
|
|
7
|
+
|
|
8
|
+
## 0.3.2
|
|
9
|
+
|
|
10
|
+
- IndexTTS2 的模型提示词明确:`emotion` / `instruct` 传自然语言,由服务映射到 `emotion_text`;不要在正文写 `emotion_text` 或方括号标签。服务 0.4.2 起这两个 options 不再由模型页覆盖。
|
|
11
|
+
|
|
12
|
+
## 0.3.1
|
|
13
|
+
|
|
14
|
+
- 自动朗读不再把 `end_turn` 当成实质工具调用。模型说完正文再调 `end_turn` 结束这一轮时,正文照常按概率朗读;仍会压掉与 `web_search` 等实际工具同轮出现的中间正文。
|
|
15
|
+
- 网关 0.1.1 已把 World 的 `outputTap` 扇进主 session;此前网关 bot 里概率朗读拿不到正文。
|
|
16
|
+
|
|
1
17
|
## 0.3.0
|
|
2
18
|
|
|
3
19
|
- 系统 Config tab 移除「音色与情绪」组;音色、情绪控制、默认情绪、默认表演指令、语速与副语言标签移到「模型」页。
|
package/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# @kmchai/cortico-world-voice
|
|
2
2
|
|
|
3
|
-
Cortico 的语音 World
|
|
3
|
+
Cortico 的语音 World:把回复念出来。触发、文本过滤、情绪语义、队列与播放留在这里;合成经 [Kmch AI · 服务](https://github.com/Kmch-AI/cortico-world-service) 的 HTTP 网关完成。
|
|
4
4
|
|
|
5
5
|
- **语音工具**:`voice_speak`、`voice_hush`、`voice_style`。
|
|
6
|
-
-
|
|
6
|
+
- **三条显式触发路径**:工具调用、用户语义请求(「念给我听」)、正文里的 `<tts>…</tts>`;不掷概率,也不会自动朗读普通回复。
|
|
7
7
|
- **情绪与标签**:自然语言指令、分类情绪、语速、副语言标签(`[Laughter]`、`[Sigh]` 等)。
|
|
8
8
|
- **文本过滤**:Markdown、代码、网址、动作标记、emoji、指定符号都不念。
|
|
9
9
|
- **出口可拆**:本机播放、发 `tts.voice` 事件交给播放插件、或只合成并交回音频附件。
|
|
@@ -34,9 +34,7 @@ corepack pnpm add "D:/path/to/cortico-world-voice" --ignore-workspace
|
|
|
34
34
|
"serviceUrl": "http://127.0.0.1:9883",
|
|
35
35
|
"serviceTokenSecret": "",
|
|
36
36
|
"model": "ming3-tts",
|
|
37
|
-
"playback": "local"
|
|
38
|
-
"autoSpeak": true,
|
|
39
|
-
"autoProbability": 0.3
|
|
37
|
+
"playback": "local"
|
|
40
38
|
}
|
|
41
39
|
}
|
|
42
40
|
}
|
|
@@ -52,8 +50,8 @@ World 页的「模型」tab 选模型与音色,并按服务 `/v1/models` 返回
|
|
|
52
50
|
|
|
53
51
|
| 工具 | 作用 |
|
|
54
52
|
|---|---|
|
|
55
|
-
| `voice_speak` | 念一句;可带 `emotion`、`instruct`、`voice`、`speed`、`silent
|
|
56
|
-
| `voice_style` | 读或改长期风格(情绪、指令、音色、语速),写回 `worlds.voice
|
|
53
|
+
| `voice_speak` | 念一句;可带 `emotion`、`instruct`、`voice`、`speed`、`silent`。 |
|
|
54
|
+
| `voice_style` | 读或改长期风格(情绪、指令、音色、语速),写回 `worlds.voice`,影响之后的语音。 |
|
|
57
55
|
| `voice_hush` | 立刻停声,清空排队。 |
|
|
58
56
|
|
|
59
57
|
`silent: true` 只合成、不在本机播放,音频随工具回执交给渠道,用于语音条这类出口。
|
|
@@ -62,22 +60,13 @@ World 页的「模型」tab 选模型与音色,并按服务 `/v1/models` 返回
|
|
|
62
60
|
|
|
63
61
|
| 路径 | 行为 |
|
|
64
62
|
|---|---|
|
|
65
|
-
| `voice_speak` 工具 |
|
|
66
|
-
| 用户语义请求 | 最近 `explicitWindowSec` 秒内的外部消息含 `explicitKeywords`(默认「语音,声音,发语音,念给我听,读给我听,voice,tts」)
|
|
67
|
-
|
|
|
68
|
-
| `<tts>…</tts>` | 只念标签内文,并强制朗读。标签会出现在可见文字里;渠道可见时优先用工具触发。 |
|
|
63
|
+
| `voice_speak` 工具 | 立即念;可带 `emotion`、`instruct`、`voice`、`speed`、`silent`。 |
|
|
64
|
+
| 用户语义请求 | 最近 `explicitWindowSec` 秒内的外部消息含 `explicitKeywords`(默认「语音,声音,发语音,念给我听,读给我听,voice,tts」)时朗读这一轮的纯文本正文。 |
|
|
65
|
+
| `<tts>…</tts>` | 只念标签内文。标签会出现在可见文字里;渠道可见时优先用工具触发。 |
|
|
69
66
|
|
|
70
|
-
|
|
67
|
+
同一段文字 5 秒内只念一次。没有概率自动朗读;普通回复不会自己出声。
|
|
71
68
|
|
|
72
|
-
|
|
73
|
-
- `autoTools`:这些工具的参数会被当作正文。默认 `pet_say, pet_ask, console_send`,取 `script` / `question` / `text` / `content` / `message` / `body` 里第一个字符串。
|
|
74
|
-
- `autoEvents`:轮询事件库,念这些事件类型的 `meta.body` 或正文。
|
|
75
|
-
|
|
76
|
-
同一段文字 5 秒内只念一次。概率按「一段话」掷,不按句掷。
|
|
77
|
-
|
|
78
|
-
只想响应工具与显式请求、不要随机朗读:保留 `autoSpeak: true`,把 `autoProbability` 设为 `0`。把 `autoSpeak` 整个关掉会让 tap 不再挂载,显式关键词也就看不到正文了;这时只保留 `voice_speak` 工具触发。
|
|
79
|
-
|
|
80
|
-
自动朗读依赖 Persona 把已挂载 World 的 `outputTap()` 扇进主 session;仓内 `cormini` 与 Coopanion 都这样做。只想要工具触发时,把 `autoSpeak` 关掉即可。
|
|
69
|
+
显式关键词依赖 Persona 把已挂载 World 的 `outputTap()` 扇进主 session;`cormini`、Coopanion 与 `Kmch AI · 网关` 0.1.1 起都这样做。模型说完正文再调 `end_turn` 不影响判定;与搜索等实际工具同轮的中间正文仍不念。`explicitTrigger: false` 会关掉关键词路径,但 `<tts>` 与 `voice_speak` 仍可用。
|
|
81
70
|
|
|
82
71
|
## 情绪、语速与标签
|
|
83
72
|
|
|
@@ -91,7 +80,7 @@ World 页的「模型」tab 选模型与音色,并按服务 `/v1/models` 返回
|
|
|
91
80
|
| `tags` | 不发情绪参数,只用正文标签。 | 正文标签。 |
|
|
92
81
|
| `none` | 都不发。 | — |
|
|
93
82
|
|
|
94
|
-
OpenAI 兼容条目把 `instructions` / `emotion` / `speed`
|
|
83
|
+
OpenAI 兼容条目把 `instructions` / `emotion` / `speed` 直接放进请求。IndexTTS2 把 `instructions` 或 `emotion` 映射成 `use_emotion_text: true` 与 `emotion_text`;这两个参数由语音的「默认情绪 / 默认表演指令」或单句 `emotion` / `instruct` 驱动,不在模型参数面板里单独出现。其他模型可在服务面板里覆盖 profile 的字段映射,语音不用改配置。
|
|
95
84
|
|
|
96
85
|
`response_format` 每次都带;`model` 与 `voice` 配了才带,工具参数里的音色优先于配置默认值。
|
|
97
86
|
|
|
@@ -131,6 +120,14 @@ OpenAI 兼容条目把 `instructions` / `emotion` / `speed` 直接放进请求
|
|
|
131
120
|
|
|
132
121
|
语音不唱歌:0.1.x 的 `voice_sing`、`【唱】` 前缀与 `sing*` 配置已删除。唱歌由服务的 `/v1/audio/sing` 和网关/唱歌插件接入。
|
|
133
122
|
|
|
123
|
+
## 从 0.3.x 迁移
|
|
124
|
+
|
|
125
|
+
| 0.3.x | 0.4.0 |
|
|
126
|
+
|---|---|
|
|
127
|
+
| `autoSpeak` / `autoProbability` / `autoFromText` / `autoTools` / `autoEvents` | 删除,配置里残留也会被忽略;语音不再自动朗读普通回复。 |
|
|
128
|
+
| 自动朗读 | 改用 `voice_speak`;或在消息里明确要求语音,让 `explicitKeywords` 命中。 |
|
|
129
|
+
| `explicitTrigger` / `explicitKeywords` / `explicitWindowSec` | 保留。 |
|
|
130
|
+
|
|
134
131
|
## 从 0.1.x 迁移
|
|
135
132
|
|
|
136
133
|
| 0.1.x | 0.2.0 |
|
|
@@ -152,7 +149,7 @@ pnpm smoke:player out.wav # 本机播放器,音量 0
|
|
|
152
149
|
pnpm smoke:chain http://127.0.0.1:9883 # 触发→过滤→经服务合成→tts.voice 全链路
|
|
153
150
|
```
|
|
154
151
|
|
|
155
|
-
装进 Cortico 后应看到:World 页多一张「Kmch AI · 语音」卡,服务灯、队列灯与 `service` / `model` / `voice` 徽标;「模型」tab 能选模型与音色,按模型调整声音与情绪,并按服务参数渲染字段、保存;服务在线时灯亮;让 bot 调 `voice_speak`
|
|
152
|
+
装进 Cortico 后应看到:World 页多一张「Kmch AI · 语音」卡,服务灯、队列灯与 `service` / `model` / `voice` 徽标;「模型」tab 能选模型与音色,按模型调整声音与情绪,并按服务参数渲染字段、保存;服务在线时灯亮;让 bot 调 `voice_speak` 能听见;说「念给我听」会朗读纯文本正文;`playback: event` 时时间线出现 `tts.voice`。
|
|
156
153
|
|
|
157
154
|
## 许可
|
|
158
155
|
|
package/describe.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"tools": [
|
|
3
3
|
{
|
|
4
4
|
"name": "voice_speak",
|
|
5
|
-
"description": "Say a line aloud in your voice through the person's speakers, or only synthesize it when silent is true. Use it when the person asks for voice, when this line needs a specific emotion or voice
|
|
5
|
+
"description": "Say a line aloud in your voice through the person's speakers, or only synthesize it when silent is true. Use it when the person asks for voice, or when this line needs a specific emotion or voice. It only makes sound; the visible reply still goes through the channel's own send tool. When the speech service supports paralinguistic tags, keep them inline in the text: [Laughter], [Sigh], [Breathing], [Uhm], [Question-ei] and the like are performed, not read out.",
|
|
6
6
|
"parameters": {
|
|
7
7
|
"type": "object",
|
|
8
8
|
"additionalProperties": false,
|
|
@@ -68,7 +68,7 @@
|
|
|
68
68
|
},
|
|
69
69
|
{
|
|
70
70
|
"name": "voice_style",
|
|
71
|
-
"description": "Read or change the standing voice style used by
|
|
71
|
+
"description": "Read or change the standing voice style used by later voice lines. Call with values to change them; call with no arguments to read the current style. Changes are saved to the World configuration and apply to later lines.",
|
|
72
72
|
"parameters": {
|
|
73
73
|
"type": "object",
|
|
74
74
|
"additionalProperties": false,
|
|
@@ -100,8 +100,8 @@
|
|
|
100
100
|
"write"
|
|
101
101
|
],
|
|
102
102
|
"summary": {
|
|
103
|
-
"zh": "
|
|
104
|
-
"en": "Read or change the standing voice style used by
|
|
103
|
+
"zh": "读或改后续语音使用的长期音色与情绪。",
|
|
104
|
+
"en": "Read or change the standing voice style used by later voice lines."
|
|
105
105
|
}
|
|
106
106
|
}
|
|
107
107
|
],
|
|
@@ -126,7 +126,7 @@
|
|
|
126
126
|
"key": "worlds.voice.envPrompt",
|
|
127
127
|
"title": "语音 · 环境提示词",
|
|
128
128
|
"role": "envPrompt",
|
|
129
|
-
"content": "语音由 Voice 提供:它只把文字变成声音,从对方的扬声器念出来,不负责把回复发到聊天渠道;可见的回复仍然走渠道自己的发送工具。\n\n
|
|
129
|
+
"content": "语音由 Voice 提供:它只把文字变成声音,从对方的扬声器念出来,不负责把回复发到聊天渠道;可见的回复仍然走渠道自己的发送工具。\n\n当前默认音色是 {{voice.voice}},情绪是 {{voice.emotion}},指令是 {{voice.instruct}}。普通回复不会自己出声,需要语音时明确调用 voice_speak。\n\n需要出声的时候:\n- 对方明确要语音、要你念出来、要听你的声音时,调用 voice_speak,把要念的正文放进 text。\n- 这一句需要特定情绪、语气、音色或语速时,调用 voice_speak 并填对应参数。想让同样的语气之后一直生效,调用 voice_style;它影响之后的语音。\n- 某一句只用于别的渠道(例如语音条),不要在本机播放:voice_speak 带 silent: true,音频随回执交给渠道。\n- 对方让你安静、停下时调用 voice_hush。\n\n文本规则:\n- text 只放真正要念的正文,不要放动作描写、Markdown、代码、网址或解释。\n- {{voice.modelPrompt}}\n- 不要把控制标记写进可见的回复正文;语音参数只放在工具调用里。\n"
|
|
130
130
|
}
|
|
131
131
|
]
|
|
132
132
|
}
|
package/package.json
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@kmchai/cortico-world-voice",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"type": "module",
|
|
5
|
-
"description": "Cortico World: voice tools,
|
|
5
|
+
"description": "Cortico World: voice tools, explicit semantic speech, emotion and paralinguistic tags, text filtering and local playback; speech is synthesized by the Kmch AI service over HTTP",
|
|
6
6
|
"author": "kmchai",
|
|
7
7
|
"license": "MIT",
|
|
8
8
|
"repository": {
|
|
@@ -35,11 +35,11 @@
|
|
|
35
35
|
"locales": {
|
|
36
36
|
"zh": {
|
|
37
37
|
"displayName": "Kmch AI · 语音",
|
|
38
|
-
"description": "语音 World
|
|
38
|
+
"description": "语音 World:语音工具、显式语义朗读、情绪与副语言标签、文本过滤、本机播放;协议与 HTTP 服务由 Kmch AI · 服务提供。"
|
|
39
39
|
},
|
|
40
40
|
"en": {
|
|
41
41
|
"displayName": "Kmch AI · Voice",
|
|
42
|
-
"description": "Voice World: tools,
|
|
42
|
+
"description": "Voice World: voice tools, explicit semantic speech, emotion and paralinguistic tags, text filtering and local playback; the Kmch AI service provides the protocol and HTTP API."
|
|
43
43
|
}
|
|
44
44
|
},
|
|
45
45
|
"tags": [
|
package/src/ENV_PROMPT.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
语音由 Voice 提供:它只把文字变成声音,从对方的扬声器念出来,不负责把回复发到聊天渠道;可见的回复仍然走渠道自己的发送工具。
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
当前默认音色是 {{voice.voice}},情绪是 {{voice.emotion}},指令是 {{voice.instruct}}。普通回复不会自己出声,需要语音时明确调用 voice_speak。
|
|
4
4
|
|
|
5
5
|
需要出声的时候:
|
|
6
|
-
- 对方明确要语音、要你念出来、要听你的声音时,调用 voice_speak,把要念的正文放进 text
|
|
7
|
-
- 这一句需要特定情绪、语气、音色或语速时,调用 voice_speak 并填对应参数。想让同样的语气之后一直生效,调用 voice_style
|
|
6
|
+
- 对方明确要语音、要你念出来、要听你的声音时,调用 voice_speak,把要念的正文放进 text。
|
|
7
|
+
- 这一句需要特定情绪、语气、音色或语速时,调用 voice_speak 并填对应参数。想让同样的语气之后一直生效,调用 voice_style;它影响之后的语音。
|
|
8
8
|
- 某一句只用于别的渠道(例如语音条),不要在本机播放:voice_speak 带 silent: true,音频随回执交给渠道。
|
|
9
9
|
- 对方让你安静、停下时调用 voice_hush。
|
|
10
10
|
|
package/src/config.ts
CHANGED
|
@@ -25,11 +25,6 @@ export const VOICE_DEFAULTS: VoiceConfigSection = {
|
|
|
25
25
|
speed: 1,
|
|
26
26
|
paralinguistic: 'auto',
|
|
27
27
|
|
|
28
|
-
autoSpeak: true,
|
|
29
|
-
autoProbability: 0.3,
|
|
30
|
-
autoFromText: true,
|
|
31
|
-
autoTools: 'pet_say, pet_ask, console_send',
|
|
32
|
-
autoEvents: '',
|
|
33
28
|
explicitTrigger: true,
|
|
34
29
|
explicitKeywords: '语音,声音,发语音,念给我听,读给我听,voice,tts',
|
|
35
30
|
explicitWindowSec: 120,
|
|
@@ -95,13 +90,6 @@ const TEXT = {
|
|
|
95
90
|
timeoutMs: '超时(毫秒)',
|
|
96
91
|
triggers: '触发',
|
|
97
92
|
triggersDesc: '',
|
|
98
|
-
autoSpeak: '自动朗读',
|
|
99
|
-
autoProbability: '自动朗读概率',
|
|
100
|
-
autoFromText: '从助手正文触发',
|
|
101
|
-
autoTools: '从工具参数触发',
|
|
102
|
-
autoToolsDesc: '逗号分隔的工具名;取参数里的 script / question / text / content / message / body。',
|
|
103
|
-
autoEvents: '从事件触发',
|
|
104
|
-
autoEventsDesc: '逗号分隔的事件类型;取 meta.body,没有就用正文。',
|
|
105
93
|
explicitTrigger: '语义显式请求',
|
|
106
94
|
explicitKeywords: '显式请求关键词',
|
|
107
95
|
explicitKeywordsDesc: '逗号/分号/换行分隔;最近的消息里出现就必定朗读。',
|
|
@@ -150,13 +138,6 @@ const TEXT = {
|
|
|
150
138
|
timeoutMs: 'Timeout (ms)',
|
|
151
139
|
triggers: 'Triggers',
|
|
152
140
|
triggersDesc: '',
|
|
153
|
-
autoSpeak: 'Automatic speech',
|
|
154
|
-
autoProbability: 'Automatic probability',
|
|
155
|
-
autoFromText: 'From assistant text',
|
|
156
|
-
autoTools: 'From tool arguments',
|
|
157
|
-
autoToolsDesc: 'Comma-separated tool names; reads script / question / text / content / message / body.',
|
|
158
|
-
autoEvents: 'From events',
|
|
159
|
-
autoEventsDesc: 'Comma-separated event types; reads meta.body, then the event text.',
|
|
160
141
|
explicitTrigger: 'Explicit semantic requests',
|
|
161
142
|
explicitKeywords: 'Explicit keywords',
|
|
162
143
|
explicitKeywordsDesc: 'Comma/semicolon/newline separated; a recent message containing one forces speech.',
|
|
@@ -217,11 +198,6 @@ export function voiceConfigGroups(language: Language): ConfigGroup[] {
|
|
|
217
198
|
type: 'object',
|
|
218
199
|
title: t.triggers,
|
|
219
200
|
properties: {
|
|
220
|
-
[`${p}autoSpeak`]: { type: 'boolean', title: t.autoSpeak, 'x-hot': true },
|
|
221
|
-
[`${p}autoProbability`]: { type: 'number', title: t.autoProbability, minimum: 0, maximum: 1, multipleOf: 0.05, 'x-hot': true },
|
|
222
|
-
[`${p}autoFromText`]: { type: 'boolean', title: t.autoFromText, 'x-hot': true },
|
|
223
|
-
[`${p}autoTools`]: { type: 'string', title: t.autoTools, description: t.autoToolsDesc, 'x-hot': true },
|
|
224
|
-
[`${p}autoEvents`]: { type: 'string', title: t.autoEvents, description: t.autoEventsDesc, 'x-hot': false },
|
|
225
201
|
[`${p}explicitTrigger`]: { type: 'boolean', title: t.explicitTrigger, 'x-hot': true },
|
|
226
202
|
[`${p}explicitKeywords`]: { type: 'string', title: t.explicitKeywords, description: t.explicitKeywordsDesc, 'x-hot': true },
|
|
227
203
|
[`${p}explicitWindowSec`]: { type: 'integer', title: t.explicitWindowSec, minimum: 0, maximum: 3600, 'x-hot': true },
|
package/src/definition.ts
CHANGED
|
@@ -19,7 +19,7 @@ export const VOICE: WorldDefinition<VoiceConfigSection> = {
|
|
|
19
19
|
tools: describeTools(VOICE_TOOL_DECLS, {
|
|
20
20
|
voice_speak: { zh: '现在念一句;可指定情绪、表演指令、音色与语速,也可以只合成不出声。', en: 'Say one line now, optionally with an emotion, direction, voice and speed, or synthesize it silently.' },
|
|
21
21
|
voice_hush: { zh: '立刻停声并清空排队。', en: 'Stop the voice and drop queued lines.' },
|
|
22
|
-
voice_style: { zh: '
|
|
22
|
+
voice_style: { zh: '读或改后续语音使用的长期音色与情绪。', en: 'Read or change the standing voice style used by later voice lines.' },
|
|
23
23
|
}),
|
|
24
24
|
events: [
|
|
25
25
|
{
|
package/src/models.ts
CHANGED
|
@@ -28,7 +28,7 @@ const INDEXTTS2: ModelInfo = {
|
|
|
28
28
|
match: ['index-tts2', 'index_tts2', 'index-tts2.5', 'index_tts2_5', '2.5'],
|
|
29
29
|
label: 'IndexTTS2',
|
|
30
30
|
tags: false,
|
|
31
|
-
prompt: '
|
|
31
|
+
prompt: '情绪用 voice_speak 的 emotion 或 instruct 传自然语言,服务会映射到 emotion_text;不要在正文写 emotion_text 或 [Laughter] 这类方括号标签。采样/时长参数在语音模型页配置,不要写进正文.',
|
|
32
32
|
};
|
|
33
33
|
|
|
34
34
|
const GENERIC: ModelInfo = {
|
package/src/speech/text.ts
CHANGED
package/src/tools.ts
CHANGED
|
@@ -6,7 +6,7 @@ export const VOICE_TOOL_DECLS: readonly ToolSpec[] = [
|
|
|
6
6
|
name: 'voice_speak',
|
|
7
7
|
description:
|
|
8
8
|
'Say a line aloud in your voice through the person\'s speakers, or only synthesize it when silent is true. '
|
|
9
|
-
+ 'Use it when the person asks for voice, when this line needs a specific emotion or voice
|
|
9
|
+
+ 'Use it when the person asks for voice, or when this line needs a specific emotion or voice. '
|
|
10
10
|
+ 'It only makes sound; the visible reply still goes through the channel\'s own send tool. '
|
|
11
11
|
+ 'When the speech service supports paralinguistic tags, keep them inline in the text: [Laughter], [Sigh], [Breathing], [Uhm], [Question-ei] and the like are performed, not read out.',
|
|
12
12
|
tags: ['speak'],
|
|
@@ -33,7 +33,7 @@ export const VOICE_TOOL_DECLS: readonly ToolSpec[] = [
|
|
|
33
33
|
{
|
|
34
34
|
name: 'voice_style',
|
|
35
35
|
description:
|
|
36
|
-
'Read or change the standing voice style used by
|
|
36
|
+
'Read or change the standing voice style used by later voice lines. Call with values to change them; call with no arguments to read the current style. '
|
|
37
37
|
+ 'Changes are saved to the World configuration and apply to later lines.',
|
|
38
38
|
tags: ['write'],
|
|
39
39
|
parameters: {
|
package/src/triggers.ts
CHANGED
|
@@ -1,13 +1,12 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* 触发层:从主 session
|
|
3
|
-
*
|
|
2
|
+
* 触发层:从主 session 的输出流里认出显式请求的正文,以及从事件库里认出
|
|
3
|
+
* 「刚才有人要语音」。这一层不认识 TTS,也不掷概率。
|
|
4
4
|
*/
|
|
5
5
|
import type { EventEnvelope, OutputTap, WorldHost } from 'cortico/core/types.ts';
|
|
6
6
|
import type { StreamEvent } from 'cortico/protocol/open-responses/index.ts';
|
|
7
|
-
import type { VoiceConfigSection } from './types.ts';
|
|
8
7
|
|
|
9
|
-
/**
|
|
10
|
-
const
|
|
8
|
+
/** 只结束这一轮的控制工具,不算「这一轮做了别的事」。 */
|
|
9
|
+
const FLOW_TOOLS = new Set(['end_turn']);
|
|
11
10
|
|
|
12
11
|
export function splitList(raw: string): string[] {
|
|
13
12
|
return raw
|
|
@@ -16,65 +15,41 @@ export function splitList(raw: string): string[] {
|
|
|
16
15
|
.filter(Boolean);
|
|
17
16
|
}
|
|
18
17
|
|
|
19
|
-
export function toolNames(cfg: VoiceConfigSection): Set<string> {
|
|
20
|
-
return new Set(splitList(cfg.autoTools));
|
|
21
|
-
}
|
|
22
|
-
|
|
23
|
-
export function eventTypes(cfg: VoiceConfigSection): Set<string> {
|
|
24
|
-
return new Set(splitList(cfg.autoEvents));
|
|
25
|
-
}
|
|
26
|
-
|
|
27
|
-
/** 从工具调用参数里取正文;取不到返回空串。 */
|
|
28
|
-
export function textOfToolArgs(raw: string): string {
|
|
29
|
-
if (!raw) return '';
|
|
30
|
-
let parsed: unknown;
|
|
31
|
-
try { parsed = JSON.parse(raw); } catch { return ''; }
|
|
32
|
-
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return '';
|
|
33
|
-
const obj = parsed as Record<string, unknown>;
|
|
34
|
-
for (const field of TEXT_FIELDS) {
|
|
35
|
-
const value = obj[field];
|
|
36
|
-
if (typeof value === 'string' && value.trim()) return value;
|
|
37
|
-
}
|
|
38
|
-
return '';
|
|
39
|
-
}
|
|
40
|
-
|
|
41
18
|
export interface TapOptions {
|
|
42
|
-
|
|
43
|
-
/** 一段完整的话;`source` 形如 `text` 或 `tool:pet_say`。 */
|
|
19
|
+
/** 一段完整的话;`source` 形如 `text`。 */
|
|
44
20
|
onText(text: string, source: string): void;
|
|
45
21
|
/** 模型调用被取消或打断;可以用来停声。 */
|
|
46
22
|
onAbort?(): void;
|
|
47
23
|
}
|
|
48
24
|
|
|
49
25
|
/**
|
|
50
|
-
* 主 session
|
|
51
|
-
*
|
|
26
|
+
* 主 session 的输出旁路。一轮里只有纯文本、没有实质函数调用时,把正文交出去;
|
|
27
|
+
* 是否朗读由显式关键词或 `<tts>` 决定。
|
|
52
28
|
*/
|
|
53
29
|
interface StreamLike {
|
|
54
30
|
type: string;
|
|
55
31
|
delta?: string;
|
|
56
|
-
arguments?: string;
|
|
57
32
|
output_index?: number;
|
|
58
|
-
item?: { type?: string; name?: string
|
|
33
|
+
item?: { type?: string; name?: string } | null;
|
|
59
34
|
}
|
|
60
35
|
|
|
61
36
|
export class VoiceTap implements OutputTap {
|
|
62
37
|
private text = '';
|
|
63
|
-
|
|
64
|
-
private readonly
|
|
65
|
-
|
|
38
|
+
/** 这一轮出现过的函数调用:output_index -> name;名字未知时存空串。 */
|
|
39
|
+
private readonly callNames = new Map<number, string>();
|
|
40
|
+
/** 没有 output_index 的函数调用;无法判断是不是 end_turn,按实质调用处理。 */
|
|
41
|
+
private orphanCall = false;
|
|
66
42
|
|
|
67
43
|
constructor(private readonly opts: TapOptions) {}
|
|
68
44
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
if (
|
|
76
|
-
this.
|
|
77
|
-
this.opts.onText(text, `tool:${name}`);
|
|
45
|
+
private noteCall(index: number, name: string): void {
|
|
46
|
+
if (index < 0) {
|
|
47
|
+
if (name && FLOW_TOOLS.has(name)) return;
|
|
48
|
+
this.orphanCall = true;
|
|
49
|
+
return;
|
|
50
|
+
}
|
|
51
|
+
if (name) this.callNames.set(index, name);
|
|
52
|
+
else if (!this.callNames.has(index)) this.callNames.set(index, '');
|
|
78
53
|
}
|
|
79
54
|
|
|
80
55
|
onEvent(event: StreamEvent): void {
|
|
@@ -85,39 +60,34 @@ export class VoiceTap implements OutputTap {
|
|
|
85
60
|
return;
|
|
86
61
|
}
|
|
87
62
|
if (e.type === 'response.output_item.added' && e.item?.type === 'function_call') {
|
|
88
|
-
this.
|
|
89
|
-
if (e.item.name) this.names.set(index, e.item.name);
|
|
63
|
+
this.noteCall(index, e.item.name ?? '');
|
|
90
64
|
return;
|
|
91
65
|
}
|
|
92
66
|
// 有的传输只给参数结束事件,不带完整 item;名字从 added 里取。
|
|
93
67
|
if (e.type === 'response.function_call_arguments.done') {
|
|
94
|
-
|
|
95
|
-
if (name) this.takeCall(index, name, e.arguments ?? '');
|
|
68
|
+
this.noteCall(index, this.callNames.get(index) ?? '');
|
|
96
69
|
return;
|
|
97
70
|
}
|
|
98
71
|
if (e.type === 'response.output_item.done' && e.item?.type === 'function_call') {
|
|
99
|
-
this.
|
|
100
|
-
const name = e.item.name ?? this.names.get(index) ?? '';
|
|
101
|
-
if (name) this.takeCall(index, name, e.item.arguments ?? '');
|
|
102
|
-
this.names.delete(index);
|
|
72
|
+
this.noteCall(index, e.item.name ?? this.callNames.get(index) ?? '');
|
|
103
73
|
}
|
|
104
74
|
}
|
|
105
75
|
|
|
106
76
|
onRoundEnd(): void {
|
|
107
77
|
const text = this.text;
|
|
108
78
|
this.text = '';
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
this.
|
|
113
|
-
|
|
79
|
+
// 只有结束轮次的调用不算实质动作;纯文本回复照样可以交给显式触发判断。
|
|
80
|
+
const nonFlow = this.orphanCall
|
|
81
|
+
|| [...this.callNames.values()].some((name) => name === '' || !FLOW_TOOLS.has(name));
|
|
82
|
+
this.callNames.clear();
|
|
83
|
+
this.orphanCall = false;
|
|
84
|
+
if (!nonFlow && text.trim()) this.opts.onText(text, 'text');
|
|
114
85
|
}
|
|
115
86
|
|
|
116
87
|
onAbort(): void {
|
|
117
88
|
this.text = '';
|
|
118
|
-
this.
|
|
119
|
-
this.
|
|
120
|
-
this.emitted.clear();
|
|
89
|
+
this.callNames.clear();
|
|
90
|
+
this.orphanCall = false;
|
|
121
91
|
this.opts.onAbort?.();
|
|
122
92
|
}
|
|
123
93
|
}
|
|
@@ -168,47 +138,3 @@ export function hasKeyword(text: string, keywords: string): boolean {
|
|
|
168
138
|
const lower = text.toLowerCase();
|
|
169
139
|
return splitList(keywords).some((word) => lower.includes(word.toLowerCase()));
|
|
170
140
|
}
|
|
171
|
-
|
|
172
|
-
export interface EventPollOptions {
|
|
173
|
-
host: WorldHost;
|
|
174
|
-
types: Set<string>;
|
|
175
|
-
onText(text: string, source: string): void;
|
|
176
|
-
intervalMs?: number;
|
|
177
|
-
log(message: string, data?: unknown): void;
|
|
178
|
-
}
|
|
179
|
-
|
|
180
|
-
/** 轮询事件库;只处理启动之后的新事件,不补读。 */
|
|
181
|
-
export class EventPoller {
|
|
182
|
-
private timer: NodeJS.Timeout | null = null;
|
|
183
|
-
private cursor = 0;
|
|
184
|
-
|
|
185
|
-
constructor(private readonly opts: EventPollOptions) {}
|
|
186
|
-
|
|
187
|
-
start(): void {
|
|
188
|
-
if (this.timer) return;
|
|
189
|
-
try { this.cursor = this.opts.host.store.latestCursor(); } catch { this.cursor = 0; }
|
|
190
|
-
const tick = (): void => {
|
|
191
|
-
let events: EventEnvelope[];
|
|
192
|
-
try {
|
|
193
|
-
events = this.opts.host.store.range({ fromCursor: this.cursor + 1, limit: 50 });
|
|
194
|
-
} catch (error) {
|
|
195
|
-
this.opts.log('事件轮询失败', { err: error instanceof Error ? error.message : String(error) });
|
|
196
|
-
return;
|
|
197
|
-
}
|
|
198
|
-
for (const event of events) {
|
|
199
|
-
this.cursor = Math.max(this.cursor, event.cursor);
|
|
200
|
-
if (event.source === 'voice') continue;
|
|
201
|
-
if (!this.opts.types.has(event.type)) continue;
|
|
202
|
-
const text = textOfEvent(event);
|
|
203
|
-
if (text.trim()) this.opts.onText(text, `event:${event.type}`);
|
|
204
|
-
}
|
|
205
|
-
};
|
|
206
|
-
this.timer = setInterval(tick, this.opts.intervalMs ?? 1000);
|
|
207
|
-
this.timer.unref?.();
|
|
208
|
-
}
|
|
209
|
-
|
|
210
|
-
stop(): void {
|
|
211
|
-
if (this.timer) clearInterval(this.timer);
|
|
212
|
-
this.timer = null;
|
|
213
|
-
}
|
|
214
|
-
}
|
package/src/types.ts
CHANGED
|
@@ -46,14 +46,6 @@ export interface VoiceConfigSection {
|
|
|
46
46
|
paralinguistic: ParalinguisticMode;
|
|
47
47
|
|
|
48
48
|
// 触发
|
|
49
|
-
autoSpeak: boolean;
|
|
50
|
-
/** 自动朗读的概率,0–1。显式请求与工具调用不受它限制。 */
|
|
51
|
-
autoProbability: number;
|
|
52
|
-
autoFromText: boolean;
|
|
53
|
-
/** 逗号分隔的工具名;这些工具的参数会被当作要念的正文。 */
|
|
54
|
-
autoTools: string;
|
|
55
|
-
/** 逗号分隔的事件类型;这些事件的正文会被念出来。 */
|
|
56
|
-
autoEvents: string;
|
|
57
49
|
explicitTrigger: boolean;
|
|
58
50
|
explicitKeywords: string;
|
|
59
51
|
/** 往回看多少秒内的用户消息算「刚才」。 */
|
package/src/world.ts
CHANGED
|
@@ -24,7 +24,7 @@ import { ServiceClient, type SpeechService } from './service.ts';
|
|
|
24
24
|
import { splitSpeech, truncateSpeech } from './speech/chunk.ts';
|
|
25
25
|
import { buildInstruct, resolveStyle } from './speech/style.ts';
|
|
26
26
|
import { filterSpeech } from './speech/text.ts';
|
|
27
|
-
import {
|
|
27
|
+
import { hasKeyword, latestUserText, VoiceTap } from './triggers.ts';
|
|
28
28
|
import { VOICE_TOOL_DECLS } from './tools.ts';
|
|
29
29
|
import type {
|
|
30
30
|
EmotionMode, ParalinguisticMode, ServiceHealth, TtsAudio, TtsRequest, VoiceConfigSection,
|
|
@@ -39,7 +39,6 @@ const HEALTH_TTL_MS = 15_000;
|
|
|
39
39
|
export interface VoiceWorldDeps {
|
|
40
40
|
fetch: typeof fetch;
|
|
41
41
|
createPlayer(log: Logger): AudioPlayer | null;
|
|
42
|
-
rng(): number;
|
|
43
42
|
now(): number;
|
|
44
43
|
sleep(ms: number): Promise<void>;
|
|
45
44
|
/** 测试用:替换服务客户端。 */
|
|
@@ -76,7 +75,6 @@ function defaultDeps(): VoiceWorldDeps {
|
|
|
76
75
|
return {
|
|
77
76
|
fetch: ((input: Parameters<typeof fetch>[0], init?: Parameters<typeof fetch>[1]) => fetch(input, init)) as typeof fetch,
|
|
78
77
|
createPlayer,
|
|
79
|
-
rng: () => Math.random(),
|
|
80
78
|
now: () => Date.now(),
|
|
81
79
|
sleep: (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)),
|
|
82
80
|
};
|
|
@@ -124,7 +122,6 @@ export class VoiceWorld implements World {
|
|
|
124
122
|
private readonly tap: VoiceTap;
|
|
125
123
|
private readonly service: SpeechService;
|
|
126
124
|
private host: WorldHost | null = null;
|
|
127
|
-
private poller: EventPoller | null = null;
|
|
128
125
|
private healthTimer: NodeJS.Timeout | null = null;
|
|
129
126
|
private started = false;
|
|
130
127
|
private tempDir = '';
|
|
@@ -157,7 +154,6 @@ export class VoiceWorld implements World {
|
|
|
157
154
|
onError: (job, message) => { void this.pushFailure(message, { source: job.source }); },
|
|
158
155
|
});
|
|
159
156
|
this.tap = new VoiceTap({
|
|
160
|
-
cfg: () => this.cfg,
|
|
161
157
|
onText: (text, source) => this.consider(text, source),
|
|
162
158
|
onAbort: () => this.queue.hush(),
|
|
163
159
|
});
|
|
@@ -175,7 +171,6 @@ export class VoiceWorld implements World {
|
|
|
175
171
|
envPromptVars(): Record<string, string> {
|
|
176
172
|
const cfg = this.styled;
|
|
177
173
|
return {
|
|
178
|
-
'voice.auto': this.cfg.autoSpeak ? `${Math.round(this.cfg.autoProbability * 100)}%` : 'off',
|
|
179
174
|
'voice.voice': cfg.voice || 'default',
|
|
180
175
|
'voice.emotion': cfg.defaultEmotion || 'neutral',
|
|
181
176
|
'voice.instruct': cfg.defaultInstruct || 'none',
|
|
@@ -247,7 +242,6 @@ export class VoiceWorld implements World {
|
|
|
247
242
|
path: ENV_PROMPT_FILE,
|
|
248
243
|
role: 'envPrompt',
|
|
249
244
|
vars: [
|
|
250
|
-
{ name: 'voice.auto', description: 'Automatic reading probability, or off.' },
|
|
251
245
|
{ name: 'voice.voice', description: 'Current voice id.' },
|
|
252
246
|
{ name: 'voice.emotion', description: 'Standing emotion.' },
|
|
253
247
|
{ name: 'voice.instruct', description: 'Standing direction.' },
|
|
@@ -423,9 +417,7 @@ export class VoiceWorld implements World {
|
|
|
423
417
|
}
|
|
424
418
|
|
|
425
419
|
outputTap() {
|
|
426
|
-
|
|
427
|
-
if (!cfg.autoSpeak) return undefined;
|
|
428
|
-
if (!cfg.autoFromText && splitList(cfg.autoTools).length === 0) return undefined;
|
|
420
|
+
// 概率触发已删除;旁路只用于显式关键词与 <tts> 的判定。
|
|
429
421
|
return this.tap;
|
|
430
422
|
}
|
|
431
423
|
|
|
@@ -435,14 +427,13 @@ export class VoiceWorld implements World {
|
|
|
435
427
|
this.log.info('语音 World 启动', {
|
|
436
428
|
service: serviceHost(this.cfg.serviceUrl),
|
|
437
429
|
playback: this.cfg.playback,
|
|
438
|
-
|
|
430
|
+
explicit: this.cfg.explicitTrigger,
|
|
439
431
|
});
|
|
440
432
|
if (this.cfg.playback === 'local') {
|
|
441
433
|
void this.player?.warm(this.ensureTempDir()).catch((error: unknown) => {
|
|
442
434
|
this.log.warn('播放器预热失败', { err: error instanceof Error ? error.message : String(error) });
|
|
443
435
|
});
|
|
444
436
|
}
|
|
445
|
-
this.syncPoller();
|
|
446
437
|
void this.probe();
|
|
447
438
|
this.healthTimer = setInterval(() => { void this.probe(); }, HEALTH_INTERVAL_MS);
|
|
448
439
|
this.healthTimer.unref?.();
|
|
@@ -452,8 +443,6 @@ export class VoiceWorld implements World {
|
|
|
452
443
|
this.started = false;
|
|
453
444
|
if (this.healthTimer) clearInterval(this.healthTimer);
|
|
454
445
|
this.healthTimer = null;
|
|
455
|
-
this.poller?.stop();
|
|
456
|
-
this.poller = null;
|
|
457
446
|
await this.queue.close();
|
|
458
447
|
if (this.tempDir) {
|
|
459
448
|
try { rmSync(this.tempDir, { recursive: true, force: true }); } catch { /* 临时目录 */ }
|
|
@@ -478,8 +467,6 @@ export class VoiceWorld implements World {
|
|
|
478
467
|
private consider(raw: string, source: string): void {
|
|
479
468
|
if (!this.started) return;
|
|
480
469
|
const cfg = this.cfg;
|
|
481
|
-
if (!cfg.autoSpeak) return;
|
|
482
|
-
|
|
483
470
|
const filtered = filterSpeech(raw, this.styled);
|
|
484
471
|
if (!filtered.text) return;
|
|
485
472
|
|
|
@@ -487,14 +474,14 @@ export class VoiceWorld implements World {
|
|
|
487
474
|
? latestUserText(this.host!, cfg.explicitWindowSec, this.deps.now())
|
|
488
475
|
: '';
|
|
489
476
|
const explicit = filtered.forced || (cfg.explicitTrigger && hasKeyword(userText, cfg.explicitKeywords));
|
|
490
|
-
if (!explicit
|
|
477
|
+
if (!explicit) return;
|
|
491
478
|
if (!this.fresh(filtered.text)) return;
|
|
492
479
|
|
|
493
480
|
this.enqueue(filtered.text, {
|
|
494
481
|
source,
|
|
495
482
|
...(filtered.emotion ? { emotion: filtered.emotion } : {}),
|
|
496
483
|
});
|
|
497
|
-
this.log.debug('
|
|
484
|
+
this.log.debug('显式朗读', { source, chars: [...filtered.text].length });
|
|
498
485
|
}
|
|
499
486
|
|
|
500
487
|
private enqueue(text: string, opts: EnqueueOptions): ReturnType<SpeechQueue['enqueue']> | null {
|
|
@@ -611,8 +598,7 @@ export class VoiceWorld implements World {
|
|
|
611
598
|
private styleSummary(): string {
|
|
612
599
|
const cfg = this.styled;
|
|
613
600
|
return `Voice: ${cfg.voice || 'default'}; emotion: ${cfg.defaultEmotion || 'neutral'}; `
|
|
614
|
-
+ `direction: ${cfg.defaultInstruct || 'none'}; speed: ${cfg.speed}
|
|
615
|
-
+ `automatic: ${this.cfg.autoSpeak ? `${Math.round(this.cfg.autoProbability * 100)}%` : 'off'}.`;
|
|
601
|
+
+ `direction: ${cfg.defaultInstruct || 'none'}; speed: ${cfg.speed}.`;
|
|
616
602
|
}
|
|
617
603
|
|
|
618
604
|
private spokenOutcome(audio: TtsAudio | undefined, seconds: number, silent: boolean): ToolOutcome {
|
|
@@ -708,7 +694,7 @@ export class VoiceWorld implements World {
|
|
|
708
694
|
private async pushFailure(detail: string, meta: Record<string, unknown>): Promise<void> {
|
|
709
695
|
const host = this.host;
|
|
710
696
|
if (!host) return;
|
|
711
|
-
//
|
|
697
|
+
// 服务长时间不可用时,连续失败会反复出现;事件留一条,日志照常。
|
|
712
698
|
const now = this.deps.now();
|
|
713
699
|
if (now - this.lastFailureAt < 30_000) return;
|
|
714
700
|
this.lastFailureAt = now;
|
|
@@ -760,23 +746,6 @@ export class VoiceWorld implements World {
|
|
|
760
746
|
}
|
|
761
747
|
}
|
|
762
748
|
|
|
763
|
-
private syncPoller(): void {
|
|
764
|
-
const types = eventTypes(this.cfg);
|
|
765
|
-
if (types.size === 0) {
|
|
766
|
-
this.poller?.stop();
|
|
767
|
-
this.poller = null;
|
|
768
|
-
return;
|
|
769
|
-
}
|
|
770
|
-
if (this.poller) return;
|
|
771
|
-
this.poller = new EventPoller({
|
|
772
|
-
host: this.host!,
|
|
773
|
-
types,
|
|
774
|
-
onText: (text, source) => this.consider(text, source),
|
|
775
|
-
log: (message, data) => this.log.warn(message, data),
|
|
776
|
-
});
|
|
777
|
-
this.poller.start();
|
|
778
|
-
}
|
|
779
|
-
|
|
780
749
|
private ensureTempDir(): string {
|
|
781
750
|
if (this.tempDir) return this.tempDir;
|
|
782
751
|
const base = join(this.opts.dataDir || tmpdir(), 'voice');
|