@agfpd/voicetalk-runtime 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/README.md +18 -2
- package/docs/ru/README.md +18 -2
- package/package.json +1 -1
- package/src/audio.ts +46 -1
- package/src/voiceSession.ts +273 -57
- package/src/wake.ts +26 -1
package/docs/README.md
CHANGED
|
@@ -48,9 +48,19 @@ voice-connect, microphone).
|
|
|
48
48
|
2. voice-connect transcribes the utterance (locally, on this host).
|
|
49
49
|
3. If it opens by addressing a peer, the body is delivered to that peer as an IAP
|
|
50
50
|
envelope from the owner's identity, framed for the spoken modality.
|
|
51
|
+
3.5. Fragments are COMPOSED into one turn. Endpointing cuts audio, not thoughts: a
|
|
52
|
+
pause inside a sentence would otherwise become a second envelope and earn a second
|
|
53
|
+
spoken answer. The turn is sent once the owner has been silent for ~3 s since their
|
|
54
|
+
last word — and never while they are still speaking or a fragment is still being
|
|
55
|
+
recognized. A fragment left hanging on a comma or a conjunction waits longer.
|
|
51
56
|
4. The agent's answer arrives on stdin, is spoken through voice-connect on the host
|
|
52
57
|
speaker, and the microphone stays open for ~9 s so the owner can continue without
|
|
53
|
-
naming the peer again.
|
|
58
|
+
naming the peer again. When that window closes, a soft low tone says so.
|
|
59
|
+
|
|
60
|
+
Inside an open conversation a wake word for ANOTHER peer must look deliberate — the
|
|
61
|
+
name heard letter for letter, followed by a real request — otherwise the fragment goes
|
|
62
|
+
to the peer already listening. Mis-hearing a name should cost a stray line, never the
|
|
63
|
+
whole conversation.
|
|
54
64
|
|
|
55
65
|
Earcons mark the states: rising = the microphone is yours, short descending = heard
|
|
56
66
|
you, double descending = nothing was routed. Processing is silent.
|
|
@@ -58,7 +68,13 @@ you, double descending = nothing was routed. Processing is silent.
|
|
|
58
68
|
Everything is tunable by env: `VOICETALK_AUDIO=0` (text-only, no microphone),
|
|
59
69
|
`VOICETALK_FOLLOWUP_MS`, `VOICETALK_HANGOVER_MS`, `VOICETALK_VAD_MIN_RMS`,
|
|
60
70
|
`VOICETALK_VAD_RATIO`, `VOICETALK_STT_LANG`, `VOICETALK_STT_ENGINE`,
|
|
61
|
-
`VOICETALK_TTS_MAX_CHARS`, `VOICETALK_EARCONS=0
|
|
71
|
+
`VOICETALK_TTS_MAX_CHARS`, `VOICETALK_EARCONS=0`, `VOICETALK_COMPOSE_MS`,
|
|
72
|
+
`VOICETALK_COMPOSE_UNFINISHED_MS`, `VOICETALK_TURN_MAX_MS`.
|
|
73
|
+
|
|
74
|
+
Two seams exist for testing turn-taking without touching a live channel:
|
|
75
|
+
`VOICETALK_MIC_FILE=<pcm>` replays a recording through the microphone path (paced in
|
|
76
|
+
real time) and `VOICETALK_PLAYBACK=0` keeps the speaker silent while synthesis is
|
|
77
|
+
still performed and timed. `scripts/replay-dialogue.sh` wires both into a sandbox.
|
|
62
78
|
|
|
63
79
|
## Requirements
|
|
64
80
|
|
package/docs/ru/README.md
CHANGED
|
@@ -48,8 +48,18 @@ self-config-хука (`VOICETALK_WAKE_WORDS`, `VOICETALK_DEFAULT_TARGET`,
|
|
|
48
48
|
2. voice-connect распознаёт реплику (локально, на этом хосте).
|
|
49
49
|
3. Если реплика начинается с обращения к пиру, тело уходит ему IAP-конвертом от
|
|
50
50
|
личности владельца, с рамкой голосовой модальности.
|
|
51
|
+
3.5. Фрагменты СОБИРАЮТСЯ в один ход. Эндпойнтер режет звук, а не мысль: пауза внутри
|
|
52
|
+
фразы иначе стала бы вторым конвертом и заработала бы второй голосовой ответ. Ход
|
|
53
|
+
уходит, когда владелец молчит ~3 с после последнего слова, и никогда — пока он
|
|
54
|
+
ещё говорит или пока фрагмент ещё распознаётся. Фрагмент, оборванный на запятой
|
|
55
|
+
или союзе, ждёт дольше.
|
|
51
56
|
4. Ответ агента приходит на stdin, озвучивается через voice-connect в динамик, и
|
|
52
|
-
микрофон остаётся открытым ~9 с — продолжить можно без повторного обращения.
|
|
57
|
+
микрофон остаётся открытым ~9 с — продолжить можно без повторного обращения. Когда
|
|
58
|
+
окно закрывается, звучит тихий низкий тон.
|
|
59
|
+
|
|
60
|
+
Внутри открытого диалога wake-слово ДРУГОГО пира обязано выглядеть намеренным — имя
|
|
61
|
+
услышано буква в букву и за ним идёт настоящая просьба, — иначе фрагмент уходит тому,
|
|
62
|
+
кто уже слушает. Ослышка должна стоить лишней строки, а не всего разговора.
|
|
53
63
|
|
|
54
64
|
Earcon'ы размечают состояния: восходящий — микрофон ваш, короткий нисходящий —
|
|
55
65
|
принял, двойной нисходящий — не отправлено. Обработка идёт молча.
|
|
@@ -57,7 +67,13 @@ Earcon'ы размечают состояния: восходящий — мик
|
|
|
57
67
|
Всё настраивается через env: `VOICETALK_AUDIO=0` (текстовый режим, без микрофона),
|
|
58
68
|
`VOICETALK_FOLLOWUP_MS`, `VOICETALK_HANGOVER_MS`, `VOICETALK_VAD_MIN_RMS`,
|
|
59
69
|
`VOICETALK_VAD_RATIO`, `VOICETALK_STT_LANG`, `VOICETALK_STT_ENGINE`,
|
|
60
|
-
`VOICETALK_TTS_MAX_CHARS`, `VOICETALK_EARCONS=0
|
|
70
|
+
`VOICETALK_TTS_MAX_CHARS`, `VOICETALK_EARCONS=0`, `VOICETALK_COMPOSE_MS`,
|
|
71
|
+
`VOICETALK_COMPOSE_UNFINISHED_MS`, `VOICETALK_TURN_MAX_MS`.
|
|
72
|
+
|
|
73
|
+
Для проверки turn-taking, не трогая живой канал, есть два шва: `VOICETALK_MIC_FILE=<pcm>`
|
|
74
|
+
проигрывает запись через тот же путь, что и микрофон (в реальном темпе), а
|
|
75
|
+
`VOICETALK_PLAYBACK=0` держит динамик молча, оставляя синтез и его замеры. Оба сведены
|
|
76
|
+
в `scripts/replay-dialogue.sh`.
|
|
61
77
|
|
|
62
78
|
## Требования
|
|
63
79
|
|
package/package.json
CHANGED
package/src/audio.ts
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
import { spawn, type ChildProcessByStdio } from 'child_process'
|
|
11
11
|
import type { Readable } from 'stream'
|
|
12
|
-
import { writeFileSync } from 'fs'
|
|
12
|
+
import { readFileSync, writeFileSync } from 'fs'
|
|
13
13
|
|
|
14
14
|
export const SAMPLE_RATE = 16_000
|
|
15
15
|
export const FRAME_MS = 20
|
|
@@ -125,6 +125,13 @@ export class Segmenter {
|
|
|
125
125
|
this.warmupFrames = Math.round((opts.warmupMs ?? 600) / FRAME_MS)
|
|
126
126
|
}
|
|
127
127
|
|
|
128
|
+
/** Is the endpointer inside an utterance right now? The turn layer asks before it
|
|
129
|
+
* decides a thought is finished: the owner resuming mid-pause is the difference
|
|
130
|
+
* between one envelope and two. */
|
|
131
|
+
get speaking(): boolean {
|
|
132
|
+
return this.inSpeech
|
|
133
|
+
}
|
|
134
|
+
|
|
128
135
|
get noiseFloor(): number {
|
|
129
136
|
return this.floor < 0 ? 0 : this.floor
|
|
130
137
|
}
|
|
@@ -293,6 +300,11 @@ export async function resolveInputDevice(
|
|
|
293
300
|
|
|
294
301
|
export type MicOptions = {
|
|
295
302
|
deviceIndex: number
|
|
303
|
+
/** Replay a recording INSTEAD of opening the microphone (test seam). Frames are paced
|
|
304
|
+
* in real time, so endpointing and the compose window behave exactly as they do on a
|
|
305
|
+
* live mic — which is the whole point: turn-taking bugs only reproduce against a
|
|
306
|
+
* clock. Accepts a WAV or headerless mono 16-bit 16 kHz PCM file. */
|
|
307
|
+
filePath?: string
|
|
296
308
|
ffmpegBin?: string
|
|
297
309
|
onData: (chunk: Buffer) => void
|
|
298
310
|
onError?: (detail: string) => void
|
|
@@ -333,6 +345,10 @@ export class Microphone {
|
|
|
333
345
|
private spawnChild(): void {
|
|
334
346
|
if (this.stopped) return
|
|
335
347
|
this.attempt += 1
|
|
348
|
+
if (this.opts.filePath) {
|
|
349
|
+
this.replayFile(this.opts.filePath)
|
|
350
|
+
return
|
|
351
|
+
}
|
|
336
352
|
const bin = this.opts.ffmpegBin ?? 'ffmpeg'
|
|
337
353
|
const args = [
|
|
338
354
|
'-hide_banner',
|
|
@@ -368,6 +384,35 @@ export class Microphone {
|
|
|
368
384
|
})
|
|
369
385
|
}
|
|
370
386
|
|
|
387
|
+
/** Feed a recording through the same path the microphone uses, one 20 ms frame per
|
|
388
|
+
* 20 ms, then fall silent (a finished recording is a quiet room, not a dead mic). */
|
|
389
|
+
private replayFile(path: string): void {
|
|
390
|
+
let pcm: Buffer
|
|
391
|
+
try {
|
|
392
|
+
const raw = readFileSync(path)
|
|
393
|
+
// Skip a RIFF header if present; anything else is treated as raw PCM.
|
|
394
|
+
pcm = raw.subarray(0, 4).toString() === 'RIFF' ? raw.subarray(44) : raw
|
|
395
|
+
} catch (err) {
|
|
396
|
+
this.opts.onError?.(`cannot replay ${path}: ${err instanceof Error ? err.message : String(err)}`)
|
|
397
|
+
return
|
|
398
|
+
}
|
|
399
|
+
this.opts.onStart?.(this.attempt)
|
|
400
|
+
let offset = 0
|
|
401
|
+
const tick = () => {
|
|
402
|
+
if (this.stopped) return
|
|
403
|
+
if (offset >= pcm.length) {
|
|
404
|
+
// Keep the stream alive with silence so compose/window timers still fire.
|
|
405
|
+
this.opts.onData(Buffer.alloc(FRAME_BYTES))
|
|
406
|
+
this.timer = setTimeout(tick, FRAME_MS)
|
|
407
|
+
return
|
|
408
|
+
}
|
|
409
|
+
this.opts.onData(Buffer.from(pcm.subarray(offset, offset + FRAME_BYTES)))
|
|
410
|
+
offset += FRAME_BYTES
|
|
411
|
+
this.timer = setTimeout(tick, FRAME_MS)
|
|
412
|
+
}
|
|
413
|
+
tick()
|
|
414
|
+
}
|
|
415
|
+
|
|
371
416
|
private scheduleRestart(): void {
|
|
372
417
|
if (this.stopped || this.timer) return
|
|
373
418
|
const delay = Math.min(30_000, 1_000 * Math.min(8, this.attempt))
|
package/src/voiceSession.ts
CHANGED
|
@@ -1,9 +1,17 @@
|
|
|
1
1
|
// voiceSession.ts — the live voice session (Фаза 3): the runtime's ears, its turn-taking
|
|
2
2
|
// and its mouth, wired between audio.ts (host mechanics) and voiceConnect.ts (speech).
|
|
3
3
|
//
|
|
4
|
-
// INBOUND mic → endpointing → STT → address match → deliver to the fleet
|
|
4
|
+
// INBOUND mic → endpointing → STT → address match → COMPOSE → deliver to the fleet
|
|
5
5
|
// OUTBOUND agent envelope → TTS → speaker, then an open-mic follow-up window
|
|
6
6
|
//
|
|
7
|
+
// A pause inside a sentence is not the end of a thought. Endpointing (fast, 700 ms) only
|
|
8
|
+
// cuts AUDIO into segments so recognition can start early; what the fleet receives is a
|
|
9
|
+
// TURN — every segment of one thought, joined, delivered ONCE when the owner has actually
|
|
10
|
+
// finished (the compose window). Without that, one spoken thought became two envelopes
|
|
11
|
+
// and earned two spoken answers, the second one replying to a fragment (live complaint,
|
|
12
|
+
// 22.07). Recognition still runs per segment and in the background, so composing costs
|
|
13
|
+
// only the silence the owner was taking anyway.
|
|
14
|
+
//
|
|
7
15
|
// Turn-taking follows the established voice-assistant pattern recorded in Описание
|
|
8
16
|
// voicetalk: the address (wake word) only INITIATES a conversation; after an agent has
|
|
9
17
|
// answered, a short open-mic window lets the owner continue without naming the peer
|
|
@@ -39,6 +47,7 @@ import {
|
|
|
39
47
|
buildAddressBook,
|
|
40
48
|
isLikelyHallucination,
|
|
41
49
|
matchWake,
|
|
50
|
+
objectionToSwitch,
|
|
42
51
|
readFleetPeers,
|
|
43
52
|
sttPrompt,
|
|
44
53
|
type Address,
|
|
@@ -57,6 +66,24 @@ const FOLLOWUP_MS = num(process.env.VOICETALK_FOLLOWUP_MS, 9_000)
|
|
|
57
66
|
* 500–800 ms for free dialogue; the upper half of the band suits a heavier recognizer —
|
|
58
67
|
* it fragments one thought into two transcriptions far less often. */
|
|
59
68
|
const HANGOVER_MS = num(process.env.VOICETALK_HANGOVER_MS, 700)
|
|
69
|
+
/** Silence (counted from the owner's last WORD) after which a turn is considered
|
|
70
|
+
* finished and sent. The two errors are not symmetric: waiting one second too long costs
|
|
71
|
+
* one second on a chain where synthesis alone is ~5 s, while not waiting costs a split
|
|
72
|
+
* thought, two envelopes and a spoken answer to a fragment — the owner's actual
|
|
73
|
+
* complaint. Measured thinking pauses in real Russian speech run to ~2.7 s, so the
|
|
74
|
+
* budget sits above that with room. Tunable live: VOICETALK_COMPOSE_MS. */
|
|
75
|
+
const COMPOSE_MS = num(process.env.VOICETALK_COMPOSE_MS, 3_000)
|
|
76
|
+
/** A fragment that ENDS mid-thought (no terminal punctuation, or a trailing comma or
|
|
77
|
+
* conjunction) buys extra patience — the speaker audibly has not finished. */
|
|
78
|
+
const COMPOSE_UNFINISHED_MS = num(process.env.VOICETALK_COMPOSE_UNFINISHED_MS, 4_500)
|
|
79
|
+
/** Segments waiting for recognition. Deep enough for a long sentence spoken over a
|
|
80
|
+
* slow recognizer, shallow enough that a wedged STT cannot grow it without bound. */
|
|
81
|
+
const SEGMENT_QUEUE_MAX = 8
|
|
82
|
+
/** A turn never grows past this — a monologue still reaches its addressee. */
|
|
83
|
+
const TURN_MAX_MS = num(process.env.VOICETALK_TURN_MAX_MS, 45_000)
|
|
84
|
+
/** How often to re-check a turn that is ready to send but whose speaker has not gone
|
|
85
|
+
* quiet yet. */
|
|
86
|
+
const HOLD_RECHECK_MS = num(process.env.VOICETALK_HOLD_RECHECK_MS, 300)
|
|
60
87
|
/** Gate the mic for a moment after the speaker stops — room echo tail. */
|
|
61
88
|
const ECHO_TAIL_MS = num(process.env.VOICETALK_ECHO_TAIL_MS, 400)
|
|
62
89
|
/** Long answers are truncated for speech: a spoken monologue is worse than a summary. */
|
|
@@ -65,6 +92,12 @@ const TTS_MAX_CHARS = num(process.env.VOICETALK_TTS_MAX_CHARS, 1_200)
|
|
|
65
92
|
const STT_LANG = process.env.VOICETALK_STT_LANG ?? 'ru'
|
|
66
93
|
/** Force a local recognizer (mlx-whisper) to keep always-on listening on-device. */
|
|
67
94
|
const STT_ENGINE = process.env.VOICETALK_STT_ENGINE?.trim() || undefined
|
|
95
|
+
/** Test seams. VOICETALK_MIC_FILE replays a recording instead of opening the microphone;
|
|
96
|
+
* VOICETALK_PLAYBACK=0 keeps the speaker silent (synthesis still runs and is still
|
|
97
|
+
* timed). Together they let a turn-taking change be verified end to end on a host whose
|
|
98
|
+
* real voice channel is in use by its owner. */
|
|
99
|
+
const MIC_FILE = process.env.VOICETALK_MIC_FILE?.trim() || undefined
|
|
100
|
+
const PLAYBACK_ENABLED = process.env.VOICETALK_PLAYBACK !== '0'
|
|
68
101
|
|
|
69
102
|
export type DeliverFn = (target: string, text: string) => Promise<{ ok: boolean; detail?: string }>
|
|
70
103
|
export type LogFn = (evt: string, fields: Record<string, unknown>) => void
|
|
@@ -80,9 +113,10 @@ export type VoiceSessionOptions = {
|
|
|
80
113
|
}
|
|
81
114
|
|
|
82
115
|
/** State sounds, per the earcon convention in the research: `open` = the microphone is
|
|
83
|
-
* yours (rising, ≤300 ms), `end` =
|
|
84
|
-
* ≤200 ms), `
|
|
85
|
-
|
|
116
|
+
* yours (rising, ≤300 ms), `end` = turn captured, processing (short descending,
|
|
117
|
+
* ≤200 ms), `close` = the open-mic window expired, a name is needed again (low, soft),
|
|
118
|
+
* `error` = nothing routed (double descending). Processing itself is silent. */
|
|
119
|
+
type Earcons = { open: string; end: string; error: string; close: string }
|
|
86
120
|
|
|
87
121
|
export class VoiceSession {
|
|
88
122
|
private mic: Microphone | null = null
|
|
@@ -95,9 +129,18 @@ export class VoiceSession {
|
|
|
95
129
|
private prompt = ''
|
|
96
130
|
private earcons: Earcons | null = null
|
|
97
131
|
private busy = false
|
|
132
|
+
private queue: Array<Segment & { speechEndedAt: number }> = []
|
|
98
133
|
private muteUntil = 0
|
|
134
|
+
/** When the microphone last stopped hearing speech — the clock the compose window runs
|
|
135
|
+
* on, so the delivery delay the owner feels is exactly the silence they took. */
|
|
136
|
+
private lastSpeechEndedAt = 0
|
|
99
137
|
private followTarget: string | null = null
|
|
100
138
|
private followUntil = 0
|
|
139
|
+
private followTimer: ReturnType<typeof setTimeout> | null = null
|
|
140
|
+
/** The thought being assembled right now — segments of one turn, not yet sent. */
|
|
141
|
+
private pending: { target: string; parts: string[]; openedAt: number; timer: ReturnType<typeof setTimeout> | null } | null = null
|
|
142
|
+
/** When the current turn was handed to the fleet — the clock for «ответ за N с». */
|
|
143
|
+
private deliveredAt = new Map<string, number>()
|
|
101
144
|
private stopped = false
|
|
102
145
|
|
|
103
146
|
constructor(private readonly opts: VoiceSessionOptions) {}
|
|
@@ -109,9 +152,12 @@ export class VoiceSession {
|
|
|
109
152
|
mkdirSync(this.opts.tmpDir, { recursive: true, mode: 0o700 })
|
|
110
153
|
this.refreshAddressBook()
|
|
111
154
|
this.earcons = this.writeEarcons()
|
|
112
|
-
const device =
|
|
155
|
+
const device = MIC_FILE
|
|
156
|
+
? { index: -1, name: `replay:${MIC_FILE}` }
|
|
157
|
+
: await resolveInputDevice(this.opts.binding.device)
|
|
113
158
|
this.mic = new Microphone({
|
|
114
159
|
deviceIndex: device.index,
|
|
160
|
+
filePath: MIC_FILE,
|
|
115
161
|
onData: chunk => this.onAudio(chunk),
|
|
116
162
|
onStart: attempt => this.opts.log('mic.start', { device: device.index, name: device.name, attempt }),
|
|
117
163
|
onError: detail => this.opts.log('mic.error', { detail }),
|
|
@@ -122,6 +168,9 @@ export class VoiceSession {
|
|
|
122
168
|
|
|
123
169
|
stop(): void {
|
|
124
170
|
this.stopped = true
|
|
171
|
+
this.clearWindow()
|
|
172
|
+
if (this.pending?.timer) clearTimeout(this.pending.timer)
|
|
173
|
+
this.pending = null
|
|
125
174
|
this.mic?.stop()
|
|
126
175
|
this.mic = null
|
|
127
176
|
}
|
|
@@ -140,6 +189,7 @@ export class VoiceSession {
|
|
|
140
189
|
open: join(this.opts.tmpDir, 'earcon-open.wav'),
|
|
141
190
|
end: join(this.opts.tmpDir, 'earcon-end.wav'),
|
|
142
191
|
error: join(this.opts.tmpDir, 'earcon-error.wav'),
|
|
192
|
+
close: join(this.opts.tmpDir, 'earcon-close.wav'),
|
|
143
193
|
}
|
|
144
194
|
// A5→C6→E6 rising, 240 ms — the microphone is open.
|
|
145
195
|
writeFileSync(files.open, earconWav([{ hz: 880, ms: 80 }, { hz: 1047, ms: 80 }, { hz: 1319, ms: 80 }]), { mode: 0o600 })
|
|
@@ -147,11 +197,15 @@ export class VoiceSession {
|
|
|
147
197
|
writeFileSync(files.end, earconWav([{ hz: 880, ms: 80 }, { hz: 660, ms: 80 }]), { mode: 0o600 })
|
|
148
198
|
// Double descending, 260 ms — tonally "closed": nothing was routed.
|
|
149
199
|
writeFileSync(files.error, earconWav([{ hz: 520, ms: 120 }, { hz: 330, ms: 140 }]), { mode: 0o600 })
|
|
200
|
+
// One low soft tone, 180 ms — the open-mic window just expired; say a name to speak
|
|
201
|
+
// again. Deliberately quieter and lower than `end` so it reads as "door closing",
|
|
202
|
+
// not as another acknowledgement.
|
|
203
|
+
writeFileSync(files.close, earconWav([{ hz: 400, ms: 180 }], 0.14), { mode: 0o600 })
|
|
150
204
|
return files
|
|
151
205
|
}
|
|
152
206
|
|
|
153
207
|
private async earcon(kind: keyof Earcons): Promise<void> {
|
|
154
|
-
if (!this.earcons || process.env.VOICETALK_EARCONS === '0') return
|
|
208
|
+
if (!this.earcons || process.env.VOICETALK_EARCONS === '0' || !PLAYBACK_ENABLED) return
|
|
155
209
|
try {
|
|
156
210
|
this.muteUntil = Math.max(this.muteUntil, Date.now() + 600)
|
|
157
211
|
await playFile(this.earcons[kind], { timeoutMs: 5_000 })
|
|
@@ -173,16 +227,35 @@ export class VoiceSession {
|
|
|
173
227
|
return
|
|
174
228
|
}
|
|
175
229
|
for (const segment of this.segmenter.push(chunk)) {
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
230
|
+
// Recognition is serialized (one STT at a time — a pile-up only adds latency), but
|
|
231
|
+
// segments QUEUE rather than being dropped: a dropped segment is a hole in the
|
|
232
|
+
// middle of the owner's sentence, which is exactly what this phase is fixing.
|
|
233
|
+
if (this.queue.length >= SEGMENT_QUEUE_MAX) {
|
|
234
|
+
this.opts.log('segment.drop', { reason: 'queue full', ms: segment.ms })
|
|
180
235
|
continue
|
|
181
236
|
}
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
237
|
+
// The endpointer hands over a segment only after the hangover silence, so speech
|
|
238
|
+
// itself stopped that much earlier. The compose budget is counted from THAT
|
|
239
|
+
// instant: the owner's pause should not be charged twice, once as endpointing and
|
|
240
|
+
// again as composing.
|
|
241
|
+
this.queue.push({ ...segment, speechEndedAt: Date.now() - HANGOVER_MS })
|
|
242
|
+
void this.drainQueue()
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
private async drainQueue(): Promise<void> {
|
|
247
|
+
if (this.busy) return
|
|
248
|
+
this.busy = true
|
|
249
|
+
try {
|
|
250
|
+
while (this.queue.length) {
|
|
251
|
+
const segment = this.queue.shift()!
|
|
252
|
+
this.lastSpeechEndedAt = segment.speechEndedAt
|
|
253
|
+
await this.handleSegment(segment)
|
|
254
|
+
}
|
|
255
|
+
} finally {
|
|
256
|
+
this.busy = false
|
|
257
|
+
// A turn may have been waiting for recognition to finish before it could be sent.
|
|
258
|
+
if (this.pending) this.rearm()
|
|
186
259
|
}
|
|
187
260
|
}
|
|
188
261
|
|
|
@@ -222,50 +295,163 @@ export class VoiceSession {
|
|
|
222
295
|
text: transcript,
|
|
223
296
|
})
|
|
224
297
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
298
|
+
this.absorb(transcript)
|
|
299
|
+
} finally {
|
|
300
|
+
rmSync(wav, { force: true })
|
|
301
|
+
}
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
/** Route one recognized fragment into the turn being composed.
|
|
305
|
+
*
|
|
306
|
+
* The decision is deliberately conservative in one direction: an unaddressed fragment
|
|
307
|
+
* arriving right after an addressed one CONTINUES that thought rather than being
|
|
308
|
+
* dropped, and a wake word for SOMEONE ELSE has to clear a higher bar than it does
|
|
309
|
+
* when no conversation is open. Mid-dialogue, the cheap error (one more line to the
|
|
310
|
+
* peer already listening) is far cheaper than the expensive one (the conversation
|
|
311
|
+
* silently handed to another agent). */
|
|
312
|
+
private absorb(transcript: string): void {
|
|
313
|
+
const active = this.pending?.target ?? (this.followTarget && Date.now() < this.followUntil ? this.followTarget : null)
|
|
314
|
+
const match = matchWake(transcript, this.book)
|
|
315
|
+
const switching = !!match && !!active && match.target !== active
|
|
316
|
+
|
|
317
|
+
if (switching) {
|
|
318
|
+
const decline = objectionToSwitch(match!)
|
|
319
|
+
if (decline) {
|
|
320
|
+
this.opts.log('wake.switch-declined', { from: active, to: match!.target, reason: decline, text: transcript })
|
|
321
|
+
this.append(active!, transcript.trim())
|
|
247
322
|
return
|
|
248
323
|
}
|
|
324
|
+
// A deliberate switch: the thought in flight belongs to the previous peer, so it
|
|
325
|
+
// goes out before the new one starts.
|
|
326
|
+
this.opts.log('wake.switch', { from: active, to: match!.target })
|
|
327
|
+
this.flush('switch')
|
|
328
|
+
this.append(match!.target, match!.body)
|
|
329
|
+
return
|
|
330
|
+
}
|
|
249
331
|
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
const verdict = await this.opts.deliver(target!, body)
|
|
259
|
-
if (verdict.ok) {
|
|
260
|
-
this.opts.log('voice.delivered', { target, chars: body.length, totalMs: Date.now() - started })
|
|
261
|
-
} else {
|
|
262
|
-
this.opts.log('voice.undelivered', { target, detail: verdict.detail })
|
|
263
|
-
await this.earcon('error')
|
|
264
|
-
await this.say(`Не доставлено: ${verdict.detail ?? 'неизвестная ошибка'}`)
|
|
332
|
+
if (match) {
|
|
333
|
+
if (!match.body) {
|
|
334
|
+
// Addressed with nothing to say yet («Борис?») — acknowledge and hold the mic
|
|
335
|
+
// open for the actual request, exactly as an assistant beeps and waits.
|
|
336
|
+
this.openWindow(match.target)
|
|
337
|
+
this.opts.log('wake.await-body', { target: match.target, alias: match.alias })
|
|
338
|
+
void this.earcon('open')
|
|
339
|
+
return
|
|
265
340
|
}
|
|
266
|
-
|
|
267
|
-
|
|
341
|
+
this.append(match.target, match.body)
|
|
342
|
+
return
|
|
268
343
|
}
|
|
344
|
+
|
|
345
|
+
if (active) {
|
|
346
|
+
this.opts.log('turn.continue', { target: active, reason: this.pending ? 'compose' : 'follow-up' })
|
|
347
|
+
this.append(active, transcript.trim())
|
|
348
|
+
return
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
this.opts.log('wake.no-match', { text: transcript })
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
/** Add a fragment to the turn in flight and (re)arm the compose timer. */
|
|
355
|
+
private append(target: string, text: string): void {
|
|
356
|
+
const body = text.trim()
|
|
357
|
+
if (!body) return
|
|
358
|
+
if (this.pending && this.pending.target !== target) this.flush('retarget')
|
|
359
|
+
if (!this.pending) {
|
|
360
|
+
this.pending = { target, parts: [], openedAt: Date.now(), timer: null }
|
|
361
|
+
this.opts.log('turn.open', { target })
|
|
362
|
+
}
|
|
363
|
+
this.pending.parts.push(body)
|
|
364
|
+
this.clearWindow()
|
|
365
|
+
this.rearm()
|
|
366
|
+
}
|
|
367
|
+
|
|
368
|
+
/** (Re)arm the compose timer for the turn in flight. */
|
|
369
|
+
private rearm(delayMs?: number): void {
|
|
370
|
+
const turn = this.pending
|
|
371
|
+
if (!turn) return
|
|
372
|
+
if (turn.timer) clearTimeout(turn.timer)
|
|
373
|
+
if (Date.now() - turn.openedAt >= TURN_MAX_MS) {
|
|
374
|
+
this.flush('max-length')
|
|
375
|
+
return
|
|
376
|
+
}
|
|
377
|
+
const budget = looksUnfinished(turn.parts[turn.parts.length - 1] ?? '') ? COMPOSE_UNFINISHED_MS : COMPOSE_MS
|
|
378
|
+
const remaining = this.lastSpeechEndedAt + budget - Date.now()
|
|
379
|
+
const delay = Math.max(0, delayMs ?? remaining)
|
|
380
|
+
turn.timer = setTimeout(() => this.flush('compose'), delay)
|
|
381
|
+
}
|
|
382
|
+
|
|
383
|
+
/** Send the composed thought as ONE envelope. */
|
|
384
|
+
private flush(reason: string): void {
|
|
385
|
+
const turn = this.pending
|
|
386
|
+
if (!turn) return
|
|
387
|
+
// A thought is NOT finished while the owner is still speaking it. The compose timer
|
|
388
|
+
// counts silence since the last RECOGNIZED fragment, and recognition lags the voice
|
|
389
|
+
// by an endpointing pause plus the recognizer's own time — so by the time it fires,
|
|
390
|
+
// the owner may already be several words into the rest of the sentence. Waiting for
|
|
391
|
+
// the microphone and the recognizer to go quiet is what turns those words into the
|
|
392
|
+
// same envelope instead of a second one. (Live defect, 22.07: two envelopes, two
|
|
393
|
+
// spoken answers, the second one replying to a fragment.)
|
|
394
|
+
if (reason === 'compose' && (this.segmenter.speaking || this.busy || this.queue.length)) {
|
|
395
|
+
this.opts.log('turn.hold', {
|
|
396
|
+
target: turn.target,
|
|
397
|
+
reason: this.segmenter.speaking ? 'still speaking' : 'recognition in flight',
|
|
398
|
+
})
|
|
399
|
+
this.rearm(HOLD_RECHECK_MS)
|
|
400
|
+
return
|
|
401
|
+
}
|
|
402
|
+
this.pending = null
|
|
403
|
+
if (turn.timer) clearTimeout(turn.timer)
|
|
404
|
+
const text = joinTurn(turn.parts)
|
|
405
|
+
if (!text) return
|
|
406
|
+
const startedAt = Date.now()
|
|
407
|
+
this.opts.log('turn.flush', {
|
|
408
|
+
target: turn.target,
|
|
409
|
+
reason,
|
|
410
|
+
segments: turn.parts.length,
|
|
411
|
+
chars: text.length,
|
|
412
|
+
// The headline latency number of this phase: from the owner's last word to the
|
|
413
|
+
// moment the fleet is handed the thought.
|
|
414
|
+
sinceSpeechEndMs: this.lastSpeechEndedAt ? startedAt - this.lastSpeechEndedAt : null,
|
|
415
|
+
})
|
|
416
|
+
void this.deliverTurn(turn.target, text, startedAt)
|
|
417
|
+
}
|
|
418
|
+
|
|
419
|
+
private async deliverTurn(target: string, text: string, startedAt: number): Promise<void> {
|
|
420
|
+
// The turn is committed → the "heard you, working on it" sound. ONE per thought, not
|
|
421
|
+
// one per segment. Delivery itself stays silent (processing earns no earcon) — the
|
|
422
|
+
// agent's own voice is the next thing the owner hears.
|
|
423
|
+
await this.earcon('end')
|
|
424
|
+
this.openWindow(target)
|
|
425
|
+
const verdict = await this.opts.deliver(target, text)
|
|
426
|
+
if (verdict.ok) {
|
|
427
|
+
this.deliveredAt.set(target, Date.now())
|
|
428
|
+
this.opts.log('voice.delivered', { target, chars: text.length, deliverMs: Date.now() - startedAt })
|
|
429
|
+
} else {
|
|
430
|
+
this.opts.log('voice.undelivered', { target, detail: verdict.detail })
|
|
431
|
+
await this.earcon('error')
|
|
432
|
+
await this.say(`Не доставлено: ${verdict.detail ?? 'неизвестная ошибка'}`)
|
|
433
|
+
}
|
|
434
|
+
}
|
|
435
|
+
|
|
436
|
+
/** Open (or extend) the follow-up window, and arm the sound that tells the owner when
|
|
437
|
+
* it has closed. An invisible window is worse than a short one: the owner kept talking
|
|
438
|
+
* into a closed mic with no idea a name was needed again (live, 22.07). */
|
|
439
|
+
private openWindow(target: string): void {
|
|
440
|
+
this.followTarget = target
|
|
441
|
+
this.followUntil = Date.now() + FOLLOWUP_MS
|
|
442
|
+
if (this.followTimer) clearTimeout(this.followTimer)
|
|
443
|
+
this.followTimer = setTimeout(() => {
|
|
444
|
+
this.followTimer = null
|
|
445
|
+
if (this.pending) return // still composing — the window is not idle
|
|
446
|
+
this.opts.log('followup.closed', { target })
|
|
447
|
+
this.followTarget = null
|
|
448
|
+
void this.earcon('close')
|
|
449
|
+
}, FOLLOWUP_MS)
|
|
450
|
+
}
|
|
451
|
+
|
|
452
|
+
private clearWindow(): void {
|
|
453
|
+
if (this.followTimer) clearTimeout(this.followTimer)
|
|
454
|
+
this.followTimer = null
|
|
269
455
|
}
|
|
270
456
|
|
|
271
457
|
/** Speak an agent's answer on the owner's speaker, then hold the follow-up window
|
|
@@ -276,16 +462,21 @@ export class VoiceSession {
|
|
|
276
462
|
this.opts.log('tts.skip', { from, reason: 'nothing speakable' })
|
|
277
463
|
return
|
|
278
464
|
}
|
|
465
|
+
// How long the owner waited between finishing their thought and hearing an answer —
|
|
466
|
+
// the number Фаза 4 is judged by. It splits into the agent's thinking time and the
|
|
467
|
+
// synthesis time (logged separately by `tts.spoken`).
|
|
468
|
+
const askedAt = this.deliveredAt.get(from)
|
|
469
|
+
if (askedAt) this.opts.log('turn.answering', { from, sinceDeliveryMs: Date.now() - askedAt })
|
|
279
470
|
// Voice by IDENTITY, not by voice name: voice-connect resolves the sender's voice
|
|
280
471
|
// after picking the engine, so the owner hears WHO is talking even when the primary
|
|
281
472
|
// engine is down and a fallback rung serves. Sending a voice name instead would only
|
|
282
473
|
// bind to the primary engine — the very defect seen live on 22.07 (boris answered in
|
|
283
474
|
// a stranger's voice once gemini was unavailable).
|
|
284
475
|
await this.say(text, { from })
|
|
285
|
-
this.followTarget = from
|
|
286
|
-
this.followUntil = Date.now() + FOLLOWUP_MS
|
|
287
476
|
// The turn passes back to the human: the mic is open for a follow-up without a
|
|
288
|
-
// wake word,
|
|
477
|
+
// wake word, the rising earcon says so, and the window's own timer will say when it
|
|
478
|
+
// closes again.
|
|
479
|
+
this.openWindow(from)
|
|
289
480
|
await this.earcon('open')
|
|
290
481
|
}
|
|
291
482
|
|
|
@@ -299,7 +490,8 @@ export class VoiceSession {
|
|
|
299
490
|
this.muteUntil = Date.now() + 3_600_000
|
|
300
491
|
this.segmenter.reset()
|
|
301
492
|
try {
|
|
302
|
-
await playFile(result.path)
|
|
493
|
+
if (PLAYBACK_ENABLED) await playFile(result.path)
|
|
494
|
+
else this.opts.log('tts.silent', { from: opts.from, chars: text.length, path: result.path })
|
|
303
495
|
} finally {
|
|
304
496
|
this.muteUntil = Date.now() + ECHO_TAIL_MS
|
|
305
497
|
this.segmenter.reset()
|
|
@@ -324,6 +516,30 @@ export class VoiceSession {
|
|
|
324
516
|
}
|
|
325
517
|
}
|
|
326
518
|
|
|
519
|
+
/** Does this fragment sound like the middle of a sentence? Recognition punctuates what
|
|
520
|
+
* it hears, so a fragment left hanging on a comma or a conjunction — or with no terminal
|
|
521
|
+
* mark at all — is a thought still in progress, and the turn waits longer for the rest. */
|
|
522
|
+
export function looksUnfinished(text: string): boolean {
|
|
523
|
+
const t = text.trim()
|
|
524
|
+
if (!t) return false
|
|
525
|
+
if (/[,;:—–-]$/.test(t)) return true
|
|
526
|
+
if (!/[.!?…]$/.test(t)) return true
|
|
527
|
+
const last = t.replace(/[.!?…]+$/, '').split(/\s+/).pop()?.toLowerCase() ?? ''
|
|
528
|
+
return ['и', 'а', 'но', 'что', 'чтобы', 'или', 'если', 'когда', 'потому', 'то'].includes(last)
|
|
529
|
+
}
|
|
530
|
+
|
|
531
|
+
/** Join the fragments of one thought back into a sentence. Recognition returns each
|
|
532
|
+
* segment already punctuated and capitalized, so a plain space is the honest join —
|
|
533
|
+
* the parts were one utterance and are re-read as one. */
|
|
534
|
+
export function joinTurn(parts: readonly string[]): string {
|
|
535
|
+
return parts
|
|
536
|
+
.map(p => p.trim())
|
|
537
|
+
.filter(Boolean)
|
|
538
|
+
.join(' ')
|
|
539
|
+
.replace(/\s+/g, ' ')
|
|
540
|
+
.trim()
|
|
541
|
+
}
|
|
542
|
+
|
|
327
543
|
/** Prepare an agent's text for the speaker: strip what only makes sense on a screen and
|
|
328
544
|
* cap the length — a spoken wall of text is unusable. The modality frame already asks
|
|
329
545
|
* agents for short plain answers; this is the floor for when one forgets. */
|
package/src/wake.ts
CHANGED
|
@@ -205,6 +205,10 @@ export type WakeMatch = {
|
|
|
205
205
|
/** The utterance with the address (and any leading filler) stripped. */
|
|
206
206
|
body: string
|
|
207
207
|
alias: string
|
|
208
|
+
/** True when every word of the address matched letter for letter. A match that only
|
|
209
|
+
* survived the one-edit tolerance is good enough to START a conversation, but not to
|
|
210
|
+
* TAKE one away from the peer already in it — see the switch bar in voiceSession. */
|
|
211
|
+
exact: boolean
|
|
208
212
|
}
|
|
209
213
|
|
|
210
214
|
/** Does this transcript open by addressing a fleet peer? Longest alias wins, so
|
|
@@ -229,6 +233,7 @@ export function matchWake(transcript: string, book: Address[]): WakeMatch | null
|
|
|
229
233
|
if (tokens.length < cand.words.length) continue
|
|
230
234
|
const ok = cand.words.every((w, i) => tokenMatches(tokens[i], w))
|
|
231
235
|
if (!ok) continue
|
|
236
|
+
const exact = cand.words.every((w, i) => tokens[i] === w)
|
|
232
237
|
// Rebuild the body from the ORIGINAL transcript so punctuation and casing of the
|
|
233
238
|
// message survive — only the address is stripped.
|
|
234
239
|
const body = rawTokens
|
|
@@ -236,11 +241,31 @@ export function matchWake(transcript: string, book: Address[]): WakeMatch | null
|
|
|
236
241
|
.join(' ')
|
|
237
242
|
.replace(/^[\s,.!:;—–-]+/, '')
|
|
238
243
|
.trim()
|
|
239
|
-
return { target: cand.target, alias: cand.alias, body }
|
|
244
|
+
return { target: cand.target, alias: cand.alias, body, exact }
|
|
240
245
|
}
|
|
241
246
|
return null
|
|
242
247
|
}
|
|
243
248
|
|
|
249
|
+
/** May this match TAKE an open conversation away from the peer currently in it?
|
|
250
|
+
*
|
|
251
|
+
* Mid-dialogue the two errors are wildly asymmetric: one more line delivered to the peer
|
|
252
|
+
* already listening is nearly free, while a mis-heard name silently hands the whole
|
|
253
|
+
* conversation to another agent — the owner keeps talking to someone who is no longer
|
|
254
|
+
* there. So a switch must look deliberate: the name heard letter for letter, followed by
|
|
255
|
+
* a real request. Returns the objection, or null when the switch is allowed.
|
|
256
|
+
*
|
|
257
|
+
* Every rule here is a live artefact, not a hypothesis: «да, различаю» came back from
|
|
258
|
+
* recognition as «Дарвин, дарвин» and moved the conversation off boris (22.07). */
|
|
259
|
+
export function objectionToSwitch(match: WakeMatch): string | null {
|
|
260
|
+
if (!match.exact) return 'inexact name match'
|
|
261
|
+
const body = normalizeSpoken(match.body)
|
|
262
|
+
if (!body) return 'name with no request'
|
|
263
|
+
const words = body.split(' ')
|
|
264
|
+
if (words.every(w => w === match.alias)) return 'body echoes the name'
|
|
265
|
+
if (words.length < 2) return 'request too short to be an address'
|
|
266
|
+
return null
|
|
267
|
+
}
|
|
268
|
+
|
|
244
269
|
/** Whisper-family recognizers emit stock phrases on near-silence («Продолжение
|
|
245
270
|
* следует…», «Субтитры сделал…»). They are not speech; drop them before matching so a
|
|
246
271
|
* quiet room cannot fabricate an utterance. */
|