@agfpd/voicetalk-runtime 0.3.3 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/docs/README.md +31 -0
- package/docs/ru/README.md +31 -0
- package/package.json +1 -1
- package/src/audio.ts +36 -2
- package/src/cli.ts +20 -3
- package/src/nativeVoice.ts +119 -0
- package/src/voiceSession.ts +203 -4
- package/src/wake.ts +4 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.4.1] - 2026-10-26
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- `native.ready {target, state}` on start when the `iapeer voice status` preflight
|
|
15
|
+
answers and the target stays in native mode — a positive receipt, so a healthy start
|
|
16
|
+
is distinguishable from one that never ran the preflight.
|
|
17
|
+
|
|
18
|
+
## [0.4.0] - 2026-10-26
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- Native voice hand-off (pilot, `interfaces.voicetalk.native_voice` or
|
|
23
|
+
`VOICETALK_NATIVE_VOICE`): an accepted spoken wake for a listed target releases the
|
|
24
|
+
microphone (capture exit confirmed) and asks iapeer to open the peer's native Codex
|
|
25
|
+
`/voice` (`iapeer voice open|status|close <peer> --json`). The microphone returns to
|
|
26
|
+
wake mode only on a confirmed `closed`; failed/timed-out opens and unreadable receipts
|
|
27
|
+
keep it released. One wake → one open; outbound speech and earcons are held while the
|
|
28
|
+
peer owns the room; a restart adopts a still-open voice. Start-up preflight: if
|
|
29
|
+
`iapeer voice status` gives no readable answer, the target degrades to the STT path
|
|
30
|
+
(`native.unavailable`) instead of a listener that could never get its mic back.
|
|
31
|
+
- `interface --native-voice <peers>`; `Microphone.pause()/resume()` (replay skips the
|
|
32
|
+
handed-off span); `scripts/replay-dialogue.sh` gains a fake `iapeer voice` and a
|
|
33
|
+
`native-voice.pcm` fixture.
|
|
34
|
+
|
|
10
35
|
## [0.3.3] - 2026-10-18
|
|
11
36
|
|
|
12
37
|
### Fixed
|
package/docs/README.md
CHANGED
|
@@ -46,6 +46,37 @@ decoder prompt, because prompt text can be generated back on unrelated audio. A
|
|
|
46
46
|
dialogue opens only on an exact leading address in the unprimed transcript; repeated
|
|
47
47
|
token loops and name-only echoes are rejected before an IAP envelope exists.
|
|
48
48
|
|
|
49
|
+
## Native voice — `native_voice` (pilot)
|
|
50
|
+
|
|
51
|
+
```jsonc
|
|
52
|
+
{ "native_voice": ["darwin"] } // or env VOICETALK_NATIVE_VOICE=darwin ("" disables)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
For a target listed here an accepted wake word does NOT start STT → envelope → TTS. It
|
|
56
|
+
hands the conversation to the peer's own realtime voice (Codex CLI `/voice`) inside its
|
|
57
|
+
live session, which iapeer opens and closes:
|
|
58
|
+
|
|
59
|
+
1. The wake word must pass every fail-closed check above — native voice changes only
|
|
60
|
+
what an accepted activation does, never what counts as one.
|
|
61
|
+
2. voicetalk kills its capture process and waits until it has actually exited (no
|
|
62
|
+
CoreAudio client left), plays the short "heard you" earcon while nobody is
|
|
63
|
+
listening, then runs `iapeer voice open <peer> --json`.
|
|
64
|
+
3. While the peer holds the room voicetalk stays silent: no earcons, and any agent
|
|
65
|
+
answer that arrives is spoken only after the hand-off ends.
|
|
66
|
+
4. voicetalk polls `iapeer voice status <peer> --json` (1 s) and takes the microphone
|
|
67
|
+
back ONLY on `state: closed`. A failed or timed-out open, an unreadable answer, and
|
|
68
|
+
`opening`/`closing` all keep the microphone released — a timeout is not evidence
|
|
69
|
+
that the peer stopped listening. Idle-close is iapeer's; after a 30-minute ceiling
|
|
70
|
+
(`VOICETALK_NATIVE_MAX_MS`) voicetalk asks `iapeer voice close`, and still waits for
|
|
71
|
+
the confirmed `closed`.
|
|
72
|
+
5. Back in wake mode a soft "door closed" tone plays; the next turn needs a name again.
|
|
73
|
+
A runtime restart adopts a voice that is still open instead of grabbing the mic.
|
|
74
|
+
If at start iapeer does not answer `voice status` at all, the target falls back to
|
|
75
|
+
the STT path (`native.unavailable` in the log).
|
|
76
|
+
|
|
77
|
+
The words spoken after the name in the activating phrase are not forwarded; the
|
|
78
|
+
conversation itself happens in the peer's native voice. Other targets keep the STT path.
|
|
79
|
+
|
|
49
80
|
## How a turn works
|
|
50
81
|
|
|
51
82
|
1. The microphone runs always-on; energy endpointing cuts an utterance when the
|
package/docs/ru/README.md
CHANGED
|
@@ -46,6 +46,37 @@ prompt: такой контекст способен сам породить и
|
|
|
46
46
|
открывает только точное имя в начале непримированного транскрипта; циклы повторяющихся
|
|
47
47
|
токенов и тело из одних настроенных имён отбрасываются до появления IAP-конверта.
|
|
48
48
|
|
|
49
|
+
## Нативный голос — `native_voice` (пилот)
|
|
50
|
+
|
|
51
|
+
```jsonc
|
|
52
|
+
{ "native_voice": ["darwin"] } // или env VOICETALK_NATIVE_VOICE=darwin ("" выключает)
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Для перечисленного адресата принятое wake-слово НЕ запускает STT → конверт → TTS, а
|
|
56
|
+
передаёт разговор в собственный realtime-голос пира (Codex CLI `/voice`) в его живой
|
|
57
|
+
сессии; открывает и закрывает голос iapeer.
|
|
58
|
+
|
|
59
|
+
1. Wake-слово проходит все fail-closed проверки выше: режим меняет только то, что делает
|
|
60
|
+
принятая активация, но не то, что ею считается.
|
|
61
|
+
2. voicetalk гасит процесс захвата и ждёт его фактического выхода (CoreAudio-клиента
|
|
62
|
+
нет), играет короткий earcon «услышал», пока никто не слушает, и вызывает
|
|
63
|
+
`iapeer voice open <peer> --json`.
|
|
64
|
+
3. Пока комната у пира, voicetalk молчит: ни earcon'ов, а пришедший ответ агента
|
|
65
|
+
озвучивается только после возврата микрофона.
|
|
66
|
+
4. voicetalk опрашивает `iapeer voice status <peer> --json` (1 с) и забирает микрофон
|
|
67
|
+
ТОЛЬКО по `state: closed`. Неудачный или просроченный open, нечитаемый ответ,
|
|
68
|
+
`opening`/`closing` — микрофон остаётся отпущенным: таймаут не доказывает, что пир
|
|
69
|
+
перестал слушать. Закрытие по простою — у iapeer; по потолку 30 минут
|
|
70
|
+
(`VOICETALK_NATIVE_MAX_MS`) voicetalk просит `iapeer voice close` и всё равно ждёт
|
|
71
|
+
подтверждённого `closed`.
|
|
72
|
+
5. По возврату в wake-режим звучит мягкий тон «дверь закрыта»; следующий ход снова
|
|
73
|
+
требует имени. Рестарт рантайма принимает ещё открытый голос, а не хватает микрофон.
|
|
74
|
+
Если на старте iapeer вообще не отвечает на `voice status`, адресат уходит на прежний
|
|
75
|
+
STT-путь (`native.unavailable` в логе).
|
|
76
|
+
|
|
77
|
+
Слова после имени в активирующей фразе не пересылаются — разговор идёт уже нативным
|
|
78
|
+
голосом пира. Остальные адресаты — прежним STT-путём.
|
|
79
|
+
|
|
49
80
|
## Как устроен ход разговора
|
|
50
81
|
|
|
51
82
|
1. Микрофон слушает постоянно; энергетический эндпойнтер закрывает реплику по паузе
|
package/package.json
CHANGED
package/src/audio.ts
CHANGED
|
@@ -320,6 +320,9 @@ export class Microphone {
|
|
|
320
320
|
private stopped = false
|
|
321
321
|
private attempt = 0
|
|
322
322
|
private timer: ReturnType<typeof setTimeout> | null = null
|
|
323
|
+
/** Replay position, kept across pause/resume (test seam only). */
|
|
324
|
+
private replayOffset = 0
|
|
325
|
+
private pausedAt = 0
|
|
323
326
|
|
|
324
327
|
constructor(private readonly opts: MicOptions) {}
|
|
325
328
|
|
|
@@ -342,6 +345,37 @@ export class Microphone {
|
|
|
342
345
|
return this.child !== null
|
|
343
346
|
}
|
|
344
347
|
|
|
348
|
+
/** Release the input device and resolve only once the capture child has actually
|
|
349
|
+
* EXITED — a killed-but-not-yet-reaped ffmpeg still holds the CoreAudio client. This is
|
|
350
|
+
* the hand-off to a second listener (native voice): the other side may open the same
|
|
351
|
+
* physical microphone only after this resolves, or both would hear the room. */
|
|
352
|
+
async pause(capMs = 3_000): Promise<{ released: boolean }> {
|
|
353
|
+
const child = this.child
|
|
354
|
+
this.stop()
|
|
355
|
+
if (this.opts.filePath) this.pausedAt = Date.now()
|
|
356
|
+
if (!child || child.exitCode !== null || child.signalCode !== null) return { released: true }
|
|
357
|
+
return await new Promise(resolve => {
|
|
358
|
+
const timer = setTimeout(() => resolve({ released: false }), capMs)
|
|
359
|
+
child.once('close', () => {
|
|
360
|
+
clearTimeout(timer)
|
|
361
|
+
resolve({ released: true })
|
|
362
|
+
})
|
|
363
|
+
})
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
/** Take the device back after a pause. A replayed recording does not rewind: the audio
|
|
367
|
+
* that "played" while the device was handed off belongs to the other listener, so the
|
|
368
|
+
* replay skips ahead by the paused wall-clock time — exactly what a real microphone
|
|
369
|
+
* would have missed. */
|
|
370
|
+
resume(): void {
|
|
371
|
+
if (this.opts.filePath && this.pausedAt) {
|
|
372
|
+
const frames = Math.floor((Date.now() - this.pausedAt) / FRAME_MS)
|
|
373
|
+
this.replayOffset += frames * FRAME_BYTES
|
|
374
|
+
this.pausedAt = 0
|
|
375
|
+
}
|
|
376
|
+
this.start()
|
|
377
|
+
}
|
|
378
|
+
|
|
345
379
|
private spawnChild(): void {
|
|
346
380
|
if (this.stopped) return
|
|
347
381
|
this.attempt += 1
|
|
@@ -397,9 +431,9 @@ export class Microphone {
|
|
|
397
431
|
return
|
|
398
432
|
}
|
|
399
433
|
this.opts.onStart?.(this.attempt)
|
|
400
|
-
let offset = 0
|
|
401
434
|
const tick = () => {
|
|
402
435
|
if (this.stopped) return
|
|
436
|
+
const offset = this.replayOffset
|
|
403
437
|
if (offset >= pcm.length) {
|
|
404
438
|
// Keep the stream alive with silence so compose/window timers still fire.
|
|
405
439
|
this.opts.onData(Buffer.alloc(FRAME_BYTES))
|
|
@@ -407,7 +441,7 @@ export class Microphone {
|
|
|
407
441
|
return
|
|
408
442
|
}
|
|
409
443
|
this.opts.onData(Buffer.from(pcm.subarray(offset, offset + FRAME_BYTES)))
|
|
410
|
-
offset
|
|
444
|
+
this.replayOffset = offset + FRAME_BYTES
|
|
411
445
|
this.timer = setTimeout(tick, FRAME_MS)
|
|
412
446
|
}
|
|
413
447
|
tick()
|
package/src/cli.ts
CHANGED
|
@@ -45,6 +45,7 @@ import { selfInstall } from './selfInstall.ts'
|
|
|
45
45
|
import { parseWakeWords, runSelfConfig } from './selfConfig.ts'
|
|
46
46
|
import { writeJsonAtomic } from './fsAtomic.ts'
|
|
47
47
|
import { VoiceSession } from './voiceSession.ts'
|
|
48
|
+
import { cliNativeVoice, nativeVoiceTargets } from './nativeVoice.ts'
|
|
48
49
|
import { voiceHealth } from './voiceConnect.ts'
|
|
49
50
|
import { listInputDevices, resolveInputDevice } from './audio.ts'
|
|
50
51
|
import { buildAddressBook, unresolvedWakeWords, wakeEntries, type VoicetalkBinding } from './wake.ts'
|
|
@@ -163,7 +164,7 @@ function usage(): string {
|
|
|
163
164
|
${BIN_NAME} # self-install (npx contract): bin on PATH + manifest
|
|
164
165
|
${BIN_NAME} self-install # explicit self-install (idempotent)
|
|
165
166
|
${BIN_NAME} self-config # per-peer self-config hook (foundation-invoked)
|
|
166
|
-
${BIN_NAME} interface --wake-words «дарвин=darwin,борис=boris» [--default-target <peer>] [--device <d>]
|
|
167
|
+
${BIN_NAME} interface --wake-words «дарвин=darwin,борис=boris» [--default-target <peer>] [--device <d>] [--native-voice darwin]
|
|
167
168
|
# set this peer's voicetalk binding (wake map)
|
|
168
169
|
${BIN_NAME} run # router loop: live mic ↔ speaker (VOICETALK_AUDIO=0 → text only)
|
|
169
170
|
${BIN_NAME} say --peer <personality> <text…> # inject an utterance without the mic (debug seam)
|
|
@@ -400,6 +401,7 @@ function setVoicetalkInterface(cwd: string, patch: VoicetalkInterface): Record<s
|
|
|
400
401
|
if (patch.wake_words !== undefined) voicetalk.wake_words = patch.wake_words
|
|
401
402
|
if (patch.default_target !== undefined) voicetalk.default_target = patch.default_target
|
|
402
403
|
if (patch.device !== undefined) voicetalk.device = patch.device
|
|
404
|
+
if (patch.native_voice !== undefined) voicetalk.native_voice = patch.native_voice
|
|
403
405
|
;(interfaces as Record<string, unknown>).voicetalk = voicetalk
|
|
404
406
|
const merged = { ...raw, interfaces }
|
|
405
407
|
writeJsonAtomic(path, merged)
|
|
@@ -972,6 +974,11 @@ async function runCommand(extraArgs: string[]): Promise<void> {
|
|
|
972
974
|
deliver: (target, text) => deliverUtterance(ctx, target, text),
|
|
973
975
|
log: logEvt,
|
|
974
976
|
tmpDir: join(globalVoicetalkRoot(), 'tmp'),
|
|
977
|
+
// Native voice is opt-in per target; with no target configured the door is not
|
|
978
|
+
// even built, so the STT path is untouched for everyone.
|
|
979
|
+
...(nativeVoiceTargets(binding).size
|
|
980
|
+
? { nativeVoice: cliNativeVoice({ bin: ctx.iapBin, env: iapEnv(owner), cwd: ctx.cwd }) }
|
|
981
|
+
: {}),
|
|
975
982
|
})
|
|
976
983
|
try {
|
|
977
984
|
const started = await candidate.start()
|
|
@@ -1013,6 +1020,7 @@ async function runCommand(extraArgs: string[]): Promise<void> {
|
|
|
1013
1020
|
owner: owner.personality,
|
|
1014
1021
|
wake_words: wakeEntries(binding).map(([w]) => w),
|
|
1015
1022
|
device: binding.device ?? null,
|
|
1023
|
+
native_voice: [...nativeVoiceTargets(binding)],
|
|
1016
1024
|
iapBin: ctx.iapBin,
|
|
1017
1025
|
inbound_socket: inboundSocketPath(owner.personality),
|
|
1018
1026
|
audio: session ? { live: true, ...audio } : { live: false, ...audio },
|
|
@@ -1115,9 +1123,17 @@ async function interfaceCommand(args: string[]): Promise<void> {
|
|
|
1115
1123
|
const wakeWords = parseWakeWords(stringFlag(flags, 'wake-words'))
|
|
1116
1124
|
const defaultTarget = stringFlag(flags, 'default-target')
|
|
1117
1125
|
const device = stringFlag(flags, 'device')
|
|
1118
|
-
|
|
1126
|
+
const nativeFlag = stringFlag(flags, 'native-voice')
|
|
1127
|
+
const nativeVoice = nativeFlag === undefined ? undefined : nativeFlag.split(',').map(v => v.trim()).filter(Boolean)
|
|
1128
|
+
if (
|
|
1129
|
+
wakeWord === undefined &&
|
|
1130
|
+
wakeWords === undefined &&
|
|
1131
|
+
defaultTarget === undefined &&
|
|
1132
|
+
device === undefined &&
|
|
1133
|
+
nativeVoice === undefined
|
|
1134
|
+
) {
|
|
1119
1135
|
throw new VoicetalkRuntimeError(
|
|
1120
|
-
'interface requires --wake-words <«слово=пир,…»>, --default-target <peer>, --wake-word <w> and/or --
|
|
1136
|
+
'interface requires --wake-words <«слово=пир,…»>, --default-target <peer>, --wake-word <w>, --device <d> and/or --native-voice <peers>',
|
|
1121
1137
|
)
|
|
1122
1138
|
}
|
|
1123
1139
|
ensureCurrentProfile() // assert the profile exists before patching
|
|
@@ -1126,6 +1142,7 @@ async function interfaceCommand(args: string[]): Promise<void> {
|
|
|
1126
1142
|
...(wakeWords !== undefined ? { wake_words: wakeWords } : {}),
|
|
1127
1143
|
...(defaultTarget !== undefined ? { default_target: defaultTarget } : {}),
|
|
1128
1144
|
...(device !== undefined ? { device } : {}),
|
|
1145
|
+
...(nativeVoice !== undefined ? { native_voice: nativeVoice } : {}),
|
|
1129
1146
|
})
|
|
1130
1147
|
process.stdout.write(`${peerProfilePath(process.cwd())}\n`)
|
|
1131
1148
|
printJson(merged)
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
// nativeVoice.ts — the door to a peer's NATIVE voice mode (Codex CLI `/voice`).
|
|
2
|
+
//
|
|
3
|
+
// For a configured target the spoken wake word does not start STT → envelope → TTS; it
|
|
4
|
+
// hands the conversation to the peer's own realtime voice inside its live session. The
|
|
5
|
+
// session belongs to iapeer (the pty supervisor), so voicetalk only REQUESTS and OBSERVES
|
|
6
|
+
// over a small CLI contract agreed with the iapeer Implementer (topic codex-voice-wake):
|
|
7
|
+
//
|
|
8
|
+
// iapeer voice open <peer> --json → {"ok":true,"state":"open","session":"…","already":bool}
|
|
9
|
+
// iapeer voice status <peer> --json → {"state":"open|opening|closing|closed","session":…,"reason"?:…}
|
|
10
|
+
// iapeer voice close <peer> --json → {"ok":true,"state":"closed"}
|
|
11
|
+
//
|
|
12
|
+
// open returns only once the realtime session is confirmed started, close once it is
|
|
13
|
+
// confirmed closed; status is what the listener polls before it takes the microphone back.
|
|
14
|
+
// Idle-close lives in iapeer — voicetalk never decides a conversation is over, it only
|
|
15
|
+
// waits for the confirmation that it is.
|
|
16
|
+
|
|
17
|
+
import { spawn } from 'child_process'
|
|
18
|
+
|
|
19
|
+
export type NativeVoiceState = 'open' | 'opening' | 'closing' | 'closed' | 'unknown'
|
|
20
|
+
|
|
21
|
+
export type NativeVoiceReceipt = {
|
|
22
|
+
ok: boolean
|
|
23
|
+
state: NativeVoiceState
|
|
24
|
+
session?: string | null
|
|
25
|
+
already?: boolean
|
|
26
|
+
reason?: string
|
|
27
|
+
error?: string
|
|
28
|
+
/** Raw stdout/stderr tail for the log when the answer did not parse. */
|
|
29
|
+
raw?: string
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
export type NativeVoiceControl = {
|
|
33
|
+
open(peer: string): Promise<NativeVoiceReceipt>
|
|
34
|
+
status(peer: string): Promise<NativeVoiceReceipt>
|
|
35
|
+
close(peer: string): Promise<NativeVoiceReceipt>
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
const STATES = new Set(['open', 'opening', 'closing', 'closed'])
|
|
39
|
+
|
|
40
|
+
/** Parse the LAST JSON object line of a contract answer. Anything else is not a receipt:
|
|
41
|
+
* a confirmation that cannot be read is treated as no confirmation. */
|
|
42
|
+
export function parseVoiceReceipt(stdout: string, stderr: string, exitCode: number | null): NativeVoiceReceipt {
|
|
43
|
+
const lines = `${stdout}`.split(/\r?\n/).map(l => l.trim()).filter(l => l.startsWith('{') && l.endsWith('}'))
|
|
44
|
+
for (const line of lines.reverse()) {
|
|
45
|
+
try {
|
|
46
|
+
const obj = JSON.parse(line) as Record<string, unknown>
|
|
47
|
+
const state = typeof obj.state === 'string' && STATES.has(obj.state) ? (obj.state as NativeVoiceState) : 'unknown'
|
|
48
|
+
const errorText = [obj.error, obj.err, obj.detail].find(v => typeof v === 'string') as string | undefined
|
|
49
|
+
return {
|
|
50
|
+
ok: obj.ok === false ? false : exitCode === 0,
|
|
51
|
+
state,
|
|
52
|
+
session: typeof obj.session === 'string' ? obj.session : null,
|
|
53
|
+
already: obj.already === true,
|
|
54
|
+
reason: typeof obj.reason === 'string' ? obj.reason : undefined,
|
|
55
|
+
error: exitCode === 0 && obj.ok !== false ? undefined : (errorText ?? `exit ${exitCode}`),
|
|
56
|
+
}
|
|
57
|
+
} catch {}
|
|
58
|
+
}
|
|
59
|
+
const raw = `${stdout}\n${stderr}`.trim().slice(-300)
|
|
60
|
+
return { ok: false, state: 'unknown', error: raw || `exit ${exitCode}`, raw }
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** The CLI implementation of the contract. Never throws; a timeout is a failed receipt. */
|
|
64
|
+
export function cliNativeVoice(opts: {
|
|
65
|
+
bin: string
|
|
66
|
+
env?: NodeJS.ProcessEnv
|
|
67
|
+
cwd?: string
|
|
68
|
+
openTimeoutMs?: number
|
|
69
|
+
statusTimeoutMs?: number
|
|
70
|
+
closeTimeoutMs?: number
|
|
71
|
+
}): NativeVoiceControl {
|
|
72
|
+
const run = (verb: string, peer: string, timeoutMs: number) =>
|
|
73
|
+
new Promise<NativeVoiceReceipt>(resolve => {
|
|
74
|
+
let stdout = ''
|
|
75
|
+
let stderr = ''
|
|
76
|
+
let settled = false
|
|
77
|
+
const finish = (r: NativeVoiceReceipt) => {
|
|
78
|
+
if (settled) return
|
|
79
|
+
settled = true
|
|
80
|
+
clearTimeout(timer)
|
|
81
|
+
resolve(r)
|
|
82
|
+
}
|
|
83
|
+
let child: ReturnType<typeof spawn>
|
|
84
|
+
try {
|
|
85
|
+
child = spawn(opts.bin, ['voice', verb, peer, '--json'], {
|
|
86
|
+
cwd: opts.cwd,
|
|
87
|
+
env: opts.env ?? process.env,
|
|
88
|
+
stdio: ['ignore', 'pipe', 'pipe'],
|
|
89
|
+
})
|
|
90
|
+
} catch (err) {
|
|
91
|
+
resolve({ ok: false, state: 'unknown', error: err instanceof Error ? err.message : String(err) })
|
|
92
|
+
return
|
|
93
|
+
}
|
|
94
|
+
const timer = setTimeout(() => {
|
|
95
|
+
try {
|
|
96
|
+
child.kill('SIGKILL')
|
|
97
|
+
} catch {}
|
|
98
|
+
finish({ ok: false, state: 'unknown', error: `voice ${verb} timed out after ${Math.ceil(timeoutMs / 1000)}s` })
|
|
99
|
+
}, timeoutMs)
|
|
100
|
+
child.stdout?.on('data', d => (stdout += String(d)))
|
|
101
|
+
child.stderr?.on('data', d => (stderr += String(d)))
|
|
102
|
+
child.on('error', err => finish({ ok: false, state: 'unknown', error: err.message }))
|
|
103
|
+
child.on('close', code => finish(parseVoiceReceipt(stdout, stderr, code)))
|
|
104
|
+
})
|
|
105
|
+
return {
|
|
106
|
+
open: peer => run('open', peer, opts.openTimeoutMs ?? 20_000),
|
|
107
|
+
status: peer => run('status', peer, opts.statusTimeoutMs ?? 5_000),
|
|
108
|
+
close: peer => run('close', peer, opts.closeTimeoutMs ?? 15_000),
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
/** Which targets are answered in native voice: the passport's `native_voice` list, or the
|
|
113
|
+
* `VOICETALK_NATIVE_VOICE` override (comma list; empty string disables). */
|
|
114
|
+
export function nativeVoiceTargets(binding: { native_voice?: unknown }, env = process.env): Set<string> {
|
|
115
|
+
const override = env.VOICETALK_NATIVE_VOICE
|
|
116
|
+
const source: unknown = override !== undefined ? override.split(',') : binding.native_voice
|
|
117
|
+
const list = Array.isArray(source) ? source : typeof source === 'string' ? source.split(',') : []
|
|
118
|
+
return new Set(list.map(v => String(v).trim()).filter(Boolean))
|
|
119
|
+
}
|
package/src/voiceSession.ts
CHANGED
|
@@ -43,6 +43,7 @@ import {
|
|
|
43
43
|
type Segment,
|
|
44
44
|
} from './audio.ts'
|
|
45
45
|
import { synthesize, transcribe } from './voiceConnect.ts'
|
|
46
|
+
import { nativeVoiceTargets, type NativeVoiceControl } from './nativeVoice.ts'
|
|
46
47
|
import {
|
|
47
48
|
buildAddressBook,
|
|
48
49
|
hallucinationReason,
|
|
@@ -104,6 +105,11 @@ const PLAYBACK_ENABLED = process.env.VOICETALK_PLAYBACK !== '0'
|
|
|
104
105
|
* are mode 0600 under the runtime scratch dir and the oldest pair is evicted. */
|
|
105
106
|
const WAKE_CAPTURE_ENABLED = process.env.VOICETALK_WAKE_CAPTURE !== '0'
|
|
106
107
|
const WAKE_CAPTURE_MAX = Math.min(100, num(process.env.VOICETALK_WAKE_CAPTURE_MAX, 16))
|
|
108
|
+
/** Native voice hand-off: how often the listener asks whether the peer's voice is closed,
|
|
109
|
+
* and the hard ceiling after which it asks iapeer to close it. Idle-close itself belongs
|
|
110
|
+
* to iapeer; the ceiling only bounds a hand-off whose closing was never confirmed. */
|
|
111
|
+
const NATIVE_POLL_MS = num(process.env.VOICETALK_NATIVE_POLL_MS, 1_000)
|
|
112
|
+
const NATIVE_MAX_MS = num(process.env.VOICETALK_NATIVE_MAX_MS, 30 * 60_000)
|
|
107
113
|
|
|
108
114
|
export type DeliverFn = (target: string, text: string) => Promise<{ ok: boolean; detail?: string }>
|
|
109
115
|
export type LogFn = (evt: string, fields: Record<string, unknown>) => void
|
|
@@ -116,6 +122,21 @@ export type VoiceSessionOptions = {
|
|
|
116
122
|
/** Scratch dir for captured WAVs and synthesized OGGs. */
|
|
117
123
|
tmpDir: string
|
|
118
124
|
peers?: FleetPeer[]
|
|
125
|
+
/** The door to a peer's native voice mode; absent = every target goes through STT. */
|
|
126
|
+
nativeVoice?: NativeVoiceControl
|
|
127
|
+
/** Override the hand-off poll / hold ceiling (tests); defaults come from the env. */
|
|
128
|
+
nativePollMs?: number
|
|
129
|
+
nativeMaxMs?: number
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
/** The microphone is lent to a peer's native voice. `releasing` and `reclaiming` are the
|
|
133
|
+
* edges where nobody listens (earcons are allowed there); during `opening` and `open`
|
|
134
|
+
* the room belongs to the peer and voicetalk stays silent. */
|
|
135
|
+
type NativeHandOff = {
|
|
136
|
+
target: string
|
|
137
|
+
phase: 'releasing' | 'opening' | 'open' | 'reclaiming'
|
|
138
|
+
since: number
|
|
139
|
+
session?: string | null
|
|
119
140
|
}
|
|
120
141
|
|
|
121
142
|
type WakeRoutingDecision = {
|
|
@@ -153,8 +174,18 @@ export class VoiceSession {
|
|
|
153
174
|
/** When the current turn was handed to the fleet — the clock for «ответ за N с». */
|
|
154
175
|
private deliveredAt = new Map<string, number>()
|
|
155
176
|
private stopped = false
|
|
177
|
+
private nativeTargets: Set<string>
|
|
178
|
+
private native: NativeHandOff | null = null
|
|
179
|
+
private nativeWaiters: Array<() => void> = []
|
|
180
|
+
|
|
181
|
+
constructor(private readonly opts: VoiceSessionOptions) {
|
|
182
|
+
this.nativeTargets = opts.nativeVoice ? nativeVoiceTargets(opts.binding) : new Set()
|
|
183
|
+
}
|
|
156
184
|
|
|
157
|
-
|
|
185
|
+
/** Who currently holds the microphone instead of the wake listener (null = we do). */
|
|
186
|
+
get nativeHolder(): { target: string; phase: string } | null {
|
|
187
|
+
return this.native ? { target: this.native.target, phase: this.native.phase } : null
|
|
188
|
+
}
|
|
158
189
|
|
|
159
190
|
/** Resolve the input device, prime the address book, open the mic. Throws if the mic
|
|
160
191
|
* cannot be resolved — the caller decides whether that is fatal (it is not: the
|
|
@@ -173,7 +204,10 @@ export class VoiceSession {
|
|
|
173
204
|
onStart: attempt => this.opts.log('mic.start', { device: device.index, name: device.name, attempt }),
|
|
174
205
|
onError: detail => this.opts.log('mic.error', { detail }),
|
|
175
206
|
})
|
|
176
|
-
|
|
207
|
+
// A restart must not grab a microphone a peer is still talking on: adopt an open
|
|
208
|
+
// native voice and wait for its close before listening.
|
|
209
|
+
const adopted = await this.adoptOpenNativeVoice()
|
|
210
|
+
if (!adopted) this.mic.start()
|
|
177
211
|
return { deviceIndex: device.index, deviceName: device.name, addresses: this.book.length }
|
|
178
212
|
}
|
|
179
213
|
|
|
@@ -216,6 +250,9 @@ export class VoiceSession {
|
|
|
216
250
|
|
|
217
251
|
private async earcon(kind: keyof Earcons): Promise<void> {
|
|
218
252
|
if (!this.earcons || process.env.VOICETALK_EARCONS === '0' || !PLAYBACK_ENABLED) return
|
|
253
|
+
// The peer's own voice session is listening to the room — a beep from us would be
|
|
254
|
+
// heard (and possibly answered) by it.
|
|
255
|
+
if (this.native && (this.native.phase === 'opening' || this.native.phase === 'open')) return
|
|
219
256
|
try {
|
|
220
257
|
this.muteUntil = Math.max(this.muteUntil, Date.now() + 600)
|
|
221
258
|
await playFile(this.earcons[kind], { timeoutMs: 5_000 })
|
|
@@ -229,7 +266,7 @@ export class VoiceSession {
|
|
|
229
266
|
}
|
|
230
267
|
|
|
231
268
|
private onAudio(chunk: Buffer): void {
|
|
232
|
-
if (this.stopped) return
|
|
269
|
+
if (this.stopped || this.native) return
|
|
233
270
|
if (this.muted()) {
|
|
234
271
|
// Our own speech (or an earcon) is in the room — throw the audio away rather than
|
|
235
272
|
// letting the endpointer build an utterance out of it.
|
|
@@ -439,10 +476,18 @@ export class VoiceSession {
|
|
|
439
476
|
// goes out before the new one starts.
|
|
440
477
|
this.opts.log('wake.switch', { from: active, to: match!.target })
|
|
441
478
|
this.flush('switch')
|
|
442
|
-
this.
|
|
479
|
+
if (this.nativeTargets.has(match!.target)) this.handOff(match!)
|
|
480
|
+
else this.append(match!.target, match!.body)
|
|
443
481
|
return { candidate: match!, accepted: true }
|
|
444
482
|
}
|
|
445
483
|
|
|
484
|
+
if (match && this.nativeTargets.has(match.target)) {
|
|
485
|
+
// An accepted, spoken, exact address to a native-voice peer: the conversation moves
|
|
486
|
+
// to the peer's own realtime voice. Nothing is transcribed onward or sent as text.
|
|
487
|
+
this.handOff(match)
|
|
488
|
+
return { candidate: match, accepted: true }
|
|
489
|
+
}
|
|
490
|
+
|
|
446
491
|
if (match) {
|
|
447
492
|
if (!match.body) {
|
|
448
493
|
// Addressed with nothing to say yet («Борис?») — acknowledge and hold the mic
|
|
@@ -569,6 +614,155 @@ export class VoiceSession {
|
|
|
569
614
|
this.followTimer = null
|
|
570
615
|
}
|
|
571
616
|
|
|
617
|
+
// ── native voice hand-off ─────────────────────────────────────────────────────
|
|
618
|
+
|
|
619
|
+
/** One accepted wake → one hand-off. Synchronously marks the microphone as lent (so no
|
|
620
|
+
* second wake can be formed from audio already captured), then runs the hand-off. */
|
|
621
|
+
private handOff(match: WakeMatch): void {
|
|
622
|
+
if (this.native) {
|
|
623
|
+
this.opts.log('native.ignored', { target: match.target, holder: this.native.target, phase: this.native.phase })
|
|
624
|
+
return
|
|
625
|
+
}
|
|
626
|
+
this.native = { target: match.target, phase: 'releasing', since: Date.now() }
|
|
627
|
+
// Audio captured before the wake was handled belongs to no one now: the owner is
|
|
628
|
+
// about to talk to the peer directly. A thought in flight to someone ELSE still goes.
|
|
629
|
+
const dropped = this.queue.length
|
|
630
|
+
this.queue = []
|
|
631
|
+
if (this.pending) this.flush('native-voice')
|
|
632
|
+
this.clearWindow()
|
|
633
|
+
this.followTarget = null
|
|
634
|
+
this.opts.log('native.wake', { target: match.target, alias: match.alias, body: match.body || null, droppedSegments: dropped })
|
|
635
|
+
void this.runHandOff(match.target)
|
|
636
|
+
}
|
|
637
|
+
|
|
638
|
+
private async runHandOff(target: string): Promise<void> {
|
|
639
|
+
const control = this.opts.nativeVoice!
|
|
640
|
+
const released = this.mic ? await this.mic.pause() : { released: true }
|
|
641
|
+
this.segmenter.reset()
|
|
642
|
+
this.opts.log('mic.released', { to: target, confirmed: released.released })
|
|
643
|
+
if (!released.released) {
|
|
644
|
+
// Our capture has not provably let go of the device; opening the peer's voice now
|
|
645
|
+
// could put two listeners on one microphone. Abort this activation instead.
|
|
646
|
+
this.native!.phase = 'reclaiming'
|
|
647
|
+
await this.earcon('error')
|
|
648
|
+
this.reclaim(target, 'release-unconfirmed')
|
|
649
|
+
return
|
|
650
|
+
}
|
|
651
|
+
// "Heard you" — played while NOBODY is listening: our capture is gone and the peer's
|
|
652
|
+
// voice is not open yet.
|
|
653
|
+
await this.earcon('end')
|
|
654
|
+
if (this.stopped) return
|
|
655
|
+
this.native!.phase = 'opening'
|
|
656
|
+
this.opts.log('native.open-requested', { target })
|
|
657
|
+
const started = Date.now()
|
|
658
|
+
const receipt = await control.open(target)
|
|
659
|
+
if (!receipt.ok || receipt.state !== 'open') {
|
|
660
|
+
// A failed open is NOT evidence that the peer is off the microphone: the realtime
|
|
661
|
+
// session may still start late, and a timeout or an unreadable answer says nothing
|
|
662
|
+
// at all. The microphone comes back only on a confirmed `closed` (contract with
|
|
663
|
+
// iapeer, 26.10) — until then we hold exactly as if the voice were open.
|
|
664
|
+
this.opts.log('native.open-failed', { target, state: receipt.state, error: receipt.error, openMs: Date.now() - started })
|
|
665
|
+
this.native!.phase = 'open'
|
|
666
|
+
await this.holdUntilClosed(target, { failedOpen: true })
|
|
667
|
+
return
|
|
668
|
+
}
|
|
669
|
+
this.native!.phase = 'open'
|
|
670
|
+
this.native!.session = receipt.session ?? null
|
|
671
|
+
this.opts.log('native.opened', { target, session: receipt.session ?? null, already: !!receipt.already, openMs: Date.now() - started })
|
|
672
|
+
await this.holdUntilClosed(target)
|
|
673
|
+
}
|
|
674
|
+
|
|
675
|
+
/** Poll the peer's voice state until iapeer confirms it is closed, then give the
|
|
676
|
+
* microphone back to the wake listener. */
|
|
677
|
+
private async holdUntilClosed(target: string, opts: { failedOpen?: boolean } = {}): Promise<void> {
|
|
678
|
+
const control = this.opts.nativeVoice!
|
|
679
|
+
const openedAt = Date.now()
|
|
680
|
+
let closeRequested = false
|
|
681
|
+
let lastAlarm = Date.now()
|
|
682
|
+
let lastState = ''
|
|
683
|
+
while (!this.stopped) {
|
|
684
|
+
await new Promise(r => setTimeout(r, this.opts.nativePollMs ?? NATIVE_POLL_MS))
|
|
685
|
+
if (this.stopped) return
|
|
686
|
+
const st = await control.status(target)
|
|
687
|
+
if (st.state !== lastState) {
|
|
688
|
+
this.opts.log('native.status', { target, state: st.state, reason: st.reason ?? null, error: st.error ?? null })
|
|
689
|
+
lastState = st.state
|
|
690
|
+
}
|
|
691
|
+
if (st.state === 'closed') {
|
|
692
|
+
this.opts.log('native.closed', { target, session: st.session ?? this.native?.session ?? null, reason: st.reason ?? null, heldMs: Date.now() - openedAt })
|
|
693
|
+
break
|
|
694
|
+
}
|
|
695
|
+
if (!closeRequested && Date.now() - openedAt >= (this.opts.nativeMaxMs ?? NATIVE_MAX_MS)) {
|
|
696
|
+
closeRequested = true
|
|
697
|
+
this.opts.log('native.close-requested', { target, reason: 'hold ceiling', heldMs: Date.now() - openedAt })
|
|
698
|
+
const closed = await control.close(target)
|
|
699
|
+
this.opts.log('native.close-receipt', { target, ok: closed.ok, state: closed.state, error: closed.error ?? null })
|
|
700
|
+
}
|
|
701
|
+
if (closeRequested && Date.now() - lastAlarm >= 60_000) {
|
|
702
|
+
// Past the ceiling and still no confirmed close. Taking the microphone back
|
|
703
|
+
// unconfirmed would put two listeners on one device; staying deaf is the
|
|
704
|
+
// recoverable failure, so it is reported loudly instead of resolved by guess.
|
|
705
|
+
lastAlarm = Date.now()
|
|
706
|
+
this.opts.log('native.stuck', { target, lastState, heldMs: Date.now() - openedAt })
|
|
707
|
+
const closed = await control.close(target)
|
|
708
|
+
this.opts.log('native.close-receipt', { target, ok: closed.ok, state: closed.state, error: closed.error ?? null })
|
|
709
|
+
}
|
|
710
|
+
}
|
|
711
|
+
if (this.stopped) return
|
|
712
|
+
this.native!.phase = 'reclaiming'
|
|
713
|
+
if (opts.failedOpen) await this.earcon('error')
|
|
714
|
+
this.reclaim(target, opts.failedOpen ? 'open-failed' : 'closed')
|
|
715
|
+
if (!opts.failedOpen) void this.earcon('close')
|
|
716
|
+
}
|
|
717
|
+
|
|
718
|
+
private reclaim(target: string, reason: string): void {
|
|
719
|
+
this.segmenter.reset()
|
|
720
|
+
this.native = null
|
|
721
|
+
// Back to wake mode proper: after a native conversation the next turn needs a name.
|
|
722
|
+
this.clearWindow()
|
|
723
|
+
this.followTarget = null
|
|
724
|
+
this.mic?.resume()
|
|
725
|
+
this.opts.log('wake.listening', { after: target, reason })
|
|
726
|
+
const waiters = this.nativeWaiters
|
|
727
|
+
this.nativeWaiters = []
|
|
728
|
+
for (const w of waiters) w()
|
|
729
|
+
}
|
|
730
|
+
|
|
731
|
+
/** On start: if a native voice is already open (the runtime restarted mid-conversation),
|
|
732
|
+
* stay off the microphone until it closes. */
|
|
733
|
+
private async adoptOpenNativeVoice(): Promise<boolean> {
|
|
734
|
+
const control = this.opts.nativeVoice
|
|
735
|
+
if (!control) return false
|
|
736
|
+
for (const target of [...this.nativeTargets]) {
|
|
737
|
+
const st = await control.status(target)
|
|
738
|
+
if (st.state === 'unknown') {
|
|
739
|
+
// Preflight: the contract does not answer at all (an iapeer without `voice`, a
|
|
740
|
+
// broken CLI). Nothing can have been opened by THIS process yet, so the safe
|
|
741
|
+
// degradation is the STT path for this target — not a listener that goes deaf
|
|
742
|
+
// forever on the first wake because no `closed` can ever be confirmed.
|
|
743
|
+
this.nativeTargets.delete(target)
|
|
744
|
+
this.opts.log('native.unavailable', { target, error: st.error ?? null, fallback: 'stt' })
|
|
745
|
+
continue
|
|
746
|
+
}
|
|
747
|
+
if (st.state === 'open' || st.state === 'opening' || st.state === 'closing') {
|
|
748
|
+
this.native = { target, phase: 'open', since: Date.now(), session: st.session ?? null }
|
|
749
|
+
this.opts.log('native.adopted', { target, state: st.state, session: st.session ?? null })
|
|
750
|
+
void this.holdUntilClosed(target)
|
|
751
|
+
return true
|
|
752
|
+
}
|
|
753
|
+
// Positive preflight receipt: the contract answered and the target is live in
|
|
754
|
+
// native mode. Without this line a healthy start is indistinguishable from one that
|
|
755
|
+
// never ran the preflight.
|
|
756
|
+
this.opts.log('native.ready', { target, state: st.state })
|
|
757
|
+
}
|
|
758
|
+
return false
|
|
759
|
+
}
|
|
760
|
+
|
|
761
|
+
private waitNativeIdle(): Promise<void> {
|
|
762
|
+
if (!this.native) return Promise.resolve()
|
|
763
|
+
return new Promise(resolve => this.nativeWaiters.push(resolve))
|
|
764
|
+
}
|
|
765
|
+
|
|
572
766
|
/** Speak an agent's answer on the owner's speaker, then hold the follow-up window
|
|
573
767
|
* open — the conversational turn passes back to the human. */
|
|
574
768
|
async speak(from: string, message: string): Promise<void> {
|
|
@@ -596,6 +790,11 @@ export class VoiceSession {
|
|
|
596
790
|
}
|
|
597
791
|
|
|
598
792
|
private async say(text: string, opts: { from?: string } = {}): Promise<void> {
|
|
793
|
+
if (this.native) {
|
|
794
|
+
// The room belongs to a peer's native voice; speaking now would be heard by it.
|
|
795
|
+
this.opts.log('tts.deferred', { from: opts.from ?? null, holder: this.native.target })
|
|
796
|
+
await this.waitNativeIdle()
|
|
797
|
+
}
|
|
599
798
|
const out = join(this.opts.tmpDir, `say-${randomUUID()}.ogg`)
|
|
600
799
|
const started = Date.now()
|
|
601
800
|
try {
|
package/src/wake.ts
CHANGED
|
@@ -120,6 +120,10 @@ export type VoicetalkBinding = {
|
|
|
120
120
|
wake_words?: Record<string, string> | string[]
|
|
121
121
|
default_target?: string
|
|
122
122
|
device?: string
|
|
123
|
+
/** Targets answered in their own NATIVE voice mode (Codex `/voice`) instead of
|
|
124
|
+
* STT → envelope → TTS. The wake word still has to be spoken and pass every activation
|
|
125
|
+
* check — this only changes what an accepted activation does. */
|
|
126
|
+
native_voice?: string[]
|
|
123
127
|
}
|
|
124
128
|
|
|
125
129
|
export type AddressBookOptions = {
|