@inetafrica/open-claudia 3.5.26 → 3.5.27
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -16
- package/channels/kazee/call/manager.js +137 -2
- package/channels/kazee/call/quick-voice.js +295 -0
- package/channels/kazee/call/session.js +24 -2
- package/core/browser-control.js +17 -147
- package/core/browser-lock.js +2 -33
- package/core/browser-view.js +5 -90
- package/core/cards.js +0 -18
- package/core/loopback.js +1 -4
- package/core/system-prompt.js +11 -3
- package/core/tool-guard.js +0 -3
- package/package.json +2 -2
- package/test-kazee-call-manager.js +5 -0
- package/test-kazee-call-session.js +3 -0
- package/test-provider-language.js +5 -8
- package/test-provider-prompt-parity.js +1 -0
- package/core/browser-drive.js +0 -619
package/CHANGELOG.md
CHANGED
|
@@ -1,23 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 3.5.
|
|
3
|
+
## 3.5.27
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Calls answer in about two seconds again, and say something while they work (Sumeet, 11 Oct 08:24 EAT on a walk, after a call where "hello" took 8.5 s and a 79 s lookup was never heard: "Yeah, let's give it a shot"). A canary build for the owner's own bot first. It builds on 3.5.24; the Browser canaries 3.5.25 and 3.5.26 are not in it yet.
|
|
6
6
|
|
|
7
|
-
- **
|
|
8
|
-
- **
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
The Browser app feels like a browser on the phone and on the web (Sumeet, 10 Oct 08:53 EAT on his phone: "it feels like a browser on my phone… like Google Chrome app", and 09:18: "the web and mobile still need to also feel native… don't forget the website"). A canary build for the Browser canary bot first; the fleet waits for his OK.
|
|
13
|
-
|
|
14
|
-
- **The bot's tab takes the driver's screen.** With Chrome in its own container, a take (or a `fit` when the screen changes: a turned phone, a resized panel, a keyboard that opens) that carries the driver's screen (`viewport`: CSS size, pixel density, phone layout, touch) fits the bot's tab to it through `core/browser-drive.js`. A phone gets the page's phone layout, full screen and sharp; the web gets a desktop page at the panel's size. Handing back puts the tab back to its own size. The browser's identity (user agent) never changes, so sites signed into stay signed in after the hand back. `GET /api/browser/control` says `fit: true`.
|
|
15
|
-
- **The driver's input goes straight to Chrome,** through the view's own DevTools session, in order. Touches keep their timing, so a swipe flings; typed text is inserted whole (`input_text`, at most 500 characters: accents, emoji, the keyboard's corrections); every point is scaled by the view's density, because Chrome reads input in the drawn picture's pixels. A bot without the view takes `input_text` as key presses.
|
|
16
|
-
- **The page's text fields.** The view reads where the page's text fields are and what kind they are (never a value) and streams them as `fields` events, so the Browser page opens the phone's own keyboard of the right kind the moment a field is tapped.
|
|
17
|
-
- **Pictures.** The view streams its own screencast at the driver's size (at most 1.6 million pixels, 30 a second). `/api/browser/stream` serves it while someone drives, up to 30 fps, and keeps the newest frame while a slow viewer drains instead of dropping it, so the last picture of a scroll always arrives.
|
|
18
|
-
- **A hand back wakes the bot only when it waits for that person:** it sent them an Open Browser card (cards now record `openApp`), or one of its browser commands was refused while they drove (the tool guard marks it). Otherwise the hand back is only logged. On 9 Oct a test hand back set off a 28-minute tangent on the canary.
|
|
19
|
-
- **Correction to 3.5.24:** in container mode, sessions are not pinned to tabs (Agent Space dropped `AGENT_BROWSER_PIN_TAB` in a98f239); a session uses the tab that is open.
|
|
20
|
-
- **Tests:** `test-browser-drive.js` (new), `test-browser-control.js`, `test-web-browser-app.js`.
|
|
7
|
+
- **A quick voice says a fitting line right away** (`channels/kazee/call/quick-voice.js`). A kept-warm small model, one process per call through the provider's own utility invocation, hears each utterance and says one short line ("Sure, let me check the call logs.", "Yep, I'm here.") within about a second, then a brief "still on it" line whenever the full agent leaves the caller in 8 s of silence, at most 3 per turn. It never answers: every utterance still goes to the full agent. The line is dropped if it is not ready in 3.5 s, if the full agent already spoke, if the caller is talking, or once a newer utterance took over. Off unless `KAZEE_CALL_QUICK_VOICE=on`; `KAZEE_CALL_QUICK_MODEL` (default `claude-haiku-4-5`), `_EFFORT` (`low`) and `_THINKING` (off) tune it. Measured on 23 call lines with traps ("Did you hear me?", "Are you sure?"): Haiku 4.5 finished its line in 0.8-1.1 s with thinking off (2-3 s with it on), Sonnet 5 in 1.7-1.9 s, and both kept every rule.
|
|
8
|
+
- **The full agent speaks first, then works.** The live-call guidance used to forbid "speculative progress", so a lookup went silent until it ended. Now the first action is a 3-12 word call-say ("Sure, let me check the call logs."), then a progress line about every ten seconds. Replayed on the four turns of the 11 Oct call, 3 times per effort level: the holding line came first 3 of 3 times at max, xhigh, high and medium (the old guidance: 1 of 3 at xhigh), simple replies unchanged. With the quick voice on, the agent skips its own acknowledgement and starts with substance.
|
|
9
|
+
- **A sound with no words no longer silences the answer.** Speech start mutes the running call turn in case it is a new ask; when the recogniser found no words (wind, footsteps, a cough) the turn stayed muted for good, its call-say got 409 and its final answer was never spoken. The session now reports a noise-only turn and the turn gets its voice back, unless someone is still talking or another segment is pending.
|
|
10
|
+
- **Tests:** `test-call-quick-voice.js` (new, in `npm test`), `test-kazee-call-manager.js`, `test-kazee-call-session.js`, `test-provider-prompt-parity.js`.
|
|
21
11
|
|
|
22
12
|
## 3.5.24
|
|
23
13
|
|
|
@@ -5,6 +5,7 @@ const { CallLeaseHeartbeat } = require("./lease");
|
|
|
5
5
|
const { publicCallIdentity, requestRoomAccess, supportedIceServers } = require("./access");
|
|
6
6
|
const { canonicalForChannel } = require("../../../core/identity");
|
|
7
7
|
const { KazeeCallSession } = require("./session");
|
|
8
|
+
const quickVoice = require("./quick-voice");
|
|
8
9
|
|
|
9
10
|
const DEFAULT_JANUS_URL = "wss://janus.kazee.africa/ws";
|
|
10
11
|
const DEFAULT_ICE_SERVERS = [
|
|
@@ -14,6 +15,34 @@ const DEFAULT_ICE_SERVERS = [
|
|
|
14
15
|
// FCM push TTL is 60s and chat-central treats ringing logs as fresh for 90s.
|
|
15
16
|
// This is the outbound invitation deadline; terminal server events end it early.
|
|
16
17
|
const RING_TIMEOUT_MS = 60_000;
|
|
18
|
+
const qlog = (msg) => console.log(`${new Date().toISOString()} Kazee call quick: ${msg}`);
|
|
19
|
+
// The quick line must be ready this soon after the words, or it is skipped:
|
|
20
|
+
// by then the full agent is close to speaking itself.
|
|
21
|
+
const QUICK_LINE_MAX_MS = 3500;
|
|
22
|
+
// While the full agent works: a "still on it" line after this much quiet,
|
|
23
|
+
// at most a few per turn, never after the answer.
|
|
24
|
+
const QUICK_FILLER_QUIET_MS = 8000;
|
|
25
|
+
const QUICK_FILLER_MAX = 3;
|
|
26
|
+
const QUICK_TICK_MS = 500;
|
|
27
|
+
|
|
28
|
+
// The provider the call's chat runs on: the quick voice uses its CLI and
|
|
29
|
+
// credentials, as the full agent's turns do.
|
|
30
|
+
function chatProvider({ adapter, chatId, userId }) {
|
|
31
|
+
const { runInChat } = require("../../../core/context");
|
|
32
|
+
const { conversationKeyFor, roomKeyFor } = require("../../../core/identity");
|
|
33
|
+
const transport = adapter?.type || "kazee";
|
|
34
|
+
const id = runInChat({
|
|
35
|
+
adapter, channelId: String(chatId), userId: String(userId), transport,
|
|
36
|
+
canonicalUserId: canonicalForChannel(transport, String(userId)),
|
|
37
|
+
conversationKey: conversationKeyFor(transport, String(chatId)),
|
|
38
|
+
roomKey: roomKeyFor(transport, String(chatId)),
|
|
39
|
+
}, () => {
|
|
40
|
+
const state = require("../../../core/state");
|
|
41
|
+
return state.getActiveProvider(state.currentState());
|
|
42
|
+
});
|
|
43
|
+
return id ? require("../../../core/providers").registry.getProvider(id) : null;
|
|
44
|
+
}
|
|
45
|
+
|
|
17
46
|
const OUTGOING_GREETING = "Hey, it's Claudia. Can you hear me okay?";
|
|
18
47
|
// Inbound must greet too: callers get silence until they speak first, which
|
|
19
48
|
// reads as a dead call (live test 2026-08-08 — media was up, caller hung up).
|
|
@@ -21,7 +50,8 @@ const INCOMING_GREETING = "Hey, it's Claudia — you're through. Can you hear me
|
|
|
21
50
|
|
|
22
51
|
class KazeeCallManager {
|
|
23
52
|
constructor({ adapter, SessionImpl = KazeeCallSession, janusUrl = process.env.KAZEE_JANUS_URL || DEFAULT_JANUS_URL,
|
|
24
|
-
iceServers = DEFAULT_ICE_SERVERS, deviceId = randomUUID()
|
|
53
|
+
iceServers = DEFAULT_ICE_SERVERS, deviceId = randomUUID(), resolveQuickProvider = chatProvider, quickSpawn = undefined,
|
|
54
|
+
quickSettings = quickVoice.quickVoiceSettings, quickTiming = {} } = {}) {
|
|
25
55
|
this.adapter = adapter;
|
|
26
56
|
this.SessionImpl = SessionImpl;
|
|
27
57
|
this.janusUrl = janusUrl;
|
|
@@ -30,6 +60,10 @@ class KazeeCallManager {
|
|
|
30
60
|
this.deviceId = deviceId;
|
|
31
61
|
this._pending = null;
|
|
32
62
|
this._ringTimer = null;
|
|
63
|
+
this.resolveQuickProvider = resolveQuickProvider;
|
|
64
|
+
this.quickSpawn = quickSpawn;
|
|
65
|
+
this.quickSettings = quickSettings;
|
|
66
|
+
this.quickTiming = { lineMaxMs: QUICK_LINE_MAX_MS, fillerQuietMs: QUICK_FILLER_QUIET_MS, fillerMax: QUICK_FILLER_MAX, tickMs: QUICK_TICK_MS, ...quickTiming };
|
|
33
67
|
}
|
|
34
68
|
|
|
35
69
|
wire(socket) {
|
|
@@ -89,6 +123,7 @@ class KazeeCallManager {
|
|
|
89
123
|
// Several terminal paths clear this.session before close() calls back.
|
|
90
124
|
// Invalidate the closed call's work even when it no longer owns the slot.
|
|
91
125
|
session?._voiceTurn?.controller.abort("call-ended");
|
|
126
|
+
this._stopQuickVoice(session);
|
|
92
127
|
if (this.session !== session) return;
|
|
93
128
|
this._detachSession(session);
|
|
94
129
|
if (this.adapter._socket?.connected) this.adapter._socket.emit("call:end", {
|
|
@@ -98,6 +133,7 @@ class KazeeCallManager {
|
|
|
98
133
|
|
|
99
134
|
_detachSession(session) {
|
|
100
135
|
session?._voiceTurn?.controller.abort("call-ended");
|
|
136
|
+
this._stopQuickVoice(session);
|
|
101
137
|
if (this.session === session) this.session = null;
|
|
102
138
|
}
|
|
103
139
|
|
|
@@ -151,12 +187,14 @@ class KazeeCallManager {
|
|
|
151
187
|
...this._sessionOptions(config, access, boundIdentity, pending.controller.signal), greeting: INCOMING_GREETING,
|
|
152
188
|
onTranscript: (text, meta) => this._transcript(text, { ...payload, ...boundIdentity }, meta, session),
|
|
153
189
|
onSpeechStart: () => this._speechStart(session),
|
|
190
|
+
onSpeechNoise: () => this._speechNoise(session),
|
|
154
191
|
onMediaReady: () => this._mediaReady(session),
|
|
155
192
|
onEnded: reason => { session?._leaseHeartbeat?.close(); pending.controller.abort(); this._sessionEnded(session, reason); },
|
|
156
193
|
});
|
|
157
194
|
session.callIdentity = boundIdentity;
|
|
158
195
|
session.chatId = chatId;
|
|
159
196
|
this.session = session;
|
|
197
|
+
this._startQuickVoice(session, chatId, callerId);
|
|
160
198
|
this._manageLease(session, access);
|
|
161
199
|
await session.start();
|
|
162
200
|
if (this.session !== session || session.closed || pending.controller.signal.aborted) return;
|
|
@@ -198,12 +236,14 @@ class KazeeCallManager {
|
|
|
198
236
|
...this._sessionOptions(config, access, identity, pending.controller.signal), asCaller: true, greeting: OUTGOING_GREETING,
|
|
199
237
|
onTranscript: (text, meta) => this._transcript(text, remote, meta, session),
|
|
200
238
|
onSpeechStart: () => this._speechStart(session),
|
|
239
|
+
onSpeechNoise: () => this._speechNoise(session),
|
|
201
240
|
onMediaReady: () => this._mediaReady(session),
|
|
202
241
|
onEnded: reason => { session?._leaseHeartbeat?.close(); pending.controller.abort(); this._clearRingTimer(); this._sessionEnded(session, reason); },
|
|
203
242
|
});
|
|
204
243
|
session.callIdentity = identity;
|
|
205
244
|
session.chatId = chat;
|
|
206
245
|
this.session = session;
|
|
246
|
+
this._startQuickVoice(session, chat, target);
|
|
207
247
|
this._manageLease(session, access);
|
|
208
248
|
await session.start();
|
|
209
249
|
if (this.session !== session || session.closed || pending.controller.signal.aborted) throw new Error("Call cancelled");
|
|
@@ -247,6 +287,96 @@ class KazeeCallManager {
|
|
|
247
287
|
// complete transcript exists; a cough must not discard an accepted ask.
|
|
248
288
|
}
|
|
249
289
|
|
|
290
|
+
// The sound that muted the turn had no words in it: not a new request, so
|
|
291
|
+
// the answer being worked on may speak again (11 Oct: wind while walking
|
|
292
|
+
// muted a lookup's answer for good, and the caller heard nothing).
|
|
293
|
+
_speechNoise(session) {
|
|
294
|
+
const turn = session?._voiceTurn;
|
|
295
|
+
if (!turn?.muted || turn.controller.signal.aborted || this.session !== session || session.closed) return;
|
|
296
|
+
turn.muted = false;
|
|
297
|
+
console.log(`${new Date().toISOString()} Kazee call noise: muted turn resumed`);
|
|
298
|
+
}
|
|
299
|
+
|
|
300
|
+
// ── Quick voice ───────────────────────────────────────────────────
|
|
301
|
+
|
|
302
|
+
_startQuickVoice(session, chatId, userId) {
|
|
303
|
+
let settings;
|
|
304
|
+
try { settings = this.quickSettings(); } catch (_) { return; }
|
|
305
|
+
if (!settings?.enabled) return;
|
|
306
|
+
let provider = null;
|
|
307
|
+
try { provider = this.resolveQuickProvider({ adapter: this.adapter, chatId, userId }); } catch (_) {}
|
|
308
|
+
if (!quickVoice.QuickVoiceProcess.supported(provider)) {
|
|
309
|
+
qlog(`off: provider ${provider?.id || "none"} has no warm stream`);
|
|
310
|
+
return;
|
|
311
|
+
}
|
|
312
|
+
const quick = quickVoice.track(new quickVoice.QuickVoiceProcess({
|
|
313
|
+
provider, settings, spawnImpl: this.quickSpawn, log: qlog,
|
|
314
|
+
systemPrompt: quickVoice.buildQuickPrompt({ ownerName: quickVoice.ownerDisplayName() }),
|
|
315
|
+
}));
|
|
316
|
+
session._quick = quick;
|
|
317
|
+
// The quick voice says the first line itself; the session's delayed cue
|
|
318
|
+
// would only double it.
|
|
319
|
+
session.acknowledgement = async () => null;
|
|
320
|
+
try { quick.start(); } catch (error) {
|
|
321
|
+
qlog(`start failed: ${error.message}`);
|
|
322
|
+
session._quick = null;
|
|
323
|
+
quick.stop().catch?.(() => {});
|
|
324
|
+
}
|
|
325
|
+
}
|
|
326
|
+
|
|
327
|
+
_stopQuickVoice(session) {
|
|
328
|
+
const quick = session?._quick;
|
|
329
|
+
if (!quick) return;
|
|
330
|
+
session._quick = null;
|
|
331
|
+
quick.stop().catch?.(() => {});
|
|
332
|
+
}
|
|
333
|
+
|
|
334
|
+
// One utterance: a short fitting line as soon as the quick voice has it,
|
|
335
|
+
// then a "still on it" line whenever the full agent leaves the caller in
|
|
336
|
+
// silence, until its answer arrives. Never over the caller, never after
|
|
337
|
+
// the answer, never once a newer utterance took over.
|
|
338
|
+
_quickTurn(session, turn, text) {
|
|
339
|
+
const quick = session._quick;
|
|
340
|
+
const timing = this.quickTiming;
|
|
341
|
+
const live = () => !turn.muted && !turn.done && !turn.controller.signal.aborted &&
|
|
342
|
+
!session.closed && this.session === session && session._voiceTurn === turn && session._quick === quick;
|
|
343
|
+
const sayLine = async (message, label, { maxMs = Infinity, unlessSpoke = false } = {}) => {
|
|
344
|
+
const asked = Date.now();
|
|
345
|
+
const { generation, text: reply } = quick.say(message);
|
|
346
|
+
const cancel = () => quick.cancel(generation);
|
|
347
|
+
turn.controller.signal.addEventListener("abort", cancel, { once: true });
|
|
348
|
+
let line = "";
|
|
349
|
+
try { line = quickVoice.spokenLine(await reply); }
|
|
350
|
+
finally { turn.controller.signal.removeEventListener("abort", cancel); }
|
|
351
|
+
const ready = Date.now() - asked;
|
|
352
|
+
if (!line || !live() || ready > maxMs || (unlessSpoke && turn.spoke) || session.callerTalking?.()) {
|
|
353
|
+
qlog(`${label} skipped ready=${ready}ms`);
|
|
354
|
+
return false;
|
|
355
|
+
}
|
|
356
|
+
qlog(`${label} ready=${ready}ms chars=${line.length}`);
|
|
357
|
+
const spoken = await session.speak(line);
|
|
358
|
+
if (spoken === false || !live()) return false;
|
|
359
|
+
this._rememberSpeech(session, "assistant", line);
|
|
360
|
+
return true;
|
|
361
|
+
};
|
|
362
|
+
(async () => {
|
|
363
|
+
await sayLine(quickVoice.utteranceMessage(text), "line", { maxMs: timing.lineMaxMs, unlessSpoke: true });
|
|
364
|
+
let fillers = 0;
|
|
365
|
+
let lastFillerAt = Date.now();
|
|
366
|
+
while (fillers < timing.fillerMax && live()) {
|
|
367
|
+
await new Promise((resolve) => { const timer = setTimeout(resolve, timing.tickMs); timer.unref?.(); });
|
|
368
|
+
if (!live()) break;
|
|
369
|
+
if (session.speechBusy?.() || session.callerTalking?.()) continue;
|
|
370
|
+
const quietSince = Math.max(turn.startedAt, lastFillerAt, session._lastSpokeAt || 0);
|
|
371
|
+
if (Date.now() - quietSince < timing.fillerQuietMs) continue;
|
|
372
|
+
fillers += 1;
|
|
373
|
+
lastFillerAt = Date.now();
|
|
374
|
+
await sayLine(quickVoice.fillerMessage(text, (Date.now() - turn.startedAt) / 1000), `filler ${fillers}`);
|
|
375
|
+
lastFillerAt = Date.now();
|
|
376
|
+
}
|
|
377
|
+
})().catch((error) => qlog(`turn failed: ${error.message}`));
|
|
378
|
+
}
|
|
379
|
+
|
|
250
380
|
_rememberSpeech(session, role, text) {
|
|
251
381
|
const history = session._voiceHistory ||= [];
|
|
252
382
|
if (role === "assistant" && history.at(-1)?.role === "assistant") {
|
|
@@ -262,7 +392,7 @@ class KazeeCallManager {
|
|
|
262
392
|
const speaker = String(meta.display || "").trim();
|
|
263
393
|
const attributed = (meta.participants || 0) > 1 && speaker ? `${speaker}: ${text}` : text;
|
|
264
394
|
session._voiceTurn?.controller.abort("superseded");
|
|
265
|
-
const turn = { controller: new AbortController(), muted: false };
|
|
395
|
+
const turn = { controller: new AbortController(), muted: false, done: false, spoke: false, startedAt: Date.now() };
|
|
266
396
|
session._voiceTurn = turn;
|
|
267
397
|
const callAdapter = Object.create(this.adapter);
|
|
268
398
|
callAdapter.callHistory = (session._voiceHistory || []).map(entry => ({ ...entry }));
|
|
@@ -277,6 +407,8 @@ class KazeeCallManager {
|
|
|
277
407
|
callAdapter.suppressChatVisibility = true;
|
|
278
408
|
// Prompt builder switches to phone-call brevity when it sees this.
|
|
279
409
|
callAdapter.liveCall = true;
|
|
410
|
+
// The quick voice says the first line; the prompt then skips its own.
|
|
411
|
+
callAdapter.quickVoice = !!session._quick?.ready();
|
|
280
412
|
const spokenSentences = [];
|
|
281
413
|
callAdapter.isCallTurnActive = () => !turn.muted && !turn.controller.signal.aborted &&
|
|
282
414
|
!session.closed && this.session === session && session._voiceTurn === turn;
|
|
@@ -287,6 +419,7 @@ class KazeeCallManager {
|
|
|
287
419
|
if (!callAdapter.isCallTurnActive()) return false;
|
|
288
420
|
const spoken = await session.speak(response);
|
|
289
421
|
if (spoken !== false && callAdapter.isCallTurnActive()) {
|
|
422
|
+
turn.spoke = true;
|
|
290
423
|
this._rememberSpeech(session, "assistant", response);
|
|
291
424
|
return true;
|
|
292
425
|
}
|
|
@@ -298,6 +431,7 @@ class KazeeCallManager {
|
|
|
298
431
|
return spoken;
|
|
299
432
|
};
|
|
300
433
|
callAdapter.send = async (_channelId, response) => {
|
|
434
|
+
turn.done = true;
|
|
301
435
|
const remainder = require("../../../core/call-speech").unspokenSuffix(response, spokenSentences);
|
|
302
436
|
if (remainder) await speak(remainder);
|
|
303
437
|
return true;
|
|
@@ -308,6 +442,7 @@ class KazeeCallManager {
|
|
|
308
442
|
console.warn(`Kazee call ignored file voice response ${voicePath}; streamed TTS is active`);
|
|
309
443
|
return true;
|
|
310
444
|
};
|
|
445
|
+
if (callAdapter.quickVoice) this._quickTurn(session, turn, attributed);
|
|
311
446
|
this.adapter._emit("message", {
|
|
312
447
|
adapter: callAdapter,
|
|
313
448
|
channelId: String(incoming.chatId),
|
|
@@ -0,0 +1,295 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
|
|
3
|
+
// The quick voice: a small, kept-warm model that says a short, fitting line
|
|
4
|
+
// the moment the caller finishes ("Sure, let me check the call logs."), and a
|
|
5
|
+
// brief "still on it" line while the full agent works. It never answers: the
|
|
6
|
+
// full agent's call turn still runs for every utterance and speaks the real
|
|
7
|
+
// reply. One process per call, spawned through the provider's own utility
|
|
8
|
+
// invocation (binary, env and credentials come from the provider, never from
|
|
9
|
+
// here). Ported from the 5 Oct front-voice test build, reduced to this job.
|
|
10
|
+
|
|
11
|
+
const { spawn } = require("child_process");
|
|
12
|
+
const { StringDecoder } = require("string_decoder");
|
|
13
|
+
const { killProcessTree } = require("../../../core/process-tree");
|
|
14
|
+
const { redactSensitive } = require("../../../core/redact");
|
|
15
|
+
|
|
16
|
+
const KILL_GRACE_MS = 2000;
|
|
17
|
+
const MAX_BUFFER_BYTES = 1024 * 1024;
|
|
18
|
+
const MAX_LINE_CHARS = 120;
|
|
19
|
+
const EFFORTS = new Set(["low", "medium", "high", "xhigh", "max"]);
|
|
20
|
+
|
|
21
|
+
// The bot's .env is parsed into the config object, not process.env: read the
|
|
22
|
+
// config first, the environment second.
|
|
23
|
+
function setting(name) {
|
|
24
|
+
let value = process.env[name];
|
|
25
|
+
try { value = require("../../../core/config").config?.[name] ?? value; } catch (_) {}
|
|
26
|
+
return String(value ?? "").trim();
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
// Measured 11 Oct, warm process, thinking off, 23 call lines with traps
|
|
30
|
+
// ("Did you hear me?", "Are you sure?"): Haiku 4.5 finished its line in
|
|
31
|
+
// 0.8-1.1 s (Sonnet 5 at low effort 1.7-1.9 s); both kept every rule once the
|
|
32
|
+
// prompt named the traps (a looser prompt let Haiku answer "you said it once").
|
|
33
|
+
function quickVoiceSettings() {
|
|
34
|
+
const effort = setting("KAZEE_CALL_QUICK_EFFORT").toLowerCase() || "low";
|
|
35
|
+
return {
|
|
36
|
+
enabled: /^(?:on|1|true|yes)$/i.test(setting("KAZEE_CALL_QUICK_VOICE")),
|
|
37
|
+
model: setting("KAZEE_CALL_QUICK_MODEL") || "claude-haiku-4-5",
|
|
38
|
+
effort: EFFORTS.has(effort) ? effort : "low",
|
|
39
|
+
thinking: /^(?:on|1|true|yes)$/i.test(setting("KAZEE_CALL_QUICK_THINKING")),
|
|
40
|
+
};
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
function ownerDisplayName() {
|
|
44
|
+
try {
|
|
45
|
+
const name = String(require("../../../core/people").owners()?.[0]?.name || "").trim();
|
|
46
|
+
return name && name.toLowerCase() !== "owner" ? name : "";
|
|
47
|
+
} catch (_) {
|
|
48
|
+
return "";
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
// Caller words and spoken history only: no file paths, links, ids or secrets
|
|
53
|
+
// reach the quick voice.
|
|
54
|
+
function scrubLine(text) {
|
|
55
|
+
return redactSensitive(String(text || ""))
|
|
56
|
+
.replace(/https?:\/\/\S+/gi, "a link")
|
|
57
|
+
.replace(/(?:~|\.{1,2})?\/[\w.@%+-]+(?:\/[\w.@%+-]*)+/g, "a file")
|
|
58
|
+
.replace(/\b(?=[A-Za-z0-9_-]*\d)[A-Za-z0-9_-]{16,}\b/g, "an id")
|
|
59
|
+
.replace(/\[REDACTED[^\]]*\]/g, "something private")
|
|
60
|
+
.replace(/[`*_#>|[\]]/g, " ")
|
|
61
|
+
.replace(/\s+/g, " ")
|
|
62
|
+
.trim();
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
function buildQuickPrompt({ ownerName = "" } = {}) {
|
|
66
|
+
const owner = scrubLine(ownerName).slice(0, 60) || "the caller";
|
|
67
|
+
return [
|
|
68
|
+
`You are the quick voice of Claudia, ${owner}'s assistant, on a live phone call with ${owner}.`,
|
|
69
|
+
"Claudia herself (the full assistant, with tools) answers every request a few seconds after you. Your only job is to fill the first moment, so the caller knows they were heard.",
|
|
70
|
+
"",
|
|
71
|
+
"Reply with ONE short spoken line, 2 to 8 words, that fits what was just said:",
|
|
72
|
+
"- a greeting or a check like \"Hello?\" or \"Are you there?\": a warm, brief reply, like \"Yep, I'm here.\"",
|
|
73
|
+
"- a question: \"Good question, one sec.\", or a line naming the topic, like \"Let me look at the budget.\"",
|
|
74
|
+
"- a request: \"Sure, let me check the call logs.\", naming the thing in two or three words.",
|
|
75
|
+
"- thanks or small talk: a short natural reply.",
|
|
76
|
+
"- a correction or \"that's wrong\": \"Okay, let me look again.\"",
|
|
77
|
+
"Never answer, even when you think you know: \"Did you hear me?\" gets \"Let me check.\", never \"Yes, once.\"; \"Are you sure?\" gets \"Let me double-check.\".",
|
|
78
|
+
"Never give facts, numbers, yes or no, opinions or apologies; never say anything was done, found, sent or changed. For sending, changing, booking or deleting anything, say \"Sure, one sec.\" and never that it is being done.",
|
|
79
|
+
"No lists, emoji, symbols or markdown.",
|
|
80
|
+
"",
|
|
81
|
+
"A message that starts with [Still working] means Claudia is still on it: say a short, varied line of 3 to 7 words, like \"Still on it, nearly there.\" Never repeat an earlier line word for word.",
|
|
82
|
+
].join("\n");
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
// What the quick voice should say for one utterance, or a filler while the
|
|
86
|
+
// full agent is still working.
|
|
87
|
+
function utteranceMessage(text) {
|
|
88
|
+
return scrubLine(text).slice(0, 600);
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
function fillerMessage(text, seconds) {
|
|
92
|
+
return `[Still working] on "${scrubLine(text).slice(0, 200)}", ${Math.round(seconds)} seconds in.`;
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
// A spoken line from model text: first line only, no control or markup,
|
|
96
|
+
// bounded. Empty means "say nothing".
|
|
97
|
+
function spokenLine(text) {
|
|
98
|
+
const line = String(text || "").split("\n").map((part) => part.trim()).find(Boolean) || "";
|
|
99
|
+
const clean = scrubLine(line).replace(/^["']|["']$/g, "").trim();
|
|
100
|
+
if (!clean || !/[A-Za-z]/.test(clean)) return "";
|
|
101
|
+
return clean.length > MAX_LINE_CHARS ? `${clean.slice(0, MAX_LINE_CHARS - 1)}…` : clean;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
class QuickVoiceProcess {
|
|
105
|
+
constructor({ provider, settings = quickVoiceSettings(), systemPrompt, cwd = process.cwd(), spawnImpl = spawn, maxRestarts = 1, log = () => {} } = {}) {
|
|
106
|
+
this.provider = provider;
|
|
107
|
+
this.settings = settings;
|
|
108
|
+
this.systemPrompt = systemPrompt;
|
|
109
|
+
this.cwd = cwd;
|
|
110
|
+
this.spawnImpl = spawnImpl;
|
|
111
|
+
this.maxRestarts = maxRestarts;
|
|
112
|
+
this.log = log;
|
|
113
|
+
this.proc = null;
|
|
114
|
+
this.restarts = 0;
|
|
115
|
+
this.stopped = false;
|
|
116
|
+
this.queue = []; // generations awaiting their result, oldest first
|
|
117
|
+
this._seq = 0;
|
|
118
|
+
this._stopPromise = null;
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
static supported(provider) {
|
|
122
|
+
return provider?.id === "claude" && provider.capabilities?.persistentUtilityStream?.support === "native" &&
|
|
123
|
+
typeof provider.buildUtilityInvocation === "function";
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
invocation() {
|
|
127
|
+
const { model, effort, thinking } = this.settings;
|
|
128
|
+
const invocation = this.provider.buildUtilityInvocation({
|
|
129
|
+
purpose: "call-quick-voice",
|
|
130
|
+
tier: "low",
|
|
131
|
+
model,
|
|
132
|
+
effort,
|
|
133
|
+
coreInstructions: this.systemPrompt,
|
|
134
|
+
dynamicContext: "",
|
|
135
|
+
userPrompt: "",
|
|
136
|
+
readOnly: true,
|
|
137
|
+
fresh: true,
|
|
138
|
+
});
|
|
139
|
+
const args = [...invocation.args];
|
|
140
|
+
// The slim prompt replaces the CLI's default system prompt (no tool docs
|
|
141
|
+
// to read), which is most of what makes a warm reply start in about a second.
|
|
142
|
+
const appendAt = args.indexOf("--append-system-prompt");
|
|
143
|
+
if (appendAt >= 0) args[appendAt] = "--system-prompt";
|
|
144
|
+
const printAt = args.indexOf("-p");
|
|
145
|
+
args.splice(printAt >= 0 ? printAt + 1 : 0, 0,
|
|
146
|
+
"--input-format", "stream-json", "--include-partial-messages",
|
|
147
|
+
"--tools", "", "--strict-mcp-config", "--setting-sources", "");
|
|
148
|
+
if (!args.includes("--no-session-persistence")) args.push("--no-session-persistence");
|
|
149
|
+
const env = { ...invocation.env, OC_PROVIDER: this.provider.id, OC_UTILITY_PURPOSE: "call-quick-voice" };
|
|
150
|
+
// Thinking off: Haiku 4.5 went from 2-3 s to 0.7 s to its first word.
|
|
151
|
+
if (!thinking) env.MAX_THINKING_TOKENS = "0";
|
|
152
|
+
return { binary: invocation.binary, args, env };
|
|
153
|
+
}
|
|
154
|
+
|
|
155
|
+
start() {
|
|
156
|
+
if (this.stopped || this.proc) return this.proc;
|
|
157
|
+
const { binary, args, env } = this.invocation();
|
|
158
|
+
const proc = this.spawnImpl(binary, args, {
|
|
159
|
+
cwd: this.cwd, env, stdio: ["pipe", "pipe", "pipe"], detached: process.platform !== "win32", shell: false,
|
|
160
|
+
});
|
|
161
|
+
this.proc = proc;
|
|
162
|
+
const decoder = new StringDecoder("utf8");
|
|
163
|
+
let buffer = "";
|
|
164
|
+
proc.stdout.on("data", (data) => {
|
|
165
|
+
if (this.proc !== proc) return;
|
|
166
|
+
buffer += decoder.write(data);
|
|
167
|
+
if (buffer.length > MAX_BUFFER_BYTES) { buffer = ""; this._crashed(proc, "buffer"); return; }
|
|
168
|
+
let newline;
|
|
169
|
+
while ((newline = buffer.indexOf("\n")) >= 0) {
|
|
170
|
+
const line = buffer.slice(0, newline).trim();
|
|
171
|
+
buffer = buffer.slice(newline + 1);
|
|
172
|
+
if (!line) continue;
|
|
173
|
+
let event;
|
|
174
|
+
try { event = JSON.parse(line); } catch (_) { continue; }
|
|
175
|
+
this._onEvent(event);
|
|
176
|
+
}
|
|
177
|
+
});
|
|
178
|
+
proc.stderr.on("data", () => {});
|
|
179
|
+
proc.stdin.on("error", () => {});
|
|
180
|
+
proc.on("error", () => this._crashed(proc, "spawn-error"));
|
|
181
|
+
proc.on("close", () => this._crashed(proc, "exited"));
|
|
182
|
+
this.log(`started pid=${proc.pid ?? "?"} model=${this.settings.model} effort=${this.settings.effort} thinking=${this.settings.thinking ? "on" : "off"}`);
|
|
183
|
+
return proc;
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
ready() {
|
|
187
|
+
const proc = this.proc;
|
|
188
|
+
return !this.stopped && !!proc && proc.exitCode === null && proc.signalCode === null && !proc.stdin.destroyed;
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
// One user message; resolves with the reply text as soon as its text block
|
|
192
|
+
// closes (before the CLI's end-of-turn work), or "" when cancelled/failed.
|
|
193
|
+
say(content) {
|
|
194
|
+
if (!this.ready()) return { generation: null, text: Promise.resolve("") };
|
|
195
|
+
let resolve;
|
|
196
|
+
const text = new Promise((done) => { resolve = done; });
|
|
197
|
+
const generation = { id: ++this._seq, text: "", cancelled: false, settled: false, resolve };
|
|
198
|
+
this.queue.push(generation);
|
|
199
|
+
try {
|
|
200
|
+
this.proc.stdin.write(`${JSON.stringify({ type: "user", message: { role: "user", content: String(content) } })}\n`);
|
|
201
|
+
} catch (_) {
|
|
202
|
+
this.queue = this.queue.filter((item) => item !== generation);
|
|
203
|
+
resolve("");
|
|
204
|
+
}
|
|
205
|
+
return { generation, text };
|
|
206
|
+
}
|
|
207
|
+
|
|
208
|
+
// Stops the in-flight generation (Claude Code's stream-json control
|
|
209
|
+
// protocol); the CLI then closes that turn with an error result.
|
|
210
|
+
cancel(generation) {
|
|
211
|
+
if (!generation || generation.cancelled) return;
|
|
212
|
+
generation.cancelled = true;
|
|
213
|
+
generation.resolve("");
|
|
214
|
+
if (generation.settled || this.queue[0] !== generation || !this.ready()) return;
|
|
215
|
+
try {
|
|
216
|
+
this.proc.stdin.write(`${JSON.stringify({ type: "control_request", request_id: `quick-interrupt-${generation.id}`, request: { subtype: "interrupt" } })}\n`);
|
|
217
|
+
} catch (_) {}
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
_onEvent(event) {
|
|
221
|
+
const generation = this.queue[0];
|
|
222
|
+
if (!generation) return;
|
|
223
|
+
if (event.type === "stream_event") {
|
|
224
|
+
const inner = event.event || {};
|
|
225
|
+
if (inner.type === "content_block_delta" && inner.delta?.type === "text_delta" && typeof inner.delta.text === "string") {
|
|
226
|
+
generation.text += inner.delta.text;
|
|
227
|
+
} else if (inner.type === "content_block_stop" && generation.text.trim()) {
|
|
228
|
+
generation.resolve(generation.cancelled ? "" : generation.text);
|
|
229
|
+
}
|
|
230
|
+
} else if (event.type === "assistant" && !generation.text && Array.isArray(event.message?.content)) {
|
|
231
|
+
// A CLI without partial messages: the whole reply at once.
|
|
232
|
+
generation.text = event.message.content.filter((block) => block?.type === "text").map((block) => block.text || "").join("");
|
|
233
|
+
} else if (event.type === "result") {
|
|
234
|
+
this.queue.shift();
|
|
235
|
+
generation.settled = true;
|
|
236
|
+
generation.resolve(generation.cancelled || event.is_error ? "" : generation.text);
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
_crashed(proc, reason) {
|
|
241
|
+
if (this.proc !== proc) return;
|
|
242
|
+
this.proc = null;
|
|
243
|
+
for (const generation of this.queue.splice(0)) { generation.settled = true; generation.resolve(""); }
|
|
244
|
+
try { killProcessTree(proc.pid, "SIGKILL"); } catch (_) {}
|
|
245
|
+
if (this.stopped) return;
|
|
246
|
+
if (this.restarts >= this.maxRestarts) {
|
|
247
|
+
this.log(`gave up reason=${reason} restarts=${this.restarts}`);
|
|
248
|
+
return;
|
|
249
|
+
}
|
|
250
|
+
this.restarts += 1;
|
|
251
|
+
this.log(`restart reason=${reason} restarts=${this.restarts}`);
|
|
252
|
+
try { this.start(); } catch (error) { this.log(`restart failed: ${error.message}`); }
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
stop() {
|
|
256
|
+
if (this._stopPromise) return this._stopPromise;
|
|
257
|
+
this.stopped = true;
|
|
258
|
+
live.delete(this);
|
|
259
|
+
const proc = this.proc;
|
|
260
|
+
this.proc = null;
|
|
261
|
+
for (const generation of this.queue.splice(0)) { generation.settled = true; generation.cancelled = true; generation.resolve(""); }
|
|
262
|
+
if (!proc) return (this._stopPromise = Promise.resolve());
|
|
263
|
+
this._stopPromise = new Promise((resolve) => {
|
|
264
|
+
let killTimer = null;
|
|
265
|
+
const finish = () => { if (killTimer) clearTimeout(killTimer); resolve(); };
|
|
266
|
+
if (proc.exitCode !== null || proc.signalCode !== null) return finish();
|
|
267
|
+
proc.once("close", finish);
|
|
268
|
+
try { proc.stdin.end(); } catch (_) {}
|
|
269
|
+
try { killProcessTree(proc.pid, "SIGTERM"); } catch (_) { try { proc.kill("SIGTERM"); } catch (_) {} }
|
|
270
|
+
killTimer = setTimeout(() => {
|
|
271
|
+
try { killProcessTree(proc.pid, "SIGKILL"); } catch (_) { try { proc.kill("SIGKILL"); } catch (_) {} }
|
|
272
|
+
}, KILL_GRACE_MS);
|
|
273
|
+
killTimer.unref?.();
|
|
274
|
+
});
|
|
275
|
+
return this._stopPromise;
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
|
|
279
|
+
// A bot exit never leaves a quick-voice process behind.
|
|
280
|
+
const live = new Set();
|
|
281
|
+
function track(quick) {
|
|
282
|
+
live.add(quick);
|
|
283
|
+
return quick;
|
|
284
|
+
}
|
|
285
|
+
process.on("exit", () => {
|
|
286
|
+
for (const quick of live) {
|
|
287
|
+
const pid = quick.proc?.pid;
|
|
288
|
+
if (pid) { try { killProcessTree(pid, "SIGKILL"); } catch (_) {} }
|
|
289
|
+
}
|
|
290
|
+
});
|
|
291
|
+
|
|
292
|
+
module.exports = {
|
|
293
|
+
QuickVoiceProcess, quickVoiceSettings, buildQuickPrompt, utteranceMessage, fillerMessage, spokenLine,
|
|
294
|
+
scrubLine, ownerDisplayName, track,
|
|
295
|
+
};
|
|
@@ -54,7 +54,7 @@ const ACK_DELAY_MS = 2800;
|
|
|
54
54
|
class KazeeCallSession {
|
|
55
55
|
constructor({ roomId, chatId, callerId, botUserId, janusUrl, janusToken, janusTokenExpiresAt, roomToken, roomTokenExpiresAt, managedRoom = false, refreshRoomAccess, iceServers = [], iceCredentialsExpiresAt, onTranscript, onEnded, onMediaReady,
|
|
56
56
|
maxCallMs, idleMs, finalWaitMs, synthesize = synthesizePcm, synthesizeStream = null, asCaller = false, greeting = null,
|
|
57
|
-
turnDecision = detectTurn, acknowledgement = selectAcknowledgement, onSpeechStart,
|
|
57
|
+
turnDecision = detectTurn, acknowledgement = selectAcknowledgement, onSpeechStart, onSpeechNoise,
|
|
58
58
|
JanusClientImpl = JanusClient, PeerConnectionImpl = RTCPeerConnection, SpeechStreamImpl = SpeechStream } = {}) {
|
|
59
59
|
this.roomId = Number(roomId);
|
|
60
60
|
this.chatId = String(chatId);
|
|
@@ -83,6 +83,7 @@ class KazeeCallSession {
|
|
|
83
83
|
this._transportExpiryTimer = null;
|
|
84
84
|
this.onTranscript = onTranscript || (() => {});
|
|
85
85
|
this.onSpeechStart = onSpeechStart || (() => {});
|
|
86
|
+
this.onSpeechNoise = onSpeechNoise || (() => {});
|
|
86
87
|
this.turnDecision = turnDecision;
|
|
87
88
|
this.acknowledgement = acknowledgement;
|
|
88
89
|
this._cueController = null;
|
|
@@ -585,6 +586,7 @@ class KazeeCallSession {
|
|
|
585
586
|
if (countWords(trimmed) < MIN_DISPATCH_WORDS) {
|
|
586
587
|
clog(`Kazee call noise turn dropped: feed=${feedId} words=${countWords(trimmed)}`);
|
|
587
588
|
if (!this._speakActive && !this._speakQueue.length) this._stopThinking();
|
|
589
|
+
this._speechWasNoise(feedId);
|
|
588
590
|
return;
|
|
589
591
|
}
|
|
590
592
|
}
|
|
@@ -593,7 +595,7 @@ class KazeeCallSession {
|
|
|
593
595
|
// Empty finalization (noise commit): nothing dispatches, no reply is
|
|
594
596
|
// coming — don't tick into the void.
|
|
595
597
|
else if (!this._speakActive && !this._speakQueue.length) this._stopThinking();
|
|
596
|
-
if (!trimmed) return;
|
|
598
|
+
if (!trimmed) { this._speechWasNoise(feedId); return; }
|
|
597
599
|
this.onTranscript(text, {
|
|
598
600
|
feedId,
|
|
599
601
|
display: sub?.display || "",
|
|
@@ -602,6 +604,16 @@ class KazeeCallSession {
|
|
|
602
604
|
});
|
|
603
605
|
}
|
|
604
606
|
|
|
607
|
+
// Speech start muted the turn being worked on, in case it was a new ask. A
|
|
608
|
+
// sound the recogniser found no words in (wind, footsteps, a cough) was not:
|
|
609
|
+
// unless someone is still talking or another segment is pending, the
|
|
610
|
+
// muted turn gets its voice back.
|
|
611
|
+
_speechWasNoise(feedId) {
|
|
612
|
+
if ([...this.subscribers.values()].some(peer => peer?.vad?.active || peer?.turnGate?.pending ||
|
|
613
|
+
peer?.awaitingWordsSince || peer?.segments?.size)) return;
|
|
614
|
+
this.onSpeechNoise({ feedId });
|
|
615
|
+
}
|
|
616
|
+
|
|
605
617
|
_reportMediaReady() {
|
|
606
618
|
if (this.closed || this._reportedMediaReady || !this._sentRtp || !this._receivedRtp) return;
|
|
607
619
|
this._reportedMediaReady = true;
|
|
@@ -751,6 +763,16 @@ class KazeeCallSession {
|
|
|
751
763
|
sub.stt.audio(pcm16).catch((error) => console.error("Kazee call STT send:", error.message));
|
|
752
764
|
}
|
|
753
765
|
|
|
766
|
+
// For the quick voice: someone is talking (or a turn is being decided), and
|
|
767
|
+
// speech is playing or queued.
|
|
768
|
+
callerTalking() {
|
|
769
|
+
return [...this.subscribers.values()].some(sub => sub?.vad?.active || sub?.turnGate?.pending || sub?.awaitingWordsSince);
|
|
770
|
+
}
|
|
771
|
+
|
|
772
|
+
speechBusy() {
|
|
773
|
+
return !!(this.speaking || this._speakActive || this._speakQueue.length);
|
|
774
|
+
}
|
|
775
|
+
|
|
754
776
|
speak(text) {
|
|
755
777
|
if (this.closed || !String(text || "").trim()) return Promise.resolve(false);
|
|
756
778
|
return new Promise((resolve, reject) => {
|