@inetafrica/open-claudia 3.5.27 → 3.5.28

package/CHANGELOG.md

@@ -1,13 +1,30 @@
 # Changelog
 
-## 3.5.27
+## 3.5.28
 
-Calls answer in about two seconds again, and say something while they work (Sumeet, 11 Oct 08:24 EAT on a walk, after a call where "hello" took 8.5 s and a 79 s lookup was never heard: "Yeah, let's give it a shot"). A canary build for the owner's own bot first. It builds on 3.5.24; the Browser canaries 3.5.25 and 3.5.26 are not in it yet.
+The Browser app is always on: opening it while the bot is not browsing opens the bot's browser, instead of a dead end (Sumeet, 10 Oct: "half the time i have to ask you switch on the browser"; 11 Oct 07:43 EAT network-mon's Browser showed only "isn't open right now"). A canary build for the Browser canary bot, on 3.5.26; it does not carry 3.5.27 (calls, Sumeet's own bot).
 
-- **A quick voice says a fitting line right away** (`channels/kazee/call/quick-voice.js`). A kept-warm small model, one process per call through the provider's own utility invocation, hears each utterance and says one short line ("Sure, let me check the call logs.", "Yep, I'm here.") within about a second, then a brief "still on it" line whenever the full agent leaves the caller in 8 s of silence, at most 3 per turn. It never answers: every utterance still goes to the full agent. The line is dropped if it is not ready in 3.5 s, if the full agent already spoke, if the caller is talking, or once a newer utterance took over. Off unless `KAZEE_CALL_QUICK_VOICE=on`; `KAZEE_CALL_QUICK_MODEL` (default `claude-haiku-4-5`), `_EFFORT` (`low`) and `_THINKING` (off) tune it. Measured on 23 call lines with traps ("Did you hear me?", "Are you sure?"): Haiku 4.5 finished its line in 0.8-1.1 s with thinking off (2-3 s with it on), Sonnet 5 in 1.7-1.9 s, and both kept every rule.
-- **The full agent speaks first, then works.** The live-call guidance used to forbid "speculative progress", so a lookup went silent until it ended. Now the first action is a 3-12 word call-say ("Sure, let me check the call logs."), then a progress line about every ten seconds. Replayed on the four turns of the 11 Oct call, 3 times per effort level: the holding line came first 3 of 3 times at max, xhigh, high and medium (the old guidance: 1 of 3 at xhigh), simple replies unchanged. With the quick voice on, the agent skips its own acknowledgement and starts with substance.
-- **A sound with no words no longer silences the answer.** Speech start mutes the running call turn in case it is a new ask; when the recogniser found no words (wind, footsteps, a cough) the turn stayed muted for good, its call-say got 409 and its final answer was never spoken. The session now reports a noise-only turn and the turn gets its voice back, unless someone is still talking or another segment is pending.
-- **Tests:** `test-call-quick-voice.js` (new, in `npm test`), `test-kazee-call-manager.js`, `test-kazee-call-session.js`, `test-provider-prompt-parity.js`.
+- **`POST /api/browser/start`** (Agent Space's control token only, like every Browser app route). With no browser open, the bot's own default session opens on a start page (`https://www.google.com/`), with the bot's own agent-browser environment and socket folder (`AGENT_BROWSER_SOCKET_DIR`, else `~/.agent-browser`), exactly as its own `agent-browser open` would, so the bot carries on in that tab. A browser already open is answered as it is (the newest session), and two people opening the app at once start one browser. `GET /api/browser/control` says `start: true`; Agent Space's front door (223b978) asks only bots that say so.
+- **Tests:** `test-browser-control.js`, `test-web-browser-app.js`.
+
+## 3.5.26
+
+The Browser app's phone and web view stays sharp after a page load. A canary build for the Browser canary bot first; the fleet waits for Sumeet's OK.
+
+- **Sharp after every page.** On the canary's Chrome 150 (Linux) the driver's pictures dropped from the view's density to 1x at the first page load (the view kept its density; the picture did not). After a main-frame navigation or a load, the window is now resized by a pixel and back, which brings the density back: 906x1760 pictures for a 412x800 phone at 2.2x on duckduckgo.com, en.m.wikipedia.org and example.com, 1 to 4 s after each load.
+- **Tests:** `test-browser-drive.js`.
+
+## 3.5.25
+
+The Browser app feels like a browser on the phone and on the web (Sumeet, 10 Oct 08:53 EAT on his phone: "it feels like a browser on my phone… like Google Chrome app", and 09:18: "the web and mobile still need to also feel native… don't forget the website"). A canary build for the Browser canary bot first; the fleet waits for his OK.
+
+- **The bot's tab takes the driver's screen.** With Chrome in its own container, a take (or a `fit` when the screen changes: a turned phone, a resized panel, a keyboard that opens) that carries the driver's screen (`viewport`: CSS size, pixel density, phone layout, touch) fits the bot's tab to it through `core/browser-drive.js`. A phone gets the page's phone layout, full screen and sharp; the web gets a desktop page at the panel's size. Handing back puts the tab back to its own size. The browser's identity (user agent) never changes, so sites signed into stay signed in after the hand back. `GET /api/browser/control` says `fit: true`.
+- **The driver's input goes straight to Chrome,** through the view's own DevTools session, in order. Touches keep their timing, so a swipe flings; typed text is inserted whole (`input_text`, at most 500 characters: accents, emoji, the keyboard's corrections); every point is scaled by the view's density, because Chrome reads input in the drawn picture's pixels. A bot without the view takes `input_text` as key presses.
+- **The page's text fields.** The view reads where the page's text fields are and what kind they are (never a value) and streams them as `fields` events, so the Browser page opens the phone's own keyboard of the right kind the moment a field is tapped.
+- **Pictures.** The view streams its own screencast at the driver's size (at most 1.6 million pixels, 30 a second). `/api/browser/stream` serves it while someone drives, up to 30 fps, and keeps the newest frame while a slow viewer drains instead of dropping it, so the last picture of a scroll always arrives.
+- **A hand back wakes the bot only when it waits for that person:** it sent them an Open Browser card (cards now record `openApp`), or one of its browser commands was refused while they drove (the tool guard marks it). Otherwise the hand back is only logged. On 9 Oct a test hand back set off a 28-minute tangent on the canary.
+- **Correction to 3.5.24:** in container mode, sessions are not pinned to tabs (Agent Space dropped `AGENT_BROWSER_PIN_TAB` in a98f239); a session uses the tab that is open.
+- **Tests:** `test-browser-drive.js` (new), `test-browser-control.js`, `test-web-browser-app.js`.
 
 ## 3.5.24
 

package/channels/kazee/call/manager.js

@@ -5,7 +5,6 @@ const { CallLeaseHeartbeat } = require("./lease");
 const { publicCallIdentity, requestRoomAccess, supportedIceServers } = require("./access");
 const { canonicalForChannel } = require("../../../core/identity");
 const { KazeeCallSession } = require("./session");
-const quickVoice = require("./quick-voice");
 
 const DEFAULT_JANUS_URL = "wss://janus.kazee.africa/ws";
 const DEFAULT_ICE_SERVERS = [
@@ -15,34 +14,6 @@ const DEFAULT_ICE_SERVERS = [
 // FCM push TTL is 60s and chat-central treats ringing logs as fresh for 90s.
 // This is the outbound invitation deadline; terminal server events end it early.
 const RING_TIMEOUT_MS = 60_000;
-const qlog = (msg) => console.log(`${new Date().toISOString()} Kazee call quick: ${msg}`);
-// The quick line must be ready this soon after the words, or it is skipped:
-// by then the full agent is close to speaking itself.
-const QUICK_LINE_MAX_MS = 3500;
-// While the full agent works: a "still on it" line after this much quiet,
-// at most a few per turn, never after the answer.
-const QUICK_FILLER_QUIET_MS = 8000;
-const QUICK_FILLER_MAX = 3;
-const QUICK_TICK_MS = 500;
-
-// The provider the call's chat runs on: the quick voice uses its CLI and
-// credentials, as the full agent's turns do.
-function chatProvider({ adapter, chatId, userId }) {
-  const { runInChat } = require("../../../core/context");
-  const { conversationKeyFor, roomKeyFor } = require("../../../core/identity");
-  const transport = adapter?.type || "kazee";
-  const id = runInChat({
-    adapter, channelId: String(chatId), userId: String(userId), transport,
-    canonicalUserId: canonicalForChannel(transport, String(userId)),
-    conversationKey: conversationKeyFor(transport, String(chatId)),
-    roomKey: roomKeyFor(transport, String(chatId)),
-  }, () => {
-    const state = require("../../../core/state");
-    return state.getActiveProvider(state.currentState());
-  });
-  return id ? require("../../../core/providers").registry.getProvider(id) : null;
-}
-
 const OUTGOING_GREETING = "Hey, it's Claudia. Can you hear me okay?";
 // Inbound must greet too: callers get silence until they speak first, which
 // reads as a dead call (live test 2026-08-08 — media was up, caller hung up).
@@ -50,8 +21,7 @@ const INCOMING_GREETING = "Hey, it's Claudia — you're through. Can you hear me
 
 class KazeeCallManager {
   constructor({ adapter, SessionImpl = KazeeCallSession, janusUrl = process.env.KAZEE_JANUS_URL || DEFAULT_JANUS_URL,
-    iceServers = DEFAULT_ICE_SERVERS, deviceId = randomUUID(), resolveQuickProvider = chatProvider, quickSpawn = undefined,
-    quickSettings = quickVoice.quickVoiceSettings, quickTiming = {} } = {}) {
+    iceServers = DEFAULT_ICE_SERVERS, deviceId = randomUUID() } = {}) {
     this.adapter = adapter;
     this.SessionImpl = SessionImpl;
     this.janusUrl = janusUrl;
@@ -60,10 +30,6 @@ class KazeeCallManager {
     this.deviceId = deviceId;
     this._pending = null;
     this._ringTimer = null;
-    this.resolveQuickProvider = resolveQuickProvider;
-    this.quickSpawn = quickSpawn;
-    this.quickSettings = quickSettings;
-    this.quickTiming = { lineMaxMs: QUICK_LINE_MAX_MS, fillerQuietMs: QUICK_FILLER_QUIET_MS, fillerMax: QUICK_FILLER_MAX, tickMs: QUICK_TICK_MS, ...quickTiming };
   }
 
   wire(socket) {
@@ -123,7 +89,6 @@ class KazeeCallManager {
     // Several terminal paths clear this.session before close() calls back.
     // Invalidate the closed call's work even when it no longer owns the slot.
     session?._voiceTurn?.controller.abort("call-ended");
-    this._stopQuickVoice(session);
     if (this.session !== session) return;
     this._detachSession(session);
     if (this.adapter._socket?.connected) this.adapter._socket.emit("call:end", {
@@ -133,7 +98,6 @@ class KazeeCallManager {
 
   _detachSession(session) {
     session?._voiceTurn?.controller.abort("call-ended");
-    this._stopQuickVoice(session);
     if (this.session === session) this.session = null;
   }
 
@@ -187,14 +151,12 @@ class KazeeCallManager {
         ...this._sessionOptions(config, access, boundIdentity, pending.controller.signal), greeting: INCOMING_GREETING,
         onTranscript: (text, meta) => this._transcript(text, { ...payload, ...boundIdentity }, meta, session),
         onSpeechStart: () => this._speechStart(session),
-        onSpeechNoise: () => this._speechNoise(session),
         onMediaReady: () => this._mediaReady(session),
         onEnded: reason => { session?._leaseHeartbeat?.close(); pending.controller.abort(); this._sessionEnded(session, reason); },
       });
       session.callIdentity = boundIdentity;
       session.chatId = chatId;
       this.session = session;
-      this._startQuickVoice(session, chatId, callerId);
       this._manageLease(session, access);
       await session.start();
       if (this.session !== session || session.closed || pending.controller.signal.aborted) return;
@@ -236,14 +198,12 @@ class KazeeCallManager {
         ...this._sessionOptions(config, access, identity, pending.controller.signal), asCaller: true, greeting: OUTGOING_GREETING,
         onTranscript: (text, meta) => this._transcript(text, remote, meta, session),
         onSpeechStart: () => this._speechStart(session),
-        onSpeechNoise: () => this._speechNoise(session),
         onMediaReady: () => this._mediaReady(session),
         onEnded: reason => { session?._leaseHeartbeat?.close(); pending.controller.abort(); this._clearRingTimer(); this._sessionEnded(session, reason); },
       });
       session.callIdentity = identity;
       session.chatId = chat;
       this.session = session;
-      this._startQuickVoice(session, chat, target);
       this._manageLease(session, access);
       await session.start();
       if (this.session !== session || session.closed || pending.controller.signal.aborted) throw new Error("Call cancelled");
@@ -287,96 +247,6 @@ class KazeeCallManager {
     // complete transcript exists; a cough must not discard an accepted ask.
   }
 
-  // The sound that muted the turn had no words in it: not a new request, so
-  // the answer being worked on may speak again (11 Oct: wind while walking
-  // muted a lookup's answer for good, and the caller heard nothing).
-  _speechNoise(session) {
-    const turn = session?._voiceTurn;
-    if (!turn?.muted || turn.controller.signal.aborted || this.session !== session || session.closed) return;
-    turn.muted = false;
-    console.log(`${new Date().toISOString()} Kazee call noise: muted turn resumed`);
-  }
-
-    // ── Quick voice ───────────────────────────────────────────────────
-
-  _startQuickVoice(session, chatId, userId) {
-    let settings;
-    try { settings = this.quickSettings(); } catch (_) { return; }
-    if (!settings?.enabled) return;
-    let provider = null;
-    try { provider = this.resolveQuickProvider({ adapter: this.adapter, chatId, userId }); } catch (_) {}
-    if (!quickVoice.QuickVoiceProcess.supported(provider)) {
-      qlog(`off: provider ${provider?.id || "none"} has no warm stream`);
-      return;
-    }
-    const quick = quickVoice.track(new quickVoice.QuickVoiceProcess({
-      provider, settings, spawnImpl: this.quickSpawn, log: qlog,
-      systemPrompt: quickVoice.buildQuickPrompt({ ownerName: quickVoice.ownerDisplayName() }),
-    }));
-    session._quick = quick;
-    // The quick voice says the first line itself; the session's delayed cue
-    // would only double it.
-    session.acknowledgement = async () => null;
-    try { quick.start(); } catch (error) {
-      qlog(`start failed: ${error.message}`);
-      session._quick = null;
-      quick.stop().catch?.(() => {});
-    }
-  }
-
-  _stopQuickVoice(session) {
-    const quick = session?._quick;
-    if (!quick) return;
-    session._quick = null;
-    quick.stop().catch?.(() => {});
-  }
-
-  // One utterance: a short fitting line as soon as the quick voice has it,
-  // then a "still on it" line whenever the full agent leaves the caller in
-  // silence, until its answer arrives. Never over the caller, never after
-  // the answer, never once a newer utterance took over.
-  _quickTurn(session, turn, text) {
-    const quick = session._quick;
-    const timing = this.quickTiming;
-    const live = () => !turn.muted && !turn.done && !turn.controller.signal.aborted &&
-      !session.closed && this.session === session && session._voiceTurn === turn && session._quick === quick;
-    const sayLine = async (message, label, { maxMs = Infinity, unlessSpoke = false } = {}) => {
-      const asked = Date.now();
-      const { generation, text: reply } = quick.say(message);
-      const cancel = () => quick.cancel(generation);
-      turn.controller.signal.addEventListener("abort", cancel, { once: true });
-      let line = "";
-      try { line = quickVoice.spokenLine(await reply); }
-      finally { turn.controller.signal.removeEventListener("abort", cancel); }
-      const ready = Date.now() - asked;
-      if (!line || !live() || ready > maxMs || (unlessSpoke && turn.spoke) || session.callerTalking?.()) {
-        qlog(`${label} skipped ready=${ready}ms`);
-        return false;
-      }
-      qlog(`${label} ready=${ready}ms chars=${line.length}`);
-      const spoken = await session.speak(line);
-      if (spoken === false || !live()) return false;
-      this._rememberSpeech(session, "assistant", line);
-      return true;
-    };
-    (async () => {
-      await sayLine(quickVoice.utteranceMessage(text), "line", { maxMs: timing.lineMaxMs, unlessSpoke: true });
-      let fillers = 0;
-      let lastFillerAt = Date.now();
-      while (fillers < timing.fillerMax && live()) {
-        await new Promise((resolve) => { const timer = setTimeout(resolve, timing.tickMs); timer.unref?.(); });
-        if (!live()) break;
-        if (session.speechBusy?.() || session.callerTalking?.()) continue;
-        const quietSince = Math.max(turn.startedAt, lastFillerAt, session._lastSpokeAt || 0);
-        if (Date.now() - quietSince < timing.fillerQuietMs) continue;
-        fillers += 1;
-        lastFillerAt = Date.now();
-        await sayLine(quickVoice.fillerMessage(text, (Date.now() - turn.startedAt) / 1000), `filler ${fillers}`);
-        lastFillerAt = Date.now();
-      }
-    })().catch((error) => qlog(`turn failed: ${error.message}`));
-  }
-
   _rememberSpeech(session, role, text) {
     const history = session._voiceHistory ||= [];
     if (role === "assistant" && history.at(-1)?.role === "assistant") {
@@ -392,7 +262,7 @@ class KazeeCallManager {
     const speaker = String(meta.display || "").trim();
     const attributed = (meta.participants || 0) > 1 && speaker ? `${speaker}: ${text}` : text;
     session._voiceTurn?.controller.abort("superseded");
-    const turn = { controller: new AbortController(), muted: false, done: false, spoke: false, startedAt: Date.now() };
+    const turn = { controller: new AbortController(), muted: false };
     session._voiceTurn = turn;
     const callAdapter = Object.create(this.adapter);
     callAdapter.callHistory = (session._voiceHistory || []).map(entry => ({ ...entry }));
@@ -407,8 +277,6 @@ class KazeeCallManager {
     callAdapter.suppressChatVisibility = true;
     // Prompt builder switches to phone-call brevity when it sees this.
     callAdapter.liveCall = true;
-    // The quick voice says the first line; the prompt then skips its own.
-    callAdapter.quickVoice = !!session._quick?.ready();
     const spokenSentences = [];
     callAdapter.isCallTurnActive = () => !turn.muted && !turn.controller.signal.aborted &&
       !session.closed && this.session === session && session._voiceTurn === turn;
@@ -419,7 +287,6 @@ class KazeeCallManager {
       if (!callAdapter.isCallTurnActive()) return false;
       const spoken = await session.speak(response);
       if (spoken !== false && callAdapter.isCallTurnActive()) {
-        turn.spoke = true;
         this._rememberSpeech(session, "assistant", response);
         return true;
       }
@@ -431,7 +298,6 @@ class KazeeCallManager {
       return spoken;
     };
     callAdapter.send = async (_channelId, response) => {
-      turn.done = true;
       const remainder = require("../../../core/call-speech").unspokenSuffix(response, spokenSentences);
       if (remainder) await speak(remainder);
       return true;
@@ -442,7 +308,6 @@ class KazeeCallManager {
       console.warn(`Kazee call ignored file voice response ${voicePath}; streamed TTS is active`);
       return true;
     };
-    if (callAdapter.quickVoice) this._quickTurn(session, turn, attributed);
     this.adapter._emit("message", {
       adapter: callAdapter,
       channelId: String(incoming.chatId),

package/channels/kazee/call/session.js

@@ -54,7 +54,7 @@ const ACK_DELAY_MS = 2800;
 class KazeeCallSession {
   constructor({ roomId, chatId, callerId, botUserId, janusUrl, janusToken, janusTokenExpiresAt, roomToken, roomTokenExpiresAt, managedRoom = false, refreshRoomAccess, iceServers = [], iceCredentialsExpiresAt, onTranscript, onEnded, onMediaReady,
     maxCallMs, idleMs, finalWaitMs, synthesize = synthesizePcm, synthesizeStream = null, asCaller = false, greeting = null,
-    turnDecision = detectTurn, acknowledgement = selectAcknowledgement, onSpeechStart, onSpeechNoise,
+    turnDecision = detectTurn, acknowledgement = selectAcknowledgement, onSpeechStart,
     JanusClientImpl = JanusClient, PeerConnectionImpl = RTCPeerConnection, SpeechStreamImpl = SpeechStream } = {}) {
     this.roomId = Number(roomId);
     this.chatId = String(chatId);
@@ -83,7 +83,6 @@ class KazeeCallSession {
     this._transportExpiryTimer = null;
     this.onTranscript = onTranscript || (() => {});
     this.onSpeechStart = onSpeechStart || (() => {});
-    this.onSpeechNoise = onSpeechNoise || (() => {});
     this.turnDecision = turnDecision;
     this.acknowledgement = acknowledgement;
     this._cueController = null;
@@ -586,7 +585,6 @@ class KazeeCallSession {
       if (countWords(trimmed) < MIN_DISPATCH_WORDS) {
         clog(`Kazee call noise turn dropped: feed=${feedId} words=${countWords(trimmed)}`);
         if (!this._speakActive && !this._speakQueue.length) this._stopThinking();
-        this._speechWasNoise(feedId);
         return;
       }
     }
@@ -595,7 +593,7 @@ class KazeeCallSession {
     // Empty finalization (noise commit): nothing dispatches, no reply is
     // coming — don't tick into the void.
     else if (!this._speakActive && !this._speakQueue.length) this._stopThinking();
-    if (!trimmed) { this._speechWasNoise(feedId); return; }
+    if (!trimmed) return;
     this.onTranscript(text, {
       feedId,
       display: sub?.display || "",
@@ -604,16 +602,6 @@ class KazeeCallSession {
     });
   }
 
-  // Speech start muted the turn being worked on, in case it was a new ask. A
-  // sound the recogniser found no words in (wind, footsteps, a cough) was not:
-  // unless someone is still talking or another segment is pending, the
-  // muted turn gets its voice back.
-  _speechWasNoise(feedId) {
-    if ([...this.subscribers.values()].some(peer => peer?.vad?.active || peer?.turnGate?.pending ||
-        peer?.awaitingWordsSince || peer?.segments?.size)) return;
-    this.onSpeechNoise({ feedId });
-  }
-
   _reportMediaReady() {
     if (this.closed || this._reportedMediaReady || !this._sentRtp || !this._receivedRtp) return;
     this._reportedMediaReady = true;
@@ -763,16 +751,6 @@ class KazeeCallSession {
     sub.stt.audio(pcm16).catch((error) => console.error("Kazee call STT send:", error.message));
   }
 
-  // For the quick voice: someone is talking (or a turn is being decided), and
-  // speech is playing or queued.
-  callerTalking() {
-    return [...this.subscribers.values()].some(sub => sub?.vad?.active || sub?.turnGate?.pending || sub?.awaitingWordsSince);
-  }
-
-  speechBusy() {
-    return !!(this.speaking || this._speakActive || this._speakQueue.length);
-  }
-
   speak(text) {
     if (this.closed || !String(text || "").trim()) return Promise.resolve(false);
     return new Promise((resolve, reject) => {