@bojackduy/opencode-voice 0.8.0 → 0.9.0

package/README.md

@@ -419,36 +419,47 @@ then `s`.
 
 ### Voice conversation
 
-| Command                    | Keybind    | Description                               |
-| -------------------------- | ---------- | ----------------------------------------- |
-| `/voice-conversation`      | `leader+v` | Toggle hands-free voice conversation mode |
-| `/voice-conversation-stop` |            | Exit voice conversation mode              |
+| Command                    | Keybind    | Description                                 |
+| -------------------------- | ---------- | ------------------------------------------- |
+| `/voice-conversation`      | `leader+v` | Toggle push-to-talk voice conversation mode |
+| `/voice-conversation-stop` |            | Exit voice conversation mode                |
 
-One key drives the whole loop - its meaning follows the toast on screen:
+One key drives the whole loop. It always means the same thing - "I want the
+floor" - and what it does follows the toast on screen:
 
 ```
-record -> transcribe -> normalize -> submit -> wait reply -> speak -> record ...
+press -> record -> press -> transcribe -> normalize -> submit -> speak -> press ...
 ```
 
-| Toast shows               | Pressing the key does                |
-| ------------------------- | ------------------------------------ |
-| ● Recording               | Finish the turn and submit           |
-| Speaking...               | Pause speech (press again to record) |
-| Paused                    | Record again                         |
-| Waiting / Transcribing... | Exit the mode                        |
-
-Empty or failed turns pause instead of re-recording, so the key never
-surprises. Saying only a stop phrase (`stop`, `stop stop`, `dừng lại đi`,
-...) ends the mode without submitting - full sentences mentioning stop still
-submit normally. `/tts-stop` pauses a speaking reply; `/voice-conversation-stop`
-exits from anywhere. While the mode is on, the plain `/stt-record` keys act
-as the conversation key and auto TTS stays silent (the loop speaks the reply
-itself).
+| Toast shows              | Pressing the key does                     |
+| ------------------------ | ----------------------------------------- |
+| Press `leader+v` to talk | Open the mic (this is the resting state)  |
+| ● Recording              | Finish the turn and submit                |
+| Working... / Speaking    | Barge in: cut audio, drop reply, open mic |
+| Answer on screen         | Same barge-in, while the agent resumes    |
+| Transcribing...          | Report busy - too short to interrupt      |
+
+The loop never reopens the mic by itself: after a reply is spoken the mode rests
+paused, so it cannot record the room while nobody is talking. Barge-in drops
+the pending reply on purpose - pressing means talking now, not hearing the
+rest.
+
+Answering a permission or a question is the one thing the key cannot do,
+because that answer is given on screen. The mode announces the gate, keeps the
+mic shut, then speaks the agent's answer once the agent resumes.
+
+Exit paths: a stop phrase (`stop`, `stop stop`, `dừng lại đi`, ...),
+`/voice-conversation-stop`, or the global `/voice-cancel` (`leader+.`). Only a
+bare stop phrase ends the mode - full sentences mentioning stop submit
+normally. Empty or failed turns pause instead of re-recording, so the key never
+surprises. `/tts-stop` pauses a speaking reply and rests paused. While the mode
+is on, the plain `/stt-record` keys act as the conversation key and auto TTS
+stays silent (the loop speaks the reply itself).
 
 While waiting, assistant text is spoken sentence by sentence as it streams in
 (local cleanup, no LLM), so the answer starts before the agent turn finishes.
 Code-like sentences are skipped; replies with nothing streamable fall back to
-the full narrated speak. Pressing the key mid-stream exits the mode.
+the full narrated speak.
 
 - `ttsNormalizeMode` _(optional)_ - `"llm"` (default, polished narration) or
   `"local"` (instant deterministic cleanup, no network). Streaming speech
@@ -667,6 +678,10 @@ live-mic session was measured.
    code-heavy responses, or briefly notify for confirmations
 3. Piper synthesizes speech locally, piped through sox for playback
 
+The LLM is only a polish layer: when the configured model is unavailable or out
+of quota, TTS speaks the local cleanup automatically instead of going silent.
+Set `ttsNormalizeMode: "local"` to skip narration entirely.
+
 ### Auto TTS
 
 When enabled (`/tts-mode`), the plugin automatically speaks:

package/lib/conversation.js

@@ -1,18 +1,27 @@
-// Voice conversation mode: hands-free talk loop with OpenCode.
+// Voice conversation mode: push-to-talk turns with barge-in.
 //
-// One key drives the whole loop; its meaning depends on the state shown in
+// One key drives the whole loop and its meaning is always the same: the user
+// wants the floor. What the key actually does depends on the state shown in
 // the toast:
 //
-//   record -> transcribe -> normalize -> submit -> wait reply -> speak -> record ...
+//   press -> record -> press -> transcribe -> normalize -> submit
+//         -> wait reply -> speak -> press -> record ...
 //
+// - paused + key: open the mic (the resting state is the user's turn)
 // - recording + key: finish the turn and submit
-// - speaking + key: pause speech (press again to record the next turn)
-// - paused + key: record again
-// - waiting/processing + key: exit the mode
+// - speaking/waiting + key: barge in - cut the audio, drop the pending reply,
+//   open the mic (one press, never a stop-then-record dance)
+// - processing + key: transcription is too short to interrupt, report busy
 //
-// Saying a stop phrase ("stop", "dừng lại", ...) ends the mode without
-// submitting. Empty or failed turns pause instead of auto-recording, so the
-// key never surprises.
+// The loop never reopens the mic by itself: after a reply is spoken the mode
+// rests paused, so it can never record the room while nobody is talking. That
+// auto-record is also why a question/permission gate used to eat the answer -
+// the mic opened, the user's muttering was submitted, and the real reply was
+// already torn down. Gates now announce and wait instead.
+//
+// Exit paths: a stop phrase ("stop", "dừng lại", ...), /voice-conversation-stop,
+// or the global /voice-cancel. Empty or failed turns pause instead of
+// auto-recording, so the key never surprises.
 //
 // Latency: while waiting, assistant text deltas are spoken sentence by
 // sentence (local cleanup, no LLM), so the answer starts before the turn
@@ -151,7 +160,34 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
       phase = "idle";
       stop("busy");
       toast("STT busy, conversation off", "warning");
+      return;
     }
+    toast(`Listening - press ${keyLabel()} to send`);
+  }
+
+  // The resting state: mic closed, one press to talk. Entered by start() and
+  // after a reply is spoken, never by the loop auto-recording.
+
+  function enterPaused() {
+    phase = "paused";
+    toast(`Press ${keyLabel()} to talk`);
+  }
+
+  // One press takes the floor: cut whatever audio is queued or playing, drop
+  // the pending reply, and open the mic. The mode stays on - a press always
+  // means "I want to talk now", so it must never exit silently.
+
+  function bargeIn() {
+    logger?.log("VOICE", "Conversation barge-in", "debug");
+    discardStreamAudio();
+    clearProcessingToast();
+    // Settle the pending reply wait as "stopped" so the abandoned answer is
+    // never flushed into the mic we are about to open.
+    if (waitCancel) {
+      waitCancel("stopped");
+      waitCancel = null;
+    }
+    beginTurn();
   }
 
   function endReplyStream() {
@@ -221,7 +257,7 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     const { sentences, rest } = splitSpokenSentences(streamBuffer);
     const ready = final && rest.trim() ? [...sentences, rest] : sentences;
     if (ready.length > 0 && streamSpokenChars === 0) {
-      showProcessingToast("Speaking...");
+      showProcessingToast(`Speaking - press ${keyLabel()} to interrupt`);
     }
     for (const sentence of ready) {
       if (!isSpeakableSentence(sentence)) continue;
@@ -234,9 +270,11 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
   }
 
   // Wait for the submitted turn to finish: idle, a permission/question gate,
-  // or timeout. Resolves early when stop() cancels the wait.
+  // or timeout. Resolves early when stop() cancels the wait. gates:false drops
+  // the permission/question subscriptions, so a second wait survives the gate
+  // the user is answering on screen and only ends on idle/timeout/stop.
 
-  function waitForReply(sessionID) {
+  function waitForReply(sessionID, { gates = true } = {}) {
     let settled = false;
     let unsubs = [];
     let timer = null;
@@ -276,12 +314,16 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
             finish("idle");
           }
         }),
-        api.event.on("permission.asked", (event) => {
-          if (forSession(event.properties)) finish("permission");
-        }),
-        api.event.on("question.asked", (event) => {
-          if (forSession(event.properties)) finish("question");
-        }),
+        ...(gates
+          ? [
+              api.event.on("permission.asked", (event) => {
+                if (forSession(event.properties)) finish("permission");
+              }),
+              api.event.on("question.asked", (event) => {
+                if (forSession(event.properties)) finish("question");
+              }),
+            ]
+          : []),
       ];
       timer = setTimeout(() => finish("timeout"), replyTimeoutMs);
     });
@@ -293,6 +335,7 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     if (!active || phase !== "recording") return;
     const myGen = generation;
     phase = "processing";
+    showProcessingToast("Transcribing...");
 
     const res = await stt.transcribeTurn();
     if (!active || myGen !== generation) return;
@@ -304,12 +347,12 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
         stop("error");
         return;
       }
-      pauseForRetry("Paused - press the key to retry");
+      pauseForRetry(`Paused - press ${keyLabel()} to retry`);
       return;
     }
     if (!res.text) {
       consecErrors = 0;
-      pauseForRetry("No speech heard - press the key to try again");
+      pauseForRetry(`No speech heard - press ${keyLabel()} to try again`);
       return;
     }
     consecErrors = 0;
@@ -331,7 +374,7 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     phase = "waiting";
     const sessionID = currentSessionID();
     const turnStart = Date.now();
-    showProcessingToast("Waiting for reply...");
+    showProcessingToast(`Working - press ${keyLabel()} to talk`);
     startReplyStream(sessionID, turnStart);
     const tWait = Date.now();
     const outcome = await waitForReply(sessionID);
@@ -351,46 +394,78 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
       return;
     }
     if (outcome === "permission" || outcome === "question") {
-      discardStreamAudio();
-      phase = "speaking";
-      showProcessingToast("Speaking...");
-      await tts.speak(
-        outcome === "permission"
-          ? "Permission requested. Please check your screen."
-          : "A question needs your answer. Please check your screen.",
-      );
-      clearProcessingToast();
-      if (!active || myGen !== generation || phase !== "speaking") return;
-      beginTurn();
+      await answerGateThenSpeak(myGen, sessionID, turnStart, outcome);
       return;
     }
     if (outcome === "stopped") return;
 
-    // Streamed path: sentences already queued; flush the tail and let the
-    // audio finish before listening again. Fall back to the full
-    // LLM-narrated speak only when nothing streamable arrived.
+    await speakReply(myGen);
+  }
+
+  // A permission/question gate is answered ON SCREEN, so the mic must stay
+  // shut. Announce the gate, then re-arm the reply stream and wait for the
+  // agent to resume so its post-answer reply is spoken aloud - before this,
+  // endReplyStream() had already run and the answer was dropped in silence.
 
+  async function answerGateThenSpeak(myGen, sessionID, turnStart, outcome) {
+    discardStreamAudio();
+    phase = "speaking";
+    showProcessingToast("Speaking...");
+    await tts.speak(
+      outcome === "permission"
+        ? "Permission requested. Please check your screen."
+        : "A question needs your answer. Please check your screen.",
+    );
+    clearProcessingToast();
+    if (!active || myGen !== generation || phase !== "speaking") return;
+
+    // Still "waiting": the user answers on screen, and a press here is an
+    // ordinary barge-in (drop the pending reply, open the mic).
+    phase = "waiting";
+    showProcessingToast(`Answer on screen - press ${keyLabel()} to talk`);
+    startReplyStream(sessionID, turnStart);
+    const resumed = await waitForReply(sessionID, { gates: false });
+    if (!active || myGen !== generation) return;
+    if (resumed === "stopped") return;
+    endReplyStream();
+    if (resumed === "timeout") {
+      discardStreamAudio();
+      stop("timeout");
+      toast("No reply in time, conversation off", "warning");
+      return;
+    }
+    await speakReply(myGen);
+  }
+
+  // Speak whatever this turn produced, then rest paused with the mic OFF.
+  // Streamed sentences are already queued, so only the tail needs flushing;
+  // the full LLM-narrated speak is the fallback when nothing streamable
+  // arrived. Returns early on any generation/phase change (barge-in, stop).
+
+  async function speakReply(myGen) {
+    phase = "speaking";
+    showProcessingToast(`Speaking - press ${keyLabel()} to interrupt`);
     if (streamSpokenChars > 0) {
-      phase = "speaking";
       flushStream(true);
       await speakTail;
       clearProcessingToast();
       if (!active || myGen !== generation || phase !== "speaking") return;
       await delay(restartDelayMs);
       if (!active || myGen !== generation || phase !== "speaking") return;
-      beginTurn();
+      enterPaused();
       return;
     }
-
-    phase = "speaking";
     const spoken = await tts.speakAssistantTurn();
     if (!active || myGen !== generation || phase !== "speaking") return;
+    // Drop the sticky speaking status before the paused toast, or it keeps
+    // re-announcing "interrupt" once the mic is closed again.
+    clearProcessingToast();
     if (!spoken.spoken) {
-      toast("No reply to speak, listening again", "warning");
+      toast("No reply to speak", "warning");
     }
     await delay(restartDelayMs);
     if (!active || myGen !== generation || phase !== "speaking") return;
-    beginTurn();
+    enterPaused();
   }
 
   // Pause instead of auto-recording so an empty or failed turn never feels
@@ -402,14 +477,15 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     if (message) toast(message, "warning");
   }
 
-  // Stop speech and hold the mode. The next keypress records again.
+  // Stop speech and hold the mode, mic still closed. The next keypress records
+  // again. Only the TTS stop key lands here; the conversation key barges in.
 
   function pauseSpeech() {
     if (!active || phase !== "speaking") return;
     logger?.log("VOICE", "Conversation speech paused", "debug");
     tts.stop();
     phase = "paused";
-    toast("Paused - press the key to speak");
+    toast(`Speech paused - press ${keyLabel()} to talk`);
   }
 
   // Called by the TTS stop command while the mode is on. Returns true when the
@@ -431,7 +507,10 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     return false;
   }
 
-  function onKey(source) {
+  // One press, one meaning: "I want the floor". The source key (record,
+  // submit, toggle) no longer changes the outcome, and a press never exits.
+
+  function onKey() {
     if (!active) {
       start();
       return;
@@ -440,21 +519,16 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
       finishTurnFlow();
       return;
     }
-    if (phase === "speaking") {
-      pauseSpeech();
+    if (phase === "speaking" || phase === "waiting") {
+      bargeIn();
       return;
     }
     if (phase === "paused") {
       beginTurn();
       return;
     }
-    // processing | waiting | idle
-    if (source === "toggle") {
-      stop("cancelled");
-      toast("Conversation off");
-    } else {
-      toast("Conversation busy, please wait...", "warning");
-    }
+    // processing | idle: the transcription in flight is too short to interrupt.
+    toast("Conversation busy, please wait...", "warning");
   }
 
   function start() {
@@ -471,12 +545,12 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     generation += 1;
     turn = 0;
     consecErrors = 0;
-    phase = "idle";
+    phase = "paused";
     tts.setConversationActive(true);
-    stt.setStopHint("<leader>v");
+    // Recording toasts must name the key the user actually has bound.
+    stt.setStopHint(kb("voice.conversation"));
     logger?.log("VOICE", "Conversation started", "debug");
-    toast("Conversation on - one key: send, pause speech, resume");
-    beginTurn();
+    toast(`Conversation on - press ${keyLabel()} to talk`);
   }
 
   api.lifecycle?.onDispose?.(() => stop("dispose"));
@@ -495,16 +569,24 @@ export function registerConversation(api, opts, logger, { stt, tts, isLiveNotesA
     return v;
   }
 
+  // Every toast names the key that acts, resolved from the user's keybinds
+  // instead of the default literal - a rebound key must never be announced as
+  // something the user does not have.
+  function keyLabel() {
+    return kb("voice.conversation") || "the key";
+  }
+
   const commands = [
     {
       title: "Voice: conversation mode",
       value: "voice.conversation",
       category: "opencode-voice",
-      description: "Toggle voice conversation (one key: send, pause speech, resume, exit)",
+      description:
+        "Toggle voice conversation (push-to-talk: press to record, press again to send, press while it speaks to interrupt)",
       ...(kb("voice.conversation") ? { keybind: kb("voice.conversation") } : {}),
       slash: { name: "voice-conversation" },
       onSelect() {
-        onKey("toggle");
+        onKey();
       },
     },
     {

package/lib/tts.js

@@ -66,6 +66,44 @@ export function localSpeechCleanup(text) {
   return out.replace(/\s+/g, " ").trim();
 }
 
+// ---- Speech must never depend on the LLM ----
+// The narrator is a cosmetic polish layer on top of local Piper, which is
+// always available. So when the narrator fails for any reason (quota
+// exhausted, bad auth, unreachable endpoint, missing model, non-2xx) we speak
+// the deterministic local cleanup instead of going silent - this is the one
+// place the decision lives, so every caller benefits.
+//
+// Pure so it is testable without a TUI, a model, or piper.
+
+export function resolveSpeechText(rawText, llmResult) {
+  if (llmResult?.text) return { text: llmResult.text };
+  return { text: localSpeechCleanup(rawText), error: llmResult?.error, fellBack: true };
+}
+
+// The llm-client retries with backoff (normalizeRetries in llm-client.js), so a
+// model that is merely out of quota stalls for many seconds before admitting
+// failure - waiting that long for a polish layer is worse than unpolished
+// speech, so bound the wait. The timer is unref'd (same trick as
+// wallTimeout in streaming-stt.js) and cleared on the happy path, so tests and
+// short-lived processes never wait out the full bound.
+const NORMALIZE_FALLBACK_TIMEOUT_MS = 12000;
+
+async function normalizeWithinBound(promise) {
+  let timer = null;
+  const bound = new Promise((resolve) => {
+    timer = setTimeout(() => {
+      timer = null;
+      resolve({ text: null, error: "LLM normalization timed out" });
+    }, NORMALIZE_FALLBACK_TIMEOUT_MS);
+    timer.unref?.();
+  });
+  try {
+    return await Promise.race([promise, bound]);
+  } finally {
+    if (timer) clearTimeout(timer);
+  }
+}
+
 // Split streamed text into complete spoken sentences. Returns what is ready
 // plus the trailing incomplete fragment to keep buffering. Sentences that
 // look like code dumps (overlong, backticks) are skipped by the caller.
@@ -473,14 +511,11 @@ export function registerTTS(api, kv, complete, prompts, opts, logger, deps = {})
       clearProcessingToast();
       return;
     }
-    if (!llmResult.text) {
+    if (!announceNormalize("Auto", llmResult)) {
       clearProcessingToast();
-      logger?.log?.("TTS", `Auto normalization failed: ${llmResult.error}`, "warn");
-      toast(`TTS normalization failed: ${llmResult.error}`, "warning");
       return;
     }
 
-    logger?.log?.("TTS", `Auto normalization succeeded chars=${llmResult.text.length}`, "debug");
     clearProcessingToast();
     await speakWithSessionPrefix(sessionID, llmResult.text, "Ready for your input.");
   });
@@ -519,14 +554,11 @@ export function registerTTS(api, kv, complete, prompts, opts, logger, deps = {})
       clearProcessingToast();
       return;
     }
-    if (!llmResult.text) {
+    if (!announceNormalize("Manual", llmResult)) {
       clearProcessingToast();
-      logger?.log?.("TTS", `Manual normalization failed: ${llmResult.error}`, "warn");
-      toast(`TTS normalization failed: ${llmResult.error}`, "warning");
       return;
     }
 
-    logger?.log?.("TTS", `Manual normalization succeeded chars=${llmResult.text.length}`, "debug");
     updateProcessingToast("Speaking...");
     await speak(llmResult.text);
     clearProcessingToast();
@@ -541,7 +573,31 @@ export function registerTTS(api, kv, complete, prompts, opts, logger, deps = {})
 
   async function normalizeOrLocal(text, systemPrompt, maxTokens) {
     if (ttsLocal) return { text: localSpeechCleanup(text) };
-    return normalizeForSpeech(text, systemPrompt, maxTokens);
+    const llmResult = await normalizeWithinBound(normalizeForSpeech(text, systemPrompt, maxTokens));
+    return resolveSpeechText(text, llmResult);
+  }
+
+  // Shared tail for all three normalize call sites: one informational toast
+  // when we are speaking the local fallback, and - when the cleanup left
+  // nothing speakable (a pure code dump) - a quiet debug log, because nothing
+  // failed there, there was just nothing to read. Returns whether to speak.
+
+  function announceNormalize(label, result) {
+    if (!result.text) {
+      logger?.log?.("TTS", `${label}: nothing speakable (${result.error})`, "debug");
+      return false;
+    }
+    if (result.fellBack) {
+      logger?.log?.(
+        "TTS",
+        `${label} narration unavailable (${result.error}); speaking local cleanup`,
+        "warn",
+      );
+      toast("Model unavailable - speaking text as-is");
+      return true;
+    }
+    logger?.log?.("TTS", `${label} normalization succeeded chars=${result.text.length}`, "debug");
+    return true;
   }
 
   async function speakAssistantTurn() {
@@ -564,11 +620,9 @@ export function registerTTS(api, kv, complete, prompts, opts, logger, deps = {})
       clearProcessingToast();
       return { spoken: false };
     }
-    if (!llmResult.text) {
+    if (!announceNormalize("Conversation", llmResult)) {
       clearProcessingToast();
-      logger?.log?.("TTS", `Conversation normalization failed: ${llmResult.error}`, "warn");
-      toast(`TTS normalization failed: ${llmResult.error}`, "warning");
-      return { spoken: false, error: llmResult.error };
+      return { spoken: false };
     }
 
     updateProcessingToast("Speaking...");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bojackduy/opencode-voice",
-  "version": "0.8.0",
+  "version": "0.9.0",
   "description": "Speech-to-text and text-to-speech for OpenCode. Record voice prompts with whisper-cpp, hear responses via Piper TTS, with LLM normalization through any OpenAI-compatible endpoint.",
   "keywords": [
     "opencode",