mac-voice-mcp 0.3.0 → 0.4.0

package/CHANGELOG.md

@@ -5,6 +5,33 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-10-06
+
+### Added
+- **An optional natural voice: Kokoro.** [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) is a small neural voice that runs on your Mac and sounds much closer to a person than `say`.
+  - **Install it only if you want it:** `voice_setup` with `kokoro=true` (Claude asks first), or `npx mac-voice-mcp setup --kokoro`. It installs `kokoro-js@1.2.1` with npm into `~/.cache/mac-voice-mcp/kokoro/` and downloads the model once, about 1 GB on disk. Nothing is bundled, and speaking never downloads anything.
+  - **Once installed it's used automatically**, with the `af_heart` voice. It runs in its own warm process and speaks sentence by sentence, so the first sentence plays while the rest is generated. The timing line shows how soon the first sound came.
+  - **It can't leave you without a voice.** If Kokoro fails, the built-in voice says whatever hadn't been said yet, and Claude is told once.
+  - **Settings:** `VOICE_MCP_TTS` (`auto`, `say` or `kokoro`), `VOICE_MCP_KOKORO_VOICE`, `VOICE_MCP_KOKORO_SPEED`, `VOICE_MCP_KOKORO_DTYPE` and `VOICE_MCP_KOKORO_DIR`.
+  - **`doctor` and the CI round trip cover Kokoro too:** whisper must understand it, and the first sound must come quickly.
+- **`doctor`: an objective self-check** (`npx mac-voice-mcp doctor`). It checks the whole pipeline on your Mac and marks each stage PASS, WARN or FAIL against fixed limits:
+  - **Setup:** everything is installed.
+  - **Speech → text:** the voice speaks a known sentence into a file, and whisper must get the words right.
+  - **Speaker → mic:** the same sentence is played aloud and recorded. It catches mic permission problems, the wrong input device and headphones.
+
+  Reports are saved as JSON under `~/.cache/mac-voice-mcp/doctor/`. Use `--no-loopback` to skip the speakers, and `--json` for machine-readable output.
+- **A "Speech round trip" CI job on a real Mac.** It installs everything with our own `setup --install`, runs `doctor`, then speaks known sentences (everyday and developer jargon) with the real voice and transcribes them with the real whisper.cpp. The build fails on too many wrong words or slow transcription.
+- **`npm run dev:plugin`** loads this checkout into Claude Code as a separate "mac-voice-mcp-dev" plugin. It runs the local build with `node`, with no npx and no clash with the installed plugin.
+
+## [0.3.1] - 2026-09-29
+
+### Fixed
+- **Recording broke on Macs without SoX when Bluetooth headphones were connected** ([#6](https://github.com/jeet0007/mac-voice-mcp/issues/6)). Every turn failed with "pure digital silence".
+  - The ffmpeg fallback now records from the input chosen in System Settings (`:default`) instead of device number 0. `VOICE_MCP_FFMPEG_DEVICE` still overrides it.
+  - `voice_setup` treats ffmpeg as a fallback, recommends SoX, and installs it with your OK.
+  - Silence recorded through ffmpeg now points at the input device and SoX, not only at mic permissions.
+- **`voice_setup` notices tools installed since the last check**, such as `brew install sox` run in a terminal, without restarting the app.
+
 ## [0.3.0] - 2026-09-28
 
 ### Added
@@ -69,7 +96,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
   - `voice_setup`: checks first, installs only what's missing, and only after you agree.
   - The `setup` and `voice_mode` prompts.
 
-[Unreleased]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.3.1...v0.4.0
+[0.3.1]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/jeet0007/mac-voice-mcp/compare/v0.1.1...v0.2.0
 [0.1.1]: https://github.com/jeet0007/mac-voice-mcp/compare/119a589...v0.1.1
 [0.1.0]: https://github.com/jeet0007/mac-voice-mcp/tree/119a589

package/README.md

@@ -123,7 +123,7 @@ Setup checks what's already there before it changes anything:
 | Needed | Provided by | If it's missing |
 |---|---|---|
 | Voice | macOS `say`, using the most natural voice installed | Nothing to do. For a far better voice, add a free Premium one (see below). |
-| Microphone capture | SoX (`rec`) | `brew install sox` |
+| Microphone capture | SoX (`rec`). ffmpeg works as a fallback, but setup recommends SoX | `brew install sox` |
 | Speech-to-text | whisper.cpp (`whisper-cli` and `whisper-server`, Metal-accelerated) | `brew install whisper-cpp` |
 | Speech model | `base.en`, ~140 MB | Downloaded once to `~/.cache/mac-voice-mcp/models/` |
 
@@ -135,6 +135,14 @@ Setup checks what's already there before it changes anything:
 
 **Get a better voice (recommended).** macOS includes free Premium voices that sound far more natural than the default. Open **System Settings → Accessibility → Spoken Content → System Voice → Manage Voices…**, and download one, for example English → *Ava (Premium)* or *Zoe (Premium)*. The next voice turn uses it automatically. To choose a specific voice, or keep the system voice, see `VOICE_MCP_VOICE` under [Configuration](#configuration).
 
+**Or try the Kokoro voice (optional).** [Kokoro](https://huggingface.co/hexgrad/Kokoro-82M) is a small neural voice that sounds close to a person and runs entirely on your Mac. Ask Claude to *"install the Kokoro voice"*, or run `npx -y mac-voice-mcp@latest setup --kokoro`.
+
+- **What it installs:** [kokoro-js](https://github.com/hexgrad/kokoro) with npm into `~/.cache/mac-voice-mcp/kokoro/`, and the model from Hugging Face, once. That's about 1 GB on disk, and nothing is bundled with this package.
+- **How it's used:** once it's installed, every turn uses it, starting with `af_heart`. Pick another voice with `VOICE_MCP_KOKORO_VOICE` (see [Configuration](#configuration)). It speaks sentence by sentence, so it starts talking before the whole reply is generated. Each turn's timing line shows how soon the first sound came.
+- **It can't leave you without a voice.** If Kokoro fails for any reason, the built-in voice takes over mid-sentence, and Claude tells you once.
+- **To stop using it:** set `VOICE_MCP_TTS=say`, or delete `~/.cache/mac-voice-mcp/kokoro/`.
+- **Limits:** English only (American and British voices). It speaks with its own pronunciation rules, which handle common developer words (JSON, `index.ts`, version numbers).
+
 ### Allow voice turns without prompts
 
 By default, Claude Code asks for approval every time Claude wants to speak, which breaks the flow of a conversation. To allow voice turns, add the tool to the `permissions.allow` list in `~/.claude/settings.json`. Use the name that matches how you installed it:
@@ -222,7 +230,12 @@ Everything is optional. Set these in your client config's `"env": { … }` block
 | Variable | Default | |
 |---|---|---|
 | `VOICE_MCP_VOICE` | most natural installed | Unset: the best Premium or Enhanced voice installed for the language, else the system voice. Set a name, e.g. `Ava (Premium)`, `Daniel`, `Kanya` (list them with `say -v '?'`), or `default` to always use the system voice. |
-| `VOICE_MCP_RATE` | system rate | Words per minute, e.g. `200`. |
+| `VOICE_MCP_RATE` | system rate | Words per minute, e.g. `200` (macOS voices). |
+| `VOICE_MCP_TTS` | `auto` | `auto`: the [Kokoro voice](#then-set-up-and-allow-the-mic) once it's installed, else the built-in voice. `say`: always the built-in voice. `kokoro`: Kokoro, and setup offers to install it. |
+| `VOICE_MCP_KOKORO_VOICE` | `af_heart` | Kokoro voice. American English starts with `a`, British with `b`, e.g. `af_bella`, `am_michael`, `bf_emma`, `bm_george`. |
+| `VOICE_MCP_KOKORO_SPEED` | `1` | Kokoro speaking speed, `0.5` to `2`. |
+| `VOICE_MCP_KOKORO_DTYPE` | `fp32` | `q8`: a smaller model (~90 MB instead of ~330 MB) that's about half as fast. |
+| `VOICE_MCP_KOKORO_DIR` | `~/.cache/mac-voice-mcp/kokoro` | Where the Kokoro voice is installed. |
 | `VOICE_MCP_MAX_SPEAK_WORDS` | `120` | Longer text is cut at a sentence boundary ("the rest is on screen"). |
 | `VOICE_MCP_CHIME` | `1` | Set to `0` to turn off the mic open/close sounds. |
 | `VOICE_MCP_LOCK_WAIT_SECONDS` | `120` | How long a turn waits while another session on this Mac is using the mic. |
@@ -235,7 +248,8 @@ Everything is optional. Set these in your client config's `"env": { … }` block
 | `VOICE_MCP_START_TIMEOUT_SECONDS` | `15` | How long to wait for you to start talking. |
 | `VOICE_MCP_SPEECH_MARGIN_DB` | `12` | How much louder than room noise counts as speech. Raise it in noisy rooms. |
 | `VOICE_MCP_MIN_SPEECH_DB` | `-48` | The quietest level that ever counts as speech (dBFS). |
-| `VOICE_MCP_RECORDER` | `auto` | `sox` or `ffmpeg` (`ffmpeg` is macOS only). |
+| `VOICE_MCP_RECORDER` | `auto` | `sox` or `ffmpeg` (`ffmpeg` is macOS only). `auto` uses SoX, falling back to ffmpeg. |
+| `VOICE_MCP_FFMPEG_DEVICE` | `:default` | Which input ffmpeg records from. `:default` follows System Settings; `:1` picks device 1 (list them with `ffmpeg -f avfoundation -list_devices true -i ""`). |
 
 **Speech-to-text**
 
@@ -271,15 +285,17 @@ Everything is optional. Set these in your client config's `"env": { … }` block
 
 | Symptom | Fix |
 |---|---|
+| Not sure what's wrong | Run `npx -y mac-voice-mcp@latest doctor`. It checks setup, the voice and transcription, and your speakers and mic, and says which part fails. |
 | "voice-mcp is not set up yet" | Ask Claude to *set up voice*, or run `npx -y mac-voice-mcp@latest setup`. |
-| "microphone returned pure digital silence" | macOS is blocking the mic for the host app. Go to **System Settings → Privacy & Security → Microphone**, enable Claude / Cursor / your terminal, then restart that app. |
+| "microphone returned pure digital silence" | macOS is blocking the mic for the host app. Go to **System Settings → Privacy & Security → Microphone**, enable Claude / Cursor / your terminal, then restart that app. If the message says it's recording with ffmpeg, the input device is the likelier cause: run `brew install sox`. |
 | No permission prompt ever appears | Run `tccutil reset Microphone <bundle id>` and restart the app. Running `test` in Terminal only gives permission to Terminal, not to Claude Desktop. |
 | It cuts me off while I'm thinking | Set `VOICE_MCP_END_SILENCE_MS=1800` (or up to `2500`). |
 | It never stops listening | The room is too noisy for the defaults. Set `VOICE_MCP_SPEECH_MARGIN_DB=18`, or use a headset. |
 | It hears its own voice | Use headphones, or turn the speaker volume down. It only listens after it finishes speaking, but echo can linger. |
 | "Homebrew: not installed" | Install it from [brew.sh](https://brew.sh). It needs your password, so it can't run from Claude. Then run setup again. |
 | It garbles names or jargon | Set `VOICE_MCP_WHISPER_PROMPT="Priya, Postgres, Kubernetes"`, or switch to `small.en`. |
-| The voice sounds robotic | Download a Premium voice (see [Get a better voice](#then-set-up-and-allow-the-mic)). It's used automatically. |
+| The voice sounds robotic | Download a Premium voice, or install the Kokoro voice (see [Get a better voice](#then-set-up-and-allow-the-mic)). Either is used automatically. |
+| "The Kokoro voice didn't work" | The built-in voice spoke instead. Ask Claude to *check voice setup*, or run `npx -y mac-voice-mcp@latest setup`. To reinstall it, delete `~/.cache/mac-voice-mcp/kokoro/` and run `setup --kokoro`. |
 | It asks for approval every turn | Add the tool to Claude Code's allow list: see [Allow voice turns without prompts](#allow-voice-turns-without-prompts). |
 | "Another voice session on this Mac…" | Another Claude window, Claude Desktop or Cursor held the speaker and mic for over 2 minutes, which means one very long turn. End that conversation, then try again. |
 | Claude keeps answering by voice after I'm done | Type anything, or say "stop voice mode". To switch the plugin's hook off entirely, see [Voice mode](#voice-mode). |
@@ -294,9 +310,10 @@ Everything is optional. Set these in your client config's `"env": { … }` block
 
 ## Privacy and safety
 
-- **Audio stays on your machine.** Recordings go to a temporary file that's deleted after each turn. The only network use is the one-time model download from Hugging Face.
+- **Audio stays on your machine.** Recordings go to a temporary file that's deleted after each turn. The only network use is the one-time model download from Hugging Face, plus npm and Hugging Face once more if you install the Kokoro voice. Speaking never downloads anything.
 - **The warm whisper server is local only.** It listens on `127.0.0.1` on a random port, and stops when idle or when this server exits.
-- **Setup can only install known packages.** Its install list is fixed in the code (`sox`, `whisper-cpp`), so nothing Claude says can make it install anything else. It never uninstalls or modifies other software.
+- **Setup can only install known packages.** Its install list is fixed in the code (`sox`, `whisper-cpp`, and `kokoro-js@1.2.1` for the optional voice), so nothing Claude says can make it install anything else. It never uninstalls or modifies other software.
+- **Licenses of the optional Kokoro voice.** The Kokoro model and kokoro-js are Apache-2.0. kokoro-js turns text into sounds with a WebAssembly build of espeak-ng (GPL-3.0), through the `phonemizer` package. None of this is part of mac-voice-mcp (MIT). It's installed on your Mac only if you ask for it.
 
 ## Security
 
@@ -315,6 +332,8 @@ npm test               # build + unit tests, end-to-end tests over MCP with stub
 npm run audit          # known-vulnerability and signature checks on dependencies
 npm run setup          # check what's installed; offers to install what's missing
 npm run test:voice     # one real speak → listen → transcribe turn
+npm run doctor         # objective self-check on this Mac: PASS / WARN / FAIL per stage, JSON report
+npm run dev:plugin     # try this checkout in Claude Code as the "mac-voice-mcp-dev" plugin
 npm run inspect        # MCP Inspector
 ```
 
@@ -324,6 +343,8 @@ npm run inspect        # MCP Inspector
 | `speech-text.ts` | Rewriting screen text for speech, cleaning up transcripts (pure, unit-tested) |
 | `endpointer.ts` | Turn-taking voice-activity detection (pure, unit-tested) |
 | `audio.ts` | Text-to-speech, chimes, streaming mic capture |
+| `kokoro.ts`, `kokoro-worker.ts` | The optional Kokoro voice: install, a warm worker process, sentence-by-sentence playback, fallback |
+| `kokoro-text.ts`, `pcm.ts` | Pronunciation fixes and sentence chunks for Kokoro, PCM conversion (pure, unit-tested) |
 | `model.ts` | Finding, symlinking or downloading the model |
 | `stt.ts` | The warm `whisper-server` with orphan guard, and the `whisper-cli` fallback |
 | `setup.ts` | Requirement checks and consent-based background installs |
@@ -333,7 +354,13 @@ npm run inspect        # MCP Inspector
 
 The package installs two commands: `mac-voice-mcp` (the one `npx -y mac-voice-mcp` runs), and `voice-mcp`.
 
-**Testing the plugin: don't start Claude Code inside this repo.** In this folder, `npx mac-voice-mcp@<this version>` finds the checkout itself instead of downloading the package, can't run it, and the server fails with `CONNECTION_CLOSED`. Start `claude` in any other folder. To try unreleased plugin files (skills, hooks), swap the marketplace to your checkout: run `/plugin marketplace remove mac-voice-mcp`, then `/plugin marketplace add /path/to/checkout`, and install at user scope. The server still comes from npm, so test server changes with `npm test` and `npm run test:voice`.
+**Testing changes.** Three layers; none of them depend on anyone's ears.
+
+1. **`npm test`** runs everywhere, with stub binaries. It covers the MCP tools, turn-taking, setup, the mic lock, the plugin hook and the release metadata.
+2. **The "Speech round trip" CI job** runs on a real Mac. Our own `setup --install` puts in SoX, whisper.cpp and the model. Then `doctor --no-loopback` runs, and `test/roundtrip.test.mjs` speaks known sentences with the real voice and transcribes them with the real whisper.cpp. The build fails if too many words come back wrong or transcription is too slow. The same job then installs the Kokoro voice with `setup --kokoro` and repeats the round trip with it, which also checks how soon Kokoro starts talking. GitHub's Macs have only basic voices and no GPU for whisper.cpp, so the limits there are looser: it guards against breakage, and doctor on a real Mac is the quality bar. The numbers are kept as a build artifact.
+3. **`npm run doctor`** on your own Mac adds the one thing CI can't test, your speakers and microphone. It plays a sentence through the speakers, records it and transcribes it. Each stage gets PASS, WARN or FAIL against fixed limits, and the report is saved under `~/.cache/mac-voice-mcp/doctor/`.
+
+**Trying the plugin from a checkout:** run `npm run dev:plugin`, then the `claude --plugin-dir …` command it prints. This loads a throwaway plugin, "mac-voice-mcp-dev", that runs this checkout's build with `node`. Its own name means it doesn't clash with an installed mac-voice-mcp. It also works inside this repo, where `npx mac-voice-mcp@<this version>` would find the checkout instead of the package and fail with `CONNECTION_CLOSED`.
 
 ### Releasing
 

package/dist/audio.js

@@ -2,8 +2,9 @@
 import { spawn } from "node:child_process";
 import { existsSync } from "node:fs";
 import { writeFile } from "node:fs/promises";
-import { CONFIG, debug, IS_MAC, IS_WIN } from "./config.js";
+import { CONFIG, debug, IS_MAC, IS_WIN, log } from "./config.js";
 import { Endpointer, FRAME_BYTES, wavHeader } from "./endpointer.js";
+import { KokoroError, kokoroEnabled, speakKokoro, synthesizeKokoroToFile } from "./kokoro.js";
 import { activeChildren, CancelledError, run, SetupError, tail, which } from "./proc.js";
 // ---------------------------------------------------------------------------
 // Text-to-speech
@@ -71,9 +72,33 @@ export function chooseVoice() {
     })();
     return voiceChoice;
 }
+/**
+ * Speak `text` out loud: with the Kokoro voice when it's installed (or VOICE_MCP_TTS=kokoro),
+ * otherwise the built-in voice. If Kokoro fails, the built-in voice says whatever it hadn't said yet.
+ */
 export async function speak(text, signal) {
     if (!text)
-        return;
+        return { engine: "built-in" };
+    if (kokoroEnabled()) {
+        try {
+            return { engine: "kokoro", ...(await speakKokoro(text, signal)) };
+        }
+        catch (err) {
+            if (err instanceof CancelledError)
+                throw err;
+            const reason = err instanceof Error ? err.message : String(err);
+            log(`The Kokoro voice failed, so the built-in voice is speaking instead: ${reason}`);
+            const rest = err instanceof KokoroError ? err.unspoken.join(" ") : text;
+            if (rest)
+                await speakBuiltIn(rest, signal);
+            return { engine: "built-in", fallback: reason };
+        }
+    }
+    await speakBuiltIn(text, signal);
+    return { engine: "built-in" };
+}
+/** macOS `say`, espeak-ng, or Windows SAPI. */
+async function speakBuiltIn(text, signal) {
     const bin = findTts();
     if (!bin)
         throw new SetupError("No text-to-speech engine found. Install espeak-ng (e.g. `sudo apt install espeak-ng`).");
@@ -103,6 +128,32 @@ export async function speak(text, signal) {
         throw new Error(`Text-to-speech failed (exit ${result.code}): ${tail(result.stderr) || "no output"}`);
     }
 }
+/**
+ * Speak `text` into a 16 kHz mono 16-bit WAV instead of the speakers, with the same voice
+ * `speak` uses. For `doctor` and the round-trip tests. A Kokoro failure is an error here, not a fallback.
+ */
+export async function synthesizeToFile(text, outFile) {
+    if (kokoroEnabled())
+        return { engine: "kokoro", ...(await synthesizeKokoroToFile(text, outFile)) };
+    const bin = findTts();
+    if (!bin)
+        throw new SetupError("No text-to-speech engine found.");
+    let result;
+    if (IS_MAC) {
+        const { voice } = await chooseVoice();
+        const args = [...(voice ? ["-v", voice] : []), "-o", outFile, "--file-format=WAVE", "--data-format=LEI16@16000", "-f", "-"];
+        result = await run(bin, args, { input: text, timeoutMs: 60_000 });
+    }
+    else if (/espeak/.test(bin)) {
+        result = await run(bin, ["-w", outFile, "--stdin"], { input: text, timeoutMs: 60_000 });
+    }
+    else {
+        throw new SetupError("Writing speech to a file needs macOS `say` or espeak-ng.");
+    }
+    if (result.code !== 0)
+        throw new Error(`Text-to-speech to file failed (exit ${result.code}): ${tail(result.stderr) || "no output"}`);
+    return { engine: "built-in" };
+}
 /** A short, quiet cue that the mic just opened ("start") or closed ("stop"). macOS only. */
 export async function chime(kind) {
     if (!CONFIG.chime || !IS_MAC)
@@ -152,6 +203,63 @@ function recorderArgs(r) {
         "-ac", "1", "-ar", "16000", "-f", "s16le", "-",
     ];
 }
+/**
+ * Record exactly `seconds` from the microphone into `outFile` (16 kHz mono WAV), no turn-taking.
+ * `onStarted` fires once audio is actually flowing, so a caller can start playback in sync.
+ */
+export async function recordForSeconds(seconds, outFile, onStarted) {
+    const recorder = findRecorder();
+    if (!recorder)
+        throw new SetupError(RECORDER_MISSING);
+    const chunks = [];
+    let bytes = 0;
+    let stderr = "";
+    const wanted = Math.round(seconds * 16000) * 2;
+    const child = spawn(recorder.bin, recorderArgs(recorder), { stdio: ["ignore", "pipe", "pipe"], windowsHide: true });
+    activeChildren.add(child);
+    await new Promise((resolve, reject) => {
+        const stop = () => child.exitCode === null && child.kill("SIGTERM");
+        const safety = setTimeout(stop, (seconds + 10) * 1000);
+        child.stderr.setEncoding("utf8").on("data", (d) => (stderr = (stderr + d).slice(-4000)));
+        child.stdout.on("data", (chunk) => {
+            if (bytes === 0)
+                onStarted?.();
+            if (bytes >= wanted)
+                return;
+            chunks.push(chunk);
+            bytes += chunk.length;
+            if (bytes >= wanted)
+                stop();
+        });
+        child.on("error", (e) => {
+            clearTimeout(safety);
+            activeChildren.delete(child);
+            reject(e);
+        });
+        child.on("close", () => {
+            clearTimeout(safety);
+            activeChildren.delete(child);
+            resolve();
+        });
+    });
+    const pcm = Buffer.concat(chunks).subarray(0, wanted - (wanted % 2));
+    if (!pcm.length)
+        throw new Error(`Recording failed (${describeRecorder(recorder)}). ${tail(stderr)}`.trim());
+    await writeFile(outFile, Buffer.concat([wavHeader(pcm.length), pcm]));
+    let peak = 0;
+    let anyNonZero = false;
+    for (let off = 0; off + FRAME_BYTES <= pcm.length; off += FRAME_BYTES) {
+        let sum = 0;
+        for (let i = off; i < off + FRAME_BYTES; i += 2) {
+            const v = pcm.readInt16LE(i);
+            if (v !== 0)
+                anyNonZero = true;
+            sum += v * v;
+        }
+        peak = Math.max(peak, sum / (FRAME_BYTES / 2));
+    }
+    return { digitalSilence: !anyNonZero, peakDb: peak > 0 ? 10 * Math.log10(peak / 32768 ** 2) : -Infinity };
+}
 /**
  * Listen for one conversational turn and write it to `outFile` (16 kHz mono WAV).
  * Streams raw PCM from the recorder, runs the endpointer on it live, and stops the

package/dist/config.js

@@ -36,6 +36,21 @@ export const CONFIG = {
     /** How long a voice turn waits for another session to finish with the mic. */
     lockWaitSeconds: Math.max(1, envNum("VOICE_MCP_LOCK_WAIT_SECONDS", 120)),
     // --- Speaking
+    /**
+     * Which voice engine: auto (default: the Kokoro voice once it's installed, else the built-in one) |
+     * say (always the built-in voice: macOS `say`, espeak-ng elsewhere) | kokoro (Kokoro; voice_setup offers to install it).
+     */
+    tts: (["say", "kokoro"].includes(process.env.VOICE_MCP_TTS?.trim().toLowerCase() ?? "")
+        ? process.env.VOICE_MCP_TTS.trim().toLowerCase()
+        : "auto"),
+    /** Kokoro voice, American (a…) or British (b…) English: af_heart (default), af_bella, am_michael, bf_emma, … */
+    kokoroVoice: process.env.VOICE_MCP_KOKORO_VOICE?.trim() || "af_heart",
+    /** Kokoro speaking speed (1 = normal, 0.5–2). */
+    kokoroSpeed: Math.min(2, Math.max(0.5, envNum("VOICE_MCP_KOKORO_SPEED", 1))),
+    /** Kokoro model precision: fp32 (default: about twice as fast on a CPU, ~330 MB) or q8 (~90 MB, slower). */
+    kokoroDtype: (process.env.VOICE_MCP_KOKORO_DTYPE?.trim().toLowerCase() === "q8" ? "q8" : "fp32"),
+    /** Where the optional Kokoro voice (kokoro-js and its model) is installed. */
+    kokoroDir: process.env.VOICE_MCP_KOKORO_DIR?.trim() || path.join(CACHE_ROOT, "kokoro"),
     /**
      * macOS voice name, e.g. "Ava (Premium)" (`say -v '?'` lists them). Unset: the most natural
      * installed voice for the language (Premium, then Enhanced), else the system voice. "default": always the system voice.
@@ -58,8 +73,8 @@ export const CONFIG = {
     minSpeechDb: envNum("VOICE_MCP_MIN_SPEECH_DB", -48),
     /** Recorder: auto (SoX, else ffmpeg) | sox | ffmpeg. */
     recorder: (process.env.VOICE_MCP_RECORDER?.trim().toLowerCase() || "auto"),
-    /** ffmpeg avfoundation audio input (macOS fallback recorder). */
-    ffmpegDevice: process.env.VOICE_MCP_FFMPEG_DEVICE?.trim() || ":0",
+    /** ffmpeg avfoundation audio input (macOS fallback recorder). ":default" follows the input chosen in System Settings. */
+    ffmpegDevice: process.env.VOICE_MCP_FFMPEG_DEVICE?.trim() || ":default",
     // --- Speech-to-text
     /** whisper.cpp model name (tiny.en, base.en, small.en, large-v3-turbo-q5_0, ...). */
     modelName,

package/dist/doctor.js

@@ -0,0 +1,185 @@
+/**
+ * `doctor`: an objective self-check of the whole voice pipeline on this Mac.
+ *
+ * Every check ends in PASS / WARN / FAIL against fixed limits, so "does it work?" is answered by
+ * numbers rather than by ear. Results are also written as JSON, so reports from different Macs
+ * and audio setups (headphones, Bluetooth, external mics) can be compared.
+ *
+ *   1. Setup          everything speak_and_listen needs is installed.
+ *   2. Speech → text  the voice speaks a known sentence into a file and whisper transcribes it.
+ *                     Checks the voice and the transcriber without involving the room.
+ *   3. Speaker → mic  the same sentence is spoken live through the speakers (exactly as in a
+ *                     conversation) while the mic records, then transcribed. Checks the real
+ *                     hardware path (skipped with --no-loopback).
+ */
+import { mkdir, mkdtemp, rm, stat, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { chooseVoice, describeRecorder, findRecorder, recordForSeconds, speak, synthesizeToFile } from "./audio.js";
+import { CONFIG, IS_MAC, PKG } from "./config.js";
+import { kokoroEnabled, kokoroVoiceLabel, stopKokoro } from "./kokoro.js";
+import { ensureModel } from "./model.js";
+import { run } from "./proc.js";
+import { runSetupFlow } from "./setup.js";
+import { describeStt, stopWhisperServer, transcribe } from "./stt.js";
+import { wordErrorRate } from "./wer.js";
+export const DOCTOR_SENTENCE = "The build passed and all tests are green. Should I open the pull request now?";
+/** Word error rate limits: [PASS at or below, WARN at or below]; above that is FAIL. */
+export const LIMITS = {
+    /** Clean synthetic speech straight into whisper should be near perfect. */
+    file: [0.1, 0.25],
+    /** Through the air, room noise and a laptop mic make it harder. */
+    loopback: [0.3, 0.5],
+};
+/** Kokoro, model loaded: milliseconds from "speak" to the first sound, [PASS at or below, WARN at or below]. */
+export const FIRST_SOUND_MS = [1500, 3000];
+/** Ends the run early without counting as an unexpected error. */
+class StopChecks extends Error {
+}
+const grade = (wer, [pass, warn]) => (wer <= pass ? "PASS" : wer <= warn ? "WARN" : "FAIL");
+const pct = (x) => `${Math.round(x * 100)}%`;
+/** WAV length in seconds (16 kHz mono 16-bit, 44-byte header). */
+async function wavSeconds(file) {
+    return Math.max(0, ((await stat(file)).size - 44) / 32000);
+}
+/** The default input and output device names (macOS), for the report. Best effort. */
+async function audioDevices() {
+    if (!IS_MAC)
+        return {};
+    try {
+        const r = await run("/usr/sbin/system_profiler", ["SPAudioDataType", "-json"], { timeoutMs: 10_000 });
+        const items = (JSON.parse(r.stdout).SPAudioDataType?.[0]?._items ?? []);
+        const find = (key) => items.find((d) => d[key] === "spaudio_yes")?._name ?? "unknown";
+        return { input: find("coreaudio_default_audio_input_device"), output: find("coreaudio_default_audio_output_device") };
+    }
+    catch {
+        return {};
+    }
+}
+export async function runDoctor(opts = {}) {
+    const say = opts.log ?? (() => { });
+    const checks = [];
+    const add = (c) => {
+        checks.push(c);
+        say(`${c.status.padEnd(4)}  ${c.name}: ${c.detail}`);
+    };
+    const tmp = await mkdtemp(path.join(os.tmpdir(), "voice-mcp-doctor-"));
+    const report = {
+        version: PKG.version,
+        date: new Date().toISOString(),
+        machine: { platform: `${process.platform} ${os.release()}`, arch: process.arch, node: process.version, ...(await audioDevices()) },
+        checks,
+        ok: false,
+    };
+    try {
+        // 1. Setup
+        const setup = await runSetupFlow();
+        add({ name: "Setup", status: setup.ready ? "PASS" : "FAIL", detail: setup.ready ? "everything is installed" : "something is missing — run `setup`" });
+        if (!setup.ready)
+            throw new StopChecks();
+        const voice = await chooseVoice();
+        const recorder = findRecorder();
+        const kokoro = kokoroEnabled();
+        Object.assign(report.machine, {
+            voice: kokoro ? `${kokoroVoiceLabel()}, built-in fallback: ${voice.label}` : voice.label,
+            recorder: recorder ? describeRecorder(recorder) : "none",
+            stt: describeStt(),
+            model: CONFIG.modelPath ?? CONFIG.modelName,
+        });
+        const model = await ensureModel({ download: false });
+        // 2. Speech → text, straight from a file
+        const spoken = path.join(tmp, "spoken.wav");
+        let t = Date.now();
+        const cold = await synthesizeToFile(DOCTOR_SENTENCE, spoken);
+        const synthMs = Date.now() - t;
+        if (cold.engine === "kokoro") {
+            // Again with the voice loaded: how long a normal turn waits before the first sound.
+            const warm = await synthesizeToFile(DOCTOR_SENTENCE, spoken);
+            const ms = warm.firstAudioMs ?? 0;
+            const [pass, warn] = FIRST_SOUND_MS;
+            add({
+                name: "Kokoro voice",
+                status: ms <= pass ? "PASS" : ms <= warn ? "WARN" : "FAIL",
+                detail: `first sound after ${ms} ms (${cold.firstAudioMs ?? 0} ms when the voice had to load first)`,
+                metrics: { firstSoundMs: ms, firstSoundColdMs: cold.firstAudioMs ?? 0 },
+            });
+        }
+        t = Date.now();
+        const heardColdText = await transcribe(spoken, model);
+        const coldMs = Date.now() - t;
+        t = Date.now();
+        const heard = await transcribe(spoken, model); // again, now that the model is loaded
+        const warmMs = Date.now() - t;
+        const fileScore = wordErrorRate(DOCTOR_SENTENCE, heard || heardColdText);
+        const fileStatus = grade(fileScore.wer, LIMITS.file);
+        add({
+            name: "Speech → text",
+            status: fileStatus,
+            detail: `${pct(1 - fileScore.wer)} of words right (heard: "${heard}") — voice ${synthMs} ms, transcribe ${warmMs} ms (first ${coldMs} ms)`,
+            metrics: { wer: fileScore.wer, engine: cold.engine, synthMs, transcribeColdMs: coldMs, transcribeWarmMs: warmMs, audioSeconds: await wavSeconds(spoken) },
+        });
+        // 3. Speaker → mic, through the air
+        if (opts.loopback === false || !IS_MAC) {
+            add({ name: "Speaker → mic", status: "SKIP", detail: IS_MAC ? "skipped (--no-loopback)" : "macOS only" });
+        }
+        else {
+            const recorded = path.join(tmp, "recorded.wav");
+            const seconds = (await wavSeconds(spoken)) + 1.5;
+            let player;
+            const rec = await recordForSeconds(seconds, recorded, () => {
+                // The live voice at full quality, the way speak_and_listen speaks — not the 16 kHz test file.
+                player ??= new Promise((r) => setTimeout(r, 300)).then(() => speak(DOCTOR_SENTENCE));
+            });
+            await player;
+            if (rec.digitalSilence) {
+                add({
+                    name: "Speaker → mic",
+                    status: "FAIL",
+                    detail: "the mic returned pure silence — allow microphone access for this app (System Settings → Privacy & Security → Microphone), or check the input device",
+                    metrics: { peakDb: rec.peakDb },
+                });
+            }
+            else if (rec.peakDb < -50) {
+                add({
+                    name: "Speaker → mic",
+                    status: "WARN",
+                    detail: `almost nothing reached the mic (peak ${Math.round(rec.peakDb)} dB). Expected with headphones — otherwise turn the volume up and run again`,
+                    metrics: { peakDb: rec.peakDb },
+                });
+            }
+            else {
+                t = Date.now();
+                const heardAir = await transcribe(recorded, model);
+                const airMs = Date.now() - t;
+                const air = wordErrorRate(DOCTOR_SENTENCE, heardAir);
+                add({
+                    name: "Speaker → mic",
+                    status: grade(air.wer, LIMITS.loopback),
+                    detail: `${pct(1 - air.wer)} of words right through the air (heard: "${heardAir}"), peak ${Math.round(rec.peakDb)} dB, transcribe ${airMs} ms`,
+                    metrics: { wer: air.wer, peakDb: rec.peakDb, transcribeMs: airMs },
+                });
+            }
+        }
+    }
+    catch (err) {
+        if (!(err instanceof StopChecks)) {
+            add({ name: "Unexpected error", status: "FAIL", detail: err instanceof Error ? err.message : String(err) });
+        }
+    }
+    finally {
+        stopWhisperServer();
+        stopKokoro();
+        await rm(tmp, { recursive: true, force: true });
+    }
+    report.ok = checks.every((c) => c.status !== "FAIL");
+    const dir = opts.reportDir ?? path.join(CONFIG.cacheDir, "doctor");
+    try {
+        await mkdir(dir, { recursive: true });
+        report.reportFile = path.join(dir, `doctor-${report.date.replace(/[:.]/g, "-")}.json`);
+        await writeFile(report.reportFile, JSON.stringify(report, null, 2) + "\n");
+    }
+    catch {
+        /* the report file is a convenience */
+    }
+    return report;
+}

package/dist/index.js

@@ -23,16 +23,25 @@
  *   speech-text.ts  screen text → ear text; transcript clean-up        (pure)
  *   endpointer.ts   turn-taking voice-activity detection               (pure)
  *   audio.ts        TTS, chimes, streaming microphone capture
+ *   kokoro.ts       the optional Kokoro voice: install, warm worker, streaming playback
+ *   kokoro-worker.ts  the process that runs the Kokoro model
+ *   kokoro-text.ts  pronunciation fixes and sentence chunks for Kokoro (pure)
+ *   pcm.ts          PCM conversion and resampling (pure)
  *   model.ts        find / symlink / download the whisper model
  *   stt.ts          warm whisper-server + whisper-cli fallback
  *   setup.ts        requirement checks and consent-based installs
+ *   lock.ts         one voice turn at a time across every session on the Mac
  *   voice.ts        the speak → listen → transcribe round trip
+ *   wer.ts          word error rate (pure)
+ *   doctor.ts       objective self-check with PASS / WARN / FAIL
  *   texts.ts        everything the model reads (rules, instructions, prompts)
  *   server.ts       MCP tools and prompts
  */
 import { createInterface } from "node:readline/promises";
 import { DEFAULT_LISTEN_SECONDS, envNum, log, PKG } from "./config.js";
 import { startServer } from "./server.js";
+import { runDoctor } from "./doctor.js";
+import { stopKokoro } from "./kokoro.js";
 import { runSetupFlow } from "./setup.js";
 import { stopWhisperServer } from "./stt.js";
 import { speakAndListen } from "./voice.js";
@@ -43,7 +52,10 @@ Usage:
   npx -y ${PKG.name}                  Start the MCP server on stdio (what MCP clients run)
   npx -y ${PKG.name} setup            Check what's installed; offers to install what's missing
   npx -y ${PKG.name} setup --install  Install what's missing without asking (brew + model download)
+  npx -y ${PKG.name} setup --kokoro   Install the optional Kokoro voice (about 1 GB on disk)
   npx -y ${PKG.name} test ["text"]    Speak, listen for one turn and print the transcript
+  npx -y ${PKG.name} doctor           Self-check: setup, voice → transcript, speakers → mic (PASS/WARN/FAIL)
+  npx -y ${PKG.name} doctor --no-loopback   …without playing sound through the speakers
   npx -y ${PKG.name} --version
 
 Environment variables (all optional):
@@ -53,8 +65,11 @@ Environment variables (all optional):
   VOICE_MCP_LANGUAGE               spoken language: en, th, de, … or auto
   VOICE_MCP_WHISPER_PROMPT         words to bias recognition toward (names, jargon)
   VOICE_MCP_VOICE / _RATE          macOS say voice and words-per-minute
+  VOICE_MCP_TTS                    auto (Kokoro once installed) | say | kokoro
+  VOICE_MCP_KOKORO_VOICE           Kokoro voice (default af_heart; e.g. af_bella, am_michael, bf_emma)
+  VOICE_MCP_KOKORO_SPEED           Kokoro speed, 0.5–2 (default 1)
   VOICE_MCP_END_SILENCE_MS         pause length that ends your turn (default 1200)
-  VOICE_MCP_START_TIMEOUT_SECONDS  wait this long for you to start talking (default 8)
+  VOICE_MCP_START_TIMEOUT_SECONDS  wait this long for you to start talking (default 15)
   VOICE_MCP_SPEECH_MARGIN_DB       how far above room noise counts as speech (default 12)
   VOICE_MCP_WHISPER_SERVER         0 to always use whisper-cli (no warm server)
   VOICE_MCP_SERVER_IDLE_MINUTES    stop the warm server after this idle time (default 15)
@@ -64,7 +79,9 @@ Environment variables (all optional):
 async function runSetupCli(args) {
     console.error(`${PKG.name} v${PKG.version} on ${process.platform}/${process.arch}, Node ${process.version}\n`);
     const force = args.includes("--install") || args.includes("-y");
-    let outcome = await runSetupFlow(force, (m) => console.error(`  … ${m}`));
+    const kokoro = args.includes("--kokoro");
+    const onProgress = (m) => console.error(`  … ${m}`);
+    let outcome = await runSetupFlow({ install: force, kokoro, onProgress });
     // Interactive terminal: offer to install right away.
     if (!force && !outcome.ready && !outcome.installing && process.stdin.isTTY && /ask the user whether/.test(outcome.report)) {
         console.error(outcome.report.replace(/\nNext:.*$/s, ""));
@@ -73,14 +90,27 @@ async function runSetupCli(args) {
         rl.close();
         if (answer !== "" && !answer.startsWith("y"))
             return 1;
-        outcome = await runSetupFlow(true, (m) => console.error(`  … ${m}`));
+        outcome = await runSetupFlow({ install: true, onProgress });
     }
     // In a terminal we can simply wait for background installs to finish.
     while (outcome.installing)
-        outcome = await runSetupFlow(false, (m) => console.error(`  … ${m}`));
+        outcome = await runSetupFlow({ onProgress });
     console.error(outcome.report);
     return outcome.ready ? 0 : 1;
 }
+async function runDoctorCli(args) {
+    console.error(`${PKG.name} v${PKG.version} doctor — checking the voice pipeline on this machine\n`);
+    const loopback = !args.includes("--no-loopback");
+    if (loopback)
+        console.error("You'll hear a short sentence from the speakers. Keep the room quiet and don't talk.\n");
+    const report = await runDoctor({ loopback, log: (line) => console.error(line) });
+    console.error(`\n${report.ok ? "All checks passed." : "Some checks failed — see the details above."}`);
+    if (report.reportFile)
+        console.error(`Report saved to ${report.reportFile}`);
+    if (args.includes("--json"))
+        process.stdout.write(JSON.stringify(report, null, 2) + "\n");
+    return report.ok ? 0 : 1;
+}
 async function runTestCli(text) {
     const prompt = text || "Voice bridge test. Say something after the chime, and I'll print what I heard.";
     try {
@@ -99,6 +129,7 @@ async function runTestCli(text) {
     }
     finally {
         stopWhisperServer();
+        stopKokoro();
     }
 }
 async function main() {
@@ -108,8 +139,9 @@ async function main() {
         case "serve":
             return startServer();
         case "setup":
-        case "doctor":
             process.exit(await runSetupCli(rest));
+        case "doctor":
+            process.exit(await runDoctorCli(rest));
         case "test":
             process.exit(await runTestCli(rest.join(" ").trim()));
         case "-v":