voice-mcp-server 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Erick Vazquez Santillan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,193 @@
+<div align="center">
+
+# 🎙️ Voice MCP Server
+
+**Give your AI agents a voice, real ears, and the ability to handle interruptions in real-time.**
+
+[![npm version](https://img.shields.io/npm/v/voice-mcp-server.svg?color=red&style=flat-square&logo=npm)](https://www.npmjs.com/package/voice-mcp-server)
+[![Platform: macOS Apple Silicon](https://img.shields.io/badge/Platform-macOS%20%7C%20Apple%20Silicon-lightgrey?style=flat-square&logo=apple)](#-target-environment)
+[![Python](https://img.shields.io/badge/Python-3.10%2B-blue?logo=python&style=flat-square)](https://python.org)
+[![MCP Compatible](https://img.shields.io/badge/MCP-Compatible-success?style=flat-square)](https://modelcontextprotocol.io/)
+
+</div>
+
+---
+
+## 💡 The Pitch
+
+Typical AI assistants generate passive blocks of text. **Voice MCP Server** changes the paradigm by granting [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) compatible agents (like Gemini, Claude, or Cursor) the ability to actively **speak and listen in real-time**.
+
+Instead of a standard text response, the AI can initiate a **bidirectional voice loop** with you. Featuring blazing-fast local transcription and true human-like **barge-in (interruption) detection**, your AI will naturally stop talking the moment you interrupt it, transcribe what you said, and gracefully pivot the conversation.
+
+---
+
+## 🏗️ Architecture: The Best of Both Worlds
+
+To make this server universally compatible via NPM while securely wielding heavy Python machine-learning models, Voice MCP Server utilizes a hybrid architecture.
+
+```mermaid
+graph TD
+    Client[MCP Client<br>Cursor, Claude, Gemini] <-->|stdio| NodeBridge(Node.js Bridge<br>npx voice-mcp-server)
+    
+    subgraph Core System
+        NodeBridge <-->|Spawns & Pipes stdio| PythonEngine[Python Core Engine<br>fastmcp]
+        PythonEngine <-->|Unix Domain Sockets<br>~/Library/Application Support/VoiceMCP| AudioDaemon[Audio & ML Daemon]
+    end
+
+    subgraph Hot-Swappable Adapters
+        AudioDaemon -.-> TTS[TTS: say / kokoro / elevenlabs]
+        AudioDaemon -.-> STT[STT: mlx_whisper_large_v3]
+        AudioDaemon -.-> VAD[VAD: silero / ptt]
+        AudioDaemon -.-> MIC[Mic: pyaudio]
+    end
+```
+
+1.  **The Entry Point (Node.js):** Distributed as a standard NPM package (`voice-mcp-server`). Running `npx voice-mcp-server` spins up a TypeScript bridge that smoothly interfaces with standard MCP `stdio` requirements. It automatically locates your Python environment and spawns the core engine.
+2.  **The Core Engine (Python):** Powered by the `fastmcp` framework, a Python daemon manages complex agent logic, ML inference, and tool execution.
+3.  **Firewall-Friendly Sockets:** The internal audio daemon communicates entirely via Unix Domain Sockets isolated in `~/Library/Application Support/VoiceMCP`. This ensures zero annoying macOS *"Do you want to allow this application to accept network connections?"* popups during local development.
+
+-----
+
+## 💻 Target Environment
+
+> [!WARNING]  
+> **Apple Silicon Required:** Because this project heavily relies on `mlx_whisper_large_v3` for hardware-accelerated local Speech-to-Text and macOS-native audio commands (like `say`), it is currently highly optimized for, and restricted to, **macOS systems with M-series chips (M1/M2/M3/M4)**.
+
+-----
+
+## 🧩 Hot-Swappable Hardware & AI Adapters
+
+The system is built on a highly modular adapter pattern configured via `hydra` YAML files. **The AI can dynamically swap these out at runtime without restarting the server.**
+
+| Component | Available Adapters | Description |
+| :--- | :--- | :--- |
+| **🔊 Speakers (TTS)** | `live_speaker` | Blazing-fast, zero-latency native macOS `say` command. |
+| | `kokoro_speaker` | High-quality, emotive local ML Text-to-Speech. |
+| | `elevenlabs_speaker` | Premium cloud-based ultra-realistic voices. |
+| **🎙️ Microphones** | `live_mic` | Direct hardware integration via PyAudio. |
+| **🤫 VAD (Activity)** | `silero_vad` | Conversational mode powered by Silero, heavily optimized for 1-second barge-ins. |
+| | `ptt_vad` | Manual Push-to-Talk / Walkie-Talkie mode for noisy environments. |
+| **📝 STT (Transcription)**| `mlx_whisper_large_v3`| Blazing fast local transcription leveraging Apple's MLX framework. |
+
+-----
+
+## 🛠️ Tools Exposed to the LLM
+
+Once connected, the server equips your AI agent with two powerful MCP tools:
+
+### 1. `voice_converse`
+
+The core communication loop. The AI calls this tool and passes a string of text it wants to say.
+
+1.  The server renders and plays the TTS.
+2.  The server instantly activates the microphone and listens for the user's reply via VAD.
+3.  The server transcribes the audio and returns the text to the AI.
+
+**Interrupt Handling (Barge-in):** If the user interrupts the AI mid-sentence, playback instantly stops. The server captures the interruption, transcribes it, and returns the response alongside a `was_interrupted: true` flag. This allows the AI to organically realize it was cut off and address the interruption naturally.
+
+### 2. `configure_audio_engine`
+
+Grants the AI meta-awareness over its own hardware and software stack. If you ask your AI, *"Switch to a more realistic voice"* or *"Change to push-to-talk mode,"* it can autonomously call this tool to swap out the active Hydra configuration on the fly.
+
+-----
+
+## 🚀 Installation & Setup
+
+Since this project bridges Node.js and Python, you will need to set up the Python environment with the required ML dependencies, followed by the NPM bridge.
+
+### 1. Prerequisites
+
+  * **Node.js** (v18+)
+  * **Python** (3.10+)
+  * **macOS** (M1/M2/M3/M4 chip heavily recommended)
+
+### 2. Setup the Python Environment
+
+```bash
+# Clone the repository
+git clone https://github.com/your-username/voice-mcp-server.git
+cd voice-mcp-server
+
+# Create and activate a virtual environment
+python3 -m venv venv
+source venv/bin/activate
+
+# Install the heavy ML dependencies
+pip install -r requirements.txt
+
+# Download required ML models locally
+# Kokoro, Silero VAD, and MLX Whisper will automatically pull their weights on the first run.
+```
+
+### 3. Install the NPM Bridge
+
+Install the package globally, or simply rely on `npx` when configuring your client:
+
+```bash
+# Build the TypeScript entry point
+npm install
+npm run build
+
+# Install globally
+npm install -g .
+```
+
+-----
+
+## 🔌 Usage: MCP Client Configuration
+
+You can seamlessly plug Voice MCP Server into any standard MCP-compatible client.
+
+### For Gemini CLI
+
+Add the server to your global configuration:
+```bash
+gemini mcp add voice-mcp-server node /absolute/path/to/voice_mcp_server/build/index.js
+```
+
+### For Cursor
+
+1.  Navigate to **Cursor Settings > Features > MCP Server**
+2.  Click **+ Add New MCP Server**
+3.  **Name:** `Voice`
+4.  **Type:** `command`
+5.  **Command:** `node /absolute/path/to/voice_mcp_server/build/index.js`
+
+### For Claude Desktop
+
+Add the following to your `claude_desktop_config.json` (usually located at `~/Library/Application Support/Claude/claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "voice-mcp": {
+      "command": "node",
+      "args": ["/absolute/path/to/voice_mcp_server/build/index.js"]
+    }
+  }
+}
+```
+
+-----
+
+## 💡 Example Prompt
+
+Once connected, test the server by sending this prompt to your AI:
+
+> *"Let's test your voice capabilities! Please use the `voice_converse` tool to introduce yourself and tell me a story about a brave robot. If I interrupt you while you are speaking, stop the story and acknowledge my interruption in your next response."*
+
+-----
+
+## 🤝 Contributing
+
+Contributions, pull requests, and bug reports are highly welcome! Whether you want to add support for Windows/Linux by removing the MLX dependency, build new STT adapters (like Groq or Deepgram), or improve the TTS engines, please open an issue or submit a PR.
+
+1.  Fork the Project
+2.  Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
+3.  Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
+4.  Push to the Branch (`git push origin feature/AmazingFeature`)
+5.  Open a Pull Request
+
+## 📄 License
+
+This project is open-sourced under the MIT License.

package/build/index.js

@@ -0,0 +1,51 @@
+#!/usr/bin/env node
+import { spawn } from "node:child_process";
+import { join, dirname } from "node:path";
+import { fileURLToPath } from "node:url";
+import { existsSync } from "node:fs";
+// Get the directory of the current module
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = dirname(__filename);
+// Root of the project
+const projectRoot = join(__dirname, "..");
+// Path to the Python script
+const pythonScriptPath = join(projectRoot, "src", "mcp_server.py");
+/**
+ * Locate the best Python executable to use.
+ * Priority:
+ * 1. Local venv inside the project
+ * 2. System python3
+ */
+function getPythonExecutable() {
+    const venvPath = join(projectRoot, "venv", "bin", "python3");
+    if (existsSync(venvPath)) {
+        return venvPath;
+    }
+    return "python3";
+}
+const pythonExecutable = getPythonExecutable();
+/**
+ * Start the Python MCP Server and bridge standard I/O.
+ */
+function startBridge() {
+    const pythonProcess = spawn(pythonExecutable, [pythonScriptPath], {
+        stdio: ["pipe", "pipe", "inherit"],
+        env: {
+            ...process.env,
+            // Ensure Python output isn't buffered
+            PYTHONUNBUFFERED: "1",
+        },
+    });
+    // Pipe our stdin into Python's stdin
+    process.stdin.pipe(pythonProcess.stdin);
+    // Pipe Python's stdout back to our stdout
+    pythonProcess.stdout.pipe(process.stdout);
+    // Handle process termination
+    pythonProcess.on("exit", (code) => {
+        process.exit(code ?? 0);
+    });
+    // Forward signals
+    process.on("SIGINT", () => pythonProcess.kill("SIGINT"));
+    process.on("SIGTERM", () => pythonProcess.kill("SIGTERM"));
+}
+startBridge();

package/config/config.yaml

@@ -0,0 +1,25 @@
+# --- Global Adapter Configuration ---
+# Use these defaults to instantly swap out hardware and AI models.
+# The configurations for each adapter are loaded dynamically from their respective folders.
+defaults:
+  # Available microphones: live_mic (PyAudio real hardware)
+  - microphone: live_mic
+  
+  # Available speakers: kokoro_speaker (local TTS), elevenlabs_speaker (cloud TTS), live_speaker (macOS 'say')
+  - speaker: kokoro_speaker
+  
+  # Available VADs: 
+  #   - ptt_vad: Walkie-Talkie mode (Hold 'Shift' to talk. Instant response. Ignores TV/noise).
+  #   - silero_vad: Conversational AI mode (Listens automatically. Tuned for 1-second barge-ins).
+  - vad: ptt_vad
+  
+  # Available STTs: mlx_whisper_large_v3 (Apple Silicon GPU), whisper_stt (Google Cloud API)
+  - stt: mlx_whisper_large_v3
+  
+  - _self_
+
+# --- Global Engine Configuration ---
+# Note: VAD-specific tuning (like barge-in thresholds and probability) 
+# is now colocated inside the specific VAD yaml files (e.g. config/vad/ptt_vad.yaml).
+config:
+  vad_silence_grace_ms: 100

package/config/microphone/live_mic.yaml

@@ -0,0 +1 @@
+_target_: adapters_real.live_mic.LiveMicrophone

package/config/speaker/elevenlabs_speaker.yaml

@@ -0,0 +1,3 @@
+_target_: adapters_real.elevenlabs_speaker.ElevenLabsSpeaker
+wpm: 150
+voice_id: "aEO01A4wXwd1O8GPgGlF" # Voice: Arabella

package/config/speaker/kokoro_speaker.yaml

@@ -0,0 +1,3 @@
+_target_: adapters_real.kokoro_speaker.KokoroSpeaker
+wpm: 150
+voice: "af_heart"

package/config/stt/mlx_whisper_large_v3.yaml

@@ -0,0 +1,2 @@
+_target_: adapters_real.whisper_stt.RealWhisperSTT
+model_size: "mlx-community/whisper-large-v3-mlx"

package/config/vad/ptt_vad.yaml

@@ -0,0 +1,8 @@
+_target_: adapters_real.ptt_vad.PushToTalkVAD
+key_name: "shift"
+
+# PTT specific tuning (tuned for instant response)
+vad_probability_threshold: 0.50
+vad_bargein_threshold_ms: 100
+endpointing_patience_normal_ms: 300
+endpointing_patience_interrupted_ms: 300

package/config/vad/silero_vad.yaml

@@ -0,0 +1,7 @@
+_target_: adapters_real.silero_vad.RealSileroVAD
+
+# Silero specific tuning (tuned for loud TV background noise)
+vad_probability_threshold: 0.95
+vad_bargein_threshold_ms: 1000
+endpointing_patience_normal_ms: 1500
+endpointing_patience_interrupted_ms: 700

package/package.json

@@ -0,0 +1,40 @@
+{
+  "name": "voice-mcp-server",
+  "version": "0.1.0",
+  "description": "An MCP server to allow LLMs to speak and listen via bidirectional voice loops",
+  "main": "build/index.js",
+  "type": "module",
+  "bin": {
+    "voice-mcp-server": "./build/index.js"
+  },
+  "files": [
+    "build",
+    "src",
+    "config",
+    "requirements.txt",
+    "tsconfig.json"
+  ],
+  "scripts": {
+    "build": "tsc && chmod +x build/index.js",
+    "start": "node build/index.js",
+    "prepublishOnly": "npm run build"
+  },
+  "keywords": [
+    "mcp",
+    "voice",
+    "llm",
+    "audio",
+    "tts",
+    "stt",
+    "vad"
+  ],
+  "author": "Erick Vazquez Santillan",
+  "license": "MIT",
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.5.0"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}

package/requirements.txt

@@ -0,0 +1,126 @@
+accelerate==1.13.0
+addict==2.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+antlr4-python3-runtime==4.9.3
+anyio==4.13.0
+attrs==26.1.0
+babel==2.18.0
+blis==1.3.3
+catalogue==2.0.10
+certifi==2026.2.25
+cffi==2.0.0
+charset-normalizer==3.4.6
+click==8.3.1
+cloudpathlib==0.23.0
+confection==1.3.3
+cryptography==46.0.6
+csvw==3.7.0
+curated-tokenizers==0.0.9
+curated-transformers==0.1.1
+cymem==2.0.13
+dlinfo==2.0.0
+docopt==0.6.2
+elevenlabs==2.40.0
+en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl#sha256=1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85
+espeakng-loader==0.2.4
+fastapi==0.135.2
+filelock==3.25.2
+fsspec==2026.3.0
+h11==0.16.0
+hf-xet==1.4.2
+httpcore==1.0.9
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface_hub==1.8.0
+hydra-core==1.3.2
+idna==3.11
+isodate==0.7.2
+Jinja2==3.1.6
+joblib==1.5.3
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kokoro==0.9.4
+language-tags==1.2.0
+llvmlite==0.46.0
+loguru==0.7.3
+markdown-it-py==4.0.0
+MarkupSafe==3.0.3
+mcp==1.26.0
+mdurl==0.1.2
+misaki==0.9.4
+mlx==0.31.1
+mlx-metal==0.31.1
+mlx-whisper==0.4.3
+more-itertools==10.8.0
+mpmath==1.3.0
+murmurhash==1.0.15
+networkx==3.6.1
+num2words==0.5.14
+numba==0.64.0
+numpy==1.26.4
+omegaconf==2.3.0
+packaging==26.0
+phonemizer-fork==3.3.2
+preshed==3.0.13
+psutil==7.2.2
+PyAudio==0.2.14
+pycparser==3.0
+pydantic==2.12.5
+pydantic-settings==2.13.1
+pydantic_core==2.41.5
+Pygments==2.20.0
+PyJWT==2.12.1
+pynput==1.8.1
+pyobjc-core==12.1
+pyobjc-framework-ApplicationServices==12.1
+pyobjc-framework-Cocoa==12.1
+pyobjc-framework-CoreText==12.1
+pyobjc-framework-Quartz==12.1
+pyparsing==3.3.2
+python-dateutil==2.9.0.post0
+python-dotenv==1.2.2
+python-multipart==0.0.22
+PyYAML==6.0.3
+rdflib==7.6.0
+referencing==0.37.0
+regex==2026.3.32
+requests==2.33.1
+rfc3986==1.5.0
+rich==14.3.3
+rpds-py==0.30.0
+safetensors==0.7.0
+scipy==1.17.1
+segments==2.4.0
+setuptools==81.0.0
+shellingham==1.5.4
+silero-vad==6.2.1
+six==1.17.0
+smart_open==7.5.1
+soundfile==0.13.1
+spacy==3.8.14
+spacy-curated-transformers==0.3.1
+spacy-legacy==3.0.12
+spacy-loggers==1.0.5
+srsly==2.5.3
+sse-starlette==3.3.4
+starlette==1.0.0
+sympy==1.14.0
+termcolor==3.3.0
+thinc==8.3.13
+tiktoken==0.12.0
+tokenizers==0.22.2
+torch==2.11.0
+torchaudio==2.11.0
+tqdm==4.67.3
+transformers==5.4.0
+typer==0.24.1
+typing-inspection==0.4.2
+typing_extensions==4.15.0
+uritemplate==4.2.0
+urllib3==2.6.3
+uvicorn==0.42.0
+wasabi==1.1.3
+weasel==1.0.0
+websockets==16.0
+wrapt==2.1.2

package/src/adapters_real/echo_llm.py

@@ -0,0 +1,28 @@
+from simulation.ports import ILLMBridge
+
+class EchoLLMBridge(ILLMBridge):
+    def __init__(self, latency_ms: int = 0):
+        self.last_call = None
+        self.is_requesting = False
+        self.latency_ms = latency_ms
+        self.current_wait = 0
+
+    def call_mcp_tool(self, context: dict) -> dict:
+        self.start_request(context)
+        return self.get_response()
+
+    def start_request(self, context: dict):
+        self.last_call = context
+        self.is_requesting = True
+        self.current_wait = 0
+
+    def tick(self, ms: int):
+        if self.is_requesting:
+            self.current_wait += ms
+
+    def get_response(self) -> dict | None:
+        if self.is_requesting and self.current_wait >= self.latency_ms:
+            self.is_requesting = False
+            user_speech = self.last_call.get("user_speech", "")
+            return {"text": f"I heard you say: {user_speech}", "expect_reply": True}
+        return None

package/src/adapters_real/elevenlabs_speaker.py

@@ -0,0 +1,117 @@
+import os
+import time
+import subprocess
+import httpx
+from simulation.ports import ISpeaker
+from simulation.models import VirtualAudioFrame
+from dotenv import load_dotenv
+
+load_dotenv()
+
+class ElevenLabsSpeaker(ISpeaker):
+    def __init__(self, wpm=150, voice_id="aEO01A4wXwd1O8GPgGlF"):
+        self.wpm = wpm
+        self.words_per_ms = (wpm / 60) / 1000
+        self.current_text = ""
+        self.words = []
+        self.process = None
+        self.start_time = 0
+        self.voice_id = voice_id
+        self.api_key = os.getenv("ELEVENLABS_API_KEY")
+        self.temp_file = "/tmp/elevenlabs_output.mp3"
+
+    def speak(self, text: str):
+        if not text.strip():
+            return
+
+        self.current_text = text
+        self.words = text.split()
+
+        if self.api_key:
+            url = f"https://api.elevenlabs.io/v1/text-to-speech/{self.voice_id}"
+            headers = {
+                "Accept": "audio/mpeg",
+                "Content-Type": "application/json",
+                "xi-api-key": self.api_key
+            }
+            data = {
+                "text": text,
+                "model_id": "eleven_multilingual_v2",
+                "voice_settings": {
+                    "stability": 0.5,
+                    "similarity_boost": 0.5
+                }
+            }
+            
+            with httpx.Client() as client:
+                try:
+                    response = client.post(url, json=data, headers=headers, timeout=10.0)
+                    response.raise_for_status()
+                    
+                    with open(self.temp_file, "wb") as f:
+                        f.write(response.content)
+                        
+                    # Play the downloaded audio
+                    self.start_time = time.time()
+                    self.process = subprocess.Popen(
+                        ["afplay", self.temp_file],
+                        stdout=subprocess.DEVNULL,
+                        stderr=subprocess.DEVNULL
+                    )
+                except Exception as e:
+                    print(f"ElevenLabs API Error: {e}")
+                    # Fallback to macOS say
+                    self.start_time = time.time()
+                    self.process = subprocess.Popen(
+                        ["say", text],
+                        stdout=subprocess.DEVNULL,
+                        stderr=subprocess.DEVNULL
+                    )
+        else:
+            print("Warning: No ELEVENLABS_API_KEY found, falling back to 'say'")
+            self.start_time = time.time()
+            self.process = subprocess.Popen(
+                ["say", text],
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL
+            )
+
+    def play_frame(self, frame: VirtualAudioFrame):
+        pass
+
+    def tick(self, ms: int):
+        pass
+
+    def is_speaking(self) -> bool:
+        if self.process is None:
+            return False
+        is_running = self.process.poll() is None
+        if not is_running:
+            self.current_text = ""
+            self.words = []
+            self.process = None
+        return is_running
+
+    def has_started_audio(self) -> bool:
+        return self.is_speaking()
+
+    def flush(self) -> str:
+        if not self.is_speaking():
+            return ""
+
+        # Immediately kill the playback process
+        self.process.kill()
+        
+        # Explicitly wait for the process to terminate and reap it
+        self.process.wait()
+        
+        elapsed_ms = (time.time() - self.start_time) * 1000
+        words_spoken = int(elapsed_ms * self.words_per_ms)
+        
+        spoken = " ".join(self.words[:words_spoken])
+        
+        self.current_text = ""
+        self.words = []
+        self.process = None
+        
+        return spoken