@codexstar/pi-listen 1.0.4

package/docs/API.md

@@ -0,0 +1,273 @@
+# pi-listen — API Documentation
+
+## Extension API
+
+pi-listen registers as a Pi extension via the `pi` field in `package.json`:
+
+```json
+{
+  "pi": {
+    "extensions": ["./extensions/voice.ts"]
+  }
+}
+```
+
+The extension exports a default function that receives the Pi `ExtensionAPI`.
+
+---
+
+## Daemon Protocol
+
+The STT daemon communicates via Unix domain sockets using newline-delimited JSON.
+
+### Socket Location
+
+Socket path is derived from config hash:
+```
+/tmp/pi-voice-<sha1_12chars>.sock
+```
+
+### Commands
+
+#### `ping`
+
+Check if daemon is alive.
+
+**Request:**
+```json
+{"cmd": "ping"}
+```
+
+**Response:**
+```json
+{"status": "ok", "pid": 12345}
+```
+
+---
+
+#### `status`
+
+Get daemon status and loaded model info.
+
+**Request:**
+```json
+{"cmd": "status"}
+```
+
+**Response:**
+```json
+{
+  "status": "running",
+  "pid": 12345,
+  "uptime": 120.5,
+  "requests": 42,
+  "idle": 5.2,
+  "backend": "faster-whisper",
+  "model": "small",
+  "model_loaded": true
+}
+```
+
+---
+
+#### `load`
+
+Load or switch to a specific backend and model.
+
+**Request:**
+```json
+{"cmd": "load", "backend": "faster-whisper", "model": "small"}
+```
+
+**Response:**
+```json
+{"status": "loaded", "backend": "faster-whisper", "model": "small", "load_time": 2.34}
+```
+
+Or if already loaded:
+```json
+{"status": "already_loaded", "backend": "faster-whisper", "model": "small"}
+```
+
+---
+
+#### `transcribe`
+
+Transcribe an audio file.
+
+**Request:**
+```json
+{
+  "cmd": "transcribe",
+  "audio": "/path/to/file.wav",
+  "language": "en",
+  "vad": true
+}
+```
+
+**Response:**
+```json
+{
+  "text": "hello world",
+  "duration": 1.23,
+  "backend": "faster-whisper",
+  "model": "small",
+  "language": "en"
+}
+```
+
+With VAD skip (no speech detected):
+```json
+{
+  "text": "",
+  "duration": 0,
+  "vad": {"has_speech": false, "vad_available": true, "segments": 0},
+  "skipped": true
+}
+```
+
+---
+
+#### `vad`
+
+Run Voice Activity Detection only (no transcription).
+
+**Request:**
+```json
+{"cmd": "vad", "audio": "/path/to/file.wav"}
+```
+
+**Response:**
+```json
+{
+  "has_speech": true,
+  "vad_available": true,
+  "segments": 3,
+  "speech_duration_ms": 4200
+}
+```
+
+---
+
+#### `backends`
+
+List all available backends and their status.
+
+**Request:**
+```json
+{"cmd": "backends"}
+```
+
+**Response:**
+```json
+{
+  "backends": [
+    {
+      "name": "faster-whisper",
+      "available": true,
+      "type": "local",
+      "default_model": "small",
+      "models": ["tiny", "base", "small", "medium", "large-v3"]
+    }
+  ]
+}
+```
+
+---
+
+#### `shutdown`
+
+Gracefully stop the daemon.
+
+**Request:**
+```json
+{"cmd": "shutdown"}
+```
+
+**Response:**
+```json
+{"status": "shutting_down"}
+```
+
+---
+
+## CLI Usage
+
+### transcribe.py
+
+```bash
+# Transcribe with auto-detected backend
+python3 transcribe.py audio.wav
+
+# Specify backend and model
+python3 transcribe.py --backend faster-whisper --model small audio.wav
+
+# Specify language
+python3 transcribe.py --language auto audio.wav
+
+# List available backends
+python3 transcribe.py --list-backends
+
+# List models for a backend
+python3 transcribe.py --list-models --backend faster-whisper
+```
+
+### daemon.py
+
+```bash
+# Start daemon
+python3 daemon.py start --backend faster-whisper --model small
+
+# Start with custom socket path
+python3 daemon.py start --socket /tmp/my-daemon.sock
+
+# Check status
+python3 daemon.py status
+
+# Transcribe via daemon
+python3 daemon.py transcribe audio.wav --language en --vad
+
+# Load a different model
+python3 daemon.py load --backend moonshine --model moonshine/base
+
+# Stop daemon
+python3 daemon.py stop
+
+# Ping daemon
+python3 daemon.py ping
+```
+
+---
+
+## Error Responses
+
+All errors follow this format:
+
+```json
+{"error": "Human-readable error message"}
+```
+
+Common errors:
+
+| Error | Cause |
+|-------|-------|
+| `"Audio file not found: /path"` | Audio file doesn't exist |
+| `"No model loaded. Send 'load' first."` | Transcribe called before loading a model |
+| `"No STT backend found"` | No backend is installed or available |
+| `"Unknown command: foo"` | Invalid command sent to daemon |
+| `"Message exceeds maximum size"` | Request larger than 1 MB |
+| `"Daemon not running"` | Socket connection failed |
+| `"Daemon timeout"` | Daemon didn't respond within timeout |
+
+---
+
+## Audio Requirements
+
+| Property | Value |
+|----------|-------|
+| Format | WAV |
+| Sample rate | 16000 Hz |
+| Channels | 1 (mono) |
+| Bit depth | 16-bit |
+
+Audio is recorded via SoX's `rec` command with these parameters automatically.

package/docs/ARCHITECTURE.md

@@ -0,0 +1,114 @@
+# pi-listen Architecture
+
+## System Overview
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                           Pi CLI (Node.js)                               │
+│                                                                          │
+│  ┌─────────────────────────────────────────────────────────────────────┐ │
+│  │                      extensions/voice.ts                            │ │
+│  │                                                                     │ │
+│  │  ┌───────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │ │
+│  │  │ Recording  │  │ Daemon   │  │   BTW    │  │   Hold-to-Talk   │  │ │
+│  │  │ (SoX rec)  │  │   IPC    │  │ Threads  │  │  (Kitty proto)   │  │ │
+│  │  └─────┬─────┘  └────┬─────┘  └────┬─────┘  └────────┬─────────┘  │ │
+│  │        │              │             │                  │            │ │
+│  └────────┼──────────────┼─────────────┼──────────────────┼────────────┘ │
+│           │              │             │                  │              │
+│  ┌────────┼──────────────┼─────────────┼──────────────────┼────────────┐ │
+│  │        │     voice/config.ts   voice/diagnostics.ts                 │ │
+│  │        │     voice/onboarding.ts  voice/install.ts                  │ │
+│  └────────┼──────────────┼─────────────────────────────────────────────┘ │
+│           │              │                                               │
+└───────────┼──────────────┼───────────────────────────────────────────────┘
+            │              │ Unix Domain Socket
+            │              │ (newline-delimited JSON)
+            │              ▼
+┌───────────┼──────────────────────────────────────────────────────────────┐
+│           │          daemon.py (Python, persistent)                       │
+│           │                                                              │
+│  ┌────────▼────────┐  ┌─────────────────┐  ┌──────────────────────────┐ │
+│  │   Audio Files    │  │   ModelCache    │  │    DaemonServer          │ │
+│  │   (WAV, /tmp)    │  │  (thread-safe)  │  │  (socket, idle timeout)  │ │
+│  └─────────────────┘  └────────┬────────┘  └──────────────────────────┘ │
+│                                │                                         │
+│                       ┌────────▼────────┐                                │
+│                       │  transcribe.py  │                                │
+│                       │   (5 backends)  │                                │
+│                       └────────┬────────┘                                │
+│                                │                                         │
+│              ┌─────────────────┼─────────────────────┐                   │
+│              ▼                 ▼                     ▼                   │
+│  ┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐             │
+│  │  faster-whisper   │ │  whisper-cpp │ │    Deepgram      │             │
+│  │  moonshine        │ │              │ │  (cloud API)     │             │
+│  │  parakeet         │ │              │ │                  │             │
+│  └──────────────────┘ └──────────────┘ └──────────────────┘             │
+└──────────────────────────────────────────────────────────────────────────┘
+```
+
+## Data Flow
+
+### Voice Input Flow
+
+```
+1. User holds SPACE (or Ctrl+Shift+V)
+2. voice.ts spawns `rec` (SoX) → writes WAV to /tmp
+3. User releases key
+4. voice.ts kills `rec`
+5. voice.ts sends {"cmd": "transcribe", "audio": "/tmp/pi-voice-XXX.wav"} to daemon
+6. daemon.py runs transcription via loaded model
+7. daemon.py returns {"text": "...", "duration": 1.23}
+8. voice.ts injects text into Pi editor
+9. voice.ts deletes temp WAV file
+```
+
+### BTW Flow
+
+```
+1. User types /btw <message> (or holds Ctrl+Shift+B for voice)
+2. voice.ts builds context from system prompt + prior BTW thread
+3. voice.ts streams LLM response via pi-ai streamSimple()
+4. Response displayed in BTW widget above editor
+5. User can /btw:inject to push thread into main agent context
+```
+
+### Daemon Lifecycle
+
+```
+1. Extension starts → ensureDaemon() checks if daemon is running
+2. If not running → spawns python3 daemon.py start --socket <path>
+3. Daemon pre-loads configured backend + model
+4. Daemon serves requests on Unix socket
+5. After 5 minutes of inactivity → auto-shutdown
+6. On session_shutdown → extension does NOT kill daemon (persists for reuse)
+```
+
+## Thread Safety
+
+- **ModelCache**: Protected by `threading.Lock()` — only one transcription runs at a time
+- **DaemonServer**: Each client connection handled in a separate daemon thread
+- **voice.ts state**: Single-threaded Node.js event loop — no mutex needed, but async races are possible (documented in bug-hunter audit)
+
+## Configuration Resolution
+
+```
+1. Check project settings: <cwd>/.pi/settings.json → voice key
+2. Check global settings: ~/.pi/agent/settings.json → voice key
+3. Fall back to DEFAULT_CONFIG (auto backend, small model, enabled)
+```
+
+Project settings override global settings completely (no merging).
+
+## Security Boundaries
+
+| Boundary | Protection |
+|----------|------------|
+| Unix socket | Local-only (no TCP), file permissions |
+| Message size | MAX_MSG_SIZE = 1 MB |
+| Audio files | Validated with os.path.exists() |
+| Backend names | Validated against BACKENDS registry |
+| Error responses | No stack traces (logged to stderr only) |
+| Idle timeout | Daemon auto-shuts down after 5 minutes |
+| Temp files | Deleted immediately after transcription |

package/docs/backends.md

@@ -0,0 +1,196 @@
+# pi-voice backend guide
+
+`pi-voice` supports both **local** and **cloud** speech-to-text (STT) backends.
+
+Today, the extension exposes backend selection through `/voice setup` and reads settings from:
+
+- global: `~/.pi/agent/settings.json`
+- project: `.pi/settings.json`
+
+Project settings override global settings when both are present.
+
+## Quick recommendation
+
+If you want a conservative default:
+
+- choose **faster-whisper** for a strong local default on macOS and general desktop use
+- choose **Deepgram** if you want the fastest route to a working setup and you already have an API key
+
+If onboarding tells you a model is already installed, prefer that ready-now path unless you have a strong reason to switch.
+
+## Backend comparison
+
+| Backend | Mode | Best for | Tradeoffs | Typical install path | Model detection confidence |
+|---|---|---|---|---|---|
+| `faster-whisper` | Local | Best overall local default, good quality/speed balance | Python dependency, model download time, CPU usage | `pip install faster-whisper` | High |
+| `moonshine` | Local | Lower-resource local experiments | Smaller ecosystem, fewer model choices | `pip install useful-moonshine[onnx]` | Medium / heuristic |
+| `whisper-cpp` | Local | CLI-oriented local setups, users already invested in whisper.cpp | Model file management is more manual | `brew install whisper-cpp` | Very high |
+| `deepgram` | Cloud | Fastest setup if API key already exists | Audio leaves the machine, network required, API billing | set `DEEPGRAM_API_KEY` | API-ready vs missing key |
+| `parakeet` | Local | NVIDIA/NeMo-oriented experimentation | Heavy dependency footprint, slower setup | `pip install nemo_toolkit[asr]` | Medium-low / heuristic |
+
+## Backend details
+
+## faster-whisper
+
+**Use this when:**
+- you want a high-confidence local default
+- privacy/offline use matters
+- you are okay with Python-based dependencies
+
+**Pros**
+- mature Whisper-family local option
+- strong model selection range
+- fits the current `pi-voice` architecture well
+
+**Cons**
+- first install can take longer than cloud setup
+- larger models increase local CPU and startup costs
+
+**Suggested starting models**
+- `small` — good conservative starting point
+- `small.en` — useful for English-only setups
+- `medium` — move here if you want more accuracy and can accept higher cost
+
+## moonshine
+
+**Use this when:**
+- you want a lighter local option
+- you are willing to trade maturity and flexibility for smaller setup/runtime needs
+
+**Pros**
+- lightweight local direction
+- fewer model decisions
+
+**Cons**
+- narrower model ecosystem
+- less common than Whisper-family defaults
+
+**Suggested starting models**
+- `moonshine/base` — safer default
+- `moonshine/tiny` — lowest-cost experiment path
+
+## whisper-cpp
+
+**Use this when:**
+- you already use whisper.cpp
+- you prefer CLI-driven local tooling
+- you want to manage model files more explicitly
+
+**Pros**
+- straightforward local CLI model
+- familiar to users already in the whisper.cpp ecosystem
+
+**Cons**
+- model file placement matters
+- setup can feel more manual than Python-backed alternatives
+
+**Suggested starting models**
+- `small` — practical starting point
+- `small.en` — good English-only choice
+
+## deepgram
+
+**Use this when:**
+- you want the fastest route to a working setup
+- you are comfortable with cloud transcription
+- you already have or plan to add `DEEPGRAM_API_KEY`
+
+**Pros**
+- minimal local dependency burden
+- fast setup when credentials are ready
+- good fit for users optimizing for time-to-value
+
+**Cons**
+- requires network connectivity
+- audio is sent to a cloud provider
+- subject to account limits and billing
+
+**Suggested starting model**
+- `nova-3` — the current default in `transcribe.py`
+
+## parakeet
+
+**Use this when:**
+- you specifically want to experiment with NVIDIA NeMo/Parakeet
+- you are comfortable with a heavier Python stack
+
+**Pros**
+- interesting specialized option
+- useful for experimentation in the NeMo ecosystem
+
+**Cons**
+- heavier setup than the other paths
+- not the best default for a first-time user
+
+## Choosing local vs cloud
+
+## Choose local when
+- privacy matters more than setup speed
+- you want offline use
+- you are comfortable installing dependencies
+- you want per-project control without relying on network services
+
+## Choose cloud when
+- you want the fastest setup path
+- you are okay with a provider API key
+- you want to avoid local model/dependency installation
+- you are comfortable with network and billing constraints
+
+## Model selection guidance
+
+The current package exposes raw backend model names plus model-aware status where it can detect it.
+
+A conservative way to choose:
+
+- prefer **already installed** models first when they meet your needs
+- start small: `small`, `small.en`, or backend defaults
+- only move up if the current model is not accurate enough
+- prefer English-only variants when your use case is English-only and the backend provides them
+- avoid jumping to the biggest model first unless you already know your machine can handle it
+
+## How to read model status
+
+During onboarding and in some diagnostics output, local models can show up as:
+
+- **installed** — the model appears to already exist locally and should not need a fresh download
+- **recommended, installed** — same as above, and also the preferred choice for the current path
+- **download required** — backend is available, but the selected model does not appear to be present
+- **unknown** — backend is present, but model presence could not be confirmed confidently
+- **api** — cloud path; no local model download applies
+
+For `unknown`, do not assume the system is broken. It means `pi-voice` is being conservative and avoiding a false claim.
+
+## Configuration scope
+
+`pi-voice` now supports both:
+
+- **Global scope** — applies across projects
+- **Project scope** — applies only in the current repository
+
+Use **project scope** when:
+- one repo needs a different voice workflow
+- you want a safer rollout before using the same config everywhere
+- you need local STT in one project but cloud STT in another
+
+## Inspect current backend state
+
+Useful commands:
+
+- `/voice info` — show current effective config and selected model status
+- `/voice backends` — list available backends, installed models, and install detection hints
+- `/voice setup` — choose backend and model interactively
+- `/voice test` — do a basic voice environment check plus current model readiness
+- `/voice doctor` — compare current setup repair steps with a recommended alternative
+- `/voice daemon status` — inspect the warm daemon state
+
+## Current limitations
+
+Be conservative about expectations:
+
+- backend installation is still partly toolchain-dependent
+- some setup steps may still require manual `brew` or `pip` work
+- not every backend has equally strong model-detection confidence
+- `unknown` model status is intentional when the system cannot verify a local cache safely
+- backend available does **not** always mean every model is already ready locally
+
+If you hit issues, see [`docs/troubleshooting.md`](./troubleshooting.md).