@codexstar/pi-listen 1.0.4

package/docs/plans/pi-voice-model-aware-review.md

@@ -0,0 +1,196 @@
+# pi-voice model-aware review
+
+## Scope reviewed
+
+Focused review of the current model-aware phase in:
+- `transcribe.py`
+- `extensions/voice/diagnostics.ts`
+- `extensions/voice/onboarding.ts`
+- `extensions/voice/install.ts`
+- `extensions/voice.ts`
+
+I also sanity-checked the stated behavior against the current implementation shape and looked specifically for:
+- false claims
+- command UX inconsistencies
+- installed-model edge cases
+- regression risks
+
+---
+
+## Summary
+
+The model-aware phase is moving in the right direction:
+- backend discovery now emits installed-model metadata
+- recommendations can prefer an installed local model
+- onboarding labels installed models distinctly
+- provisioning distinguishes between backend-missing and model-missing cases
+- doctor/test/info now surface model-aware state
+
+However, there are still a few important issues before calling this fully product-grade.
+
+---
+
+## Findings
+
+### 1. Onboarding stops offering alternative local backends as soon as any local backend is discovered
+**Severity:** high
+
+**Where:**
+- `extensions/voice/onboarding.ts:81-83`
+
+**What happens:**
+`buildSelectableBackends("local", diagnostics)` returns `discoveredLocalBackends` immediately when *any* local backend is discovered. That means if the machine has only `faster-whisper` installed, onboarding will no longer offer:
+- `moonshine`
+- `whisper-cpp`
+- `parakeet`
+
+with install hints.
+
+**Why it matters:**
+This conflicts with the intended UX:
+- “you already have this, we can configure it now”
+- but still let the user pick another backend if they want
+
+Right now the model-aware path improves the ready-now story, but narrows backend choice too aggressively.
+
+**Recommended fix:**
+Return a merged local backend list:
+- discovered backends first
+- then undiscovered fallback backends with install hints
+- deduplicated by backend name
+
+That preserves the “already installed” path without removing other selectable options.
+
+---
+
+### 2. `faster-whisper` model detection likely produces false negatives for some models
+**Severity:** high
+
+**Where:**
+- `transcribe.py:86-93`
+
+**What happens:**
+`faster_whisper_repo_ids()` assumes:
+- standard models map to `Systran/faster-whisper-<model>`
+- distil models map to `distil-whisper/<model>`
+
+But the actual repo mapping for some models differs. In particular:
+- `large-v3-turbo` is not guaranteed to live under `Systran/faster-whisper-large-v3-turbo`
+- distil model repos are not reliably `distil-whisper/<model>`
+
+**Why it matters:**
+The product will claim “download required” or fail to recognize installed models when they actually already exist.
+
+This undermines the core model-aware onboarding promise.
+
+**Recommended fix:**
+Use the real `faster_whisper` model mapping if available from the library, or maintain an explicit repo map for the supported model IDs instead of constructing repo IDs heuristically.
+
+---
+
+### 3. Heuristic `installed_models` are treated like high-confidence installed models in recommendation ranking
+**Severity:** medium
+
+**Where:**
+- `extensions/voice/diagnostics.ts:89-97`
+- `extensions/voice/diagnostics.ts:118-132`
+
+**What happens:**
+`getPreferredLocalBackend()` prefers the first local backend with any `installed_models`, regardless of detection confidence.
+
+That means a heuristic backend such as:
+- `moonshine`
+- `parakeet`
+
+can outrank a safer backend if its heuristic reports an installed model.
+
+**Why it matters:**
+The code already treats some detectors as low confidence (`unknown` in `getModelReadiness()`), but recommendation ranking does not use that same confidence model.
+
+So the recommendation engine can still over-trust heuristic detections.
+
+**Recommended fix:**
+In recommendation ranking, prefer installed models only for high-confidence detectors first, for example:
+1. high-confidence installed model
+2. high-confidence available backend
+3. heuristic installed model
+4. heuristic available backend
+
+At minimum, gate “already installed and ready to configure” recommendation language behind the same confidence rules used by `getModelReadiness()`.
+
+---
+
+### 4. `/voice backends` output is still backend-centric when a backend is available but no installed models are confirmed
+**Severity:** low
+
+**Where:**
+- `extensions/voice.ts:915-931`
+
+**What happens:**
+If a backend is available and `installed_models` is empty, the output falls back to:
+- `models: <count>`
+
+That is not wrong, but it is weaker than the new model-aware UX elsewhere.
+
+**Why it matters:**
+Users comparing `/voice backends` with onboarding/doctor/test may get less clarity here than they expect.
+
+Example ambiguity:
+- backend ready
+- zero confirmed installed models
+- but output only says `models: 13`
+
+That does not tell the user whether the likely next state is:
+- download required
+- unknown confidence
+- or ready-now via API
+
+**Recommended fix:**
+Use model-aware wording here too, for example:
+- `installed: small, medium`
+- `no confirmed installed models`
+- `model detection: unknown confidence`
+- `api ready`
+
+---
+
+### 5. Moonshine cache detection may under-detect because it only checks directories, not concrete model files
+**Severity:** low
+
+**Where:**
+- `transcribe.py:131-140`
+- `_existing_dirs()` at `transcribe.py:28-34`
+
+**What happens:**
+The moonshine fallback local candidates are passed through `_existing_dirs()`, which only returns existing directories. If a practical moonshine install stores key artifacts as files rather than dedicated directories, this logic will miss them.
+
+**Why it matters:**
+This will most likely cause false negatives rather than false positives, but it still weakens the ready-now experience for moonshine.
+
+**Recommended fix:**
+If moonshine detection remains heuristic, consider checking both:
+- directories
+- likely file artifacts under those roots
+
+Or explicitly document the backend as lower-confidence and keep recommendation weight conservative.
+
+---
+
+## What looks good
+
+These parts are solid and worth keeping:
+- model-aware tests were added before/with behavior changes
+- `getModelReadiness()` now distinguishes high-confidence local detection from heuristic/unknown paths
+- provisioning wording is more conservative for heuristic backends
+- doctor output separates repair current setup from recommended alternative
+- onboarding model labels are much better than before
+
+---
+
+## Recommended priority order
+
+1. **Fix local backend list merging in onboarding**
+2. **Fix `faster-whisper` repo/model mapping**
+3. **Make recommendation ranking confidence-aware**
+4. **Polish `/voice backends` output to match model-aware UX**
+5. **Optionally improve moonshine detection fidelity**

package/docs/plans/pi-voice-model-detection-qa-plan.md

@@ -0,0 +1,226 @@
+# pi-voice model-detection QA / release execution checklist
+
+## Objective
+
+Validate the next onboarding iteration where `pi-voice` can detect already-available models, prefer ready-to-use local setups when appropriate, and clearly distinguish:
+- **already installed / ready now**
+- **backend installed but model missing**
+- **download required**
+- **cloud/API path**
+
+This checklist focuses on release confidence for **model-detection-aware onboarding**, not just the current onboarding baseline.
+
+---
+
+## Release gates
+
+A release is not ready until all four gates pass.
+
+### Gate 1 — Static / automated checks
+- [ ] `bun run check`
+- [ ] model-detection unit tests pass
+- [ ] onboarding recommendation tests pass with installed-model scenarios
+- [ ] provisioning-plan tests pass with installed vs missing model scenarios
+- [ ] any new Python smoke checks for model discovery pass
+
+### Gate 2 — Onboarding behavior
+- [ ] first-run onboarding still launches correctly
+- [ ] onboarding clearly marks installed models as available immediately
+- [ ] onboarding does not ask users to re-download models already present
+- [ ] onboarding still offers alternative models/backends when installed assets exist
+- [ ] onboarding summary accurately reflects whether validation is complete or repair is still required
+
+### Gate 3 — Runtime correctness
+- [ ] selected installed model is actually used at runtime
+- [ ] model-detection state does not get out of sync with daemon/runtime state
+- [ ] project scope vs global scope still works correctly after model-aware setup
+- [ ] `/voice info`, `/voice test`, and `/voice doctor` reflect installed-model state accurately
+
+### Gate 4 — Docs / support readiness
+- [ ] README explains installed-model detection behavior
+- [ ] backend docs explain when cached/existing models may be reused
+- [ ] troubleshooting docs include “backend installed but model missing” and “model detected but validation failed” cases
+- [ ] QA evidence is captured in `docs/qa-results.md`
+
+---
+
+## Test matrix
+
+## A. Fresh install / no cached models
+
+### A1. Fresh startup, no config, no local model assets
+- [ ] onboarding prompt appears
+- [ ] user can choose API or Local
+- [ ] local path marks all local options as requiring install/download
+- [ ] doctor output does not falsely claim any model is already available
+
+### A2. Local mode with nothing installed
+- [ ] onboarding still offers backend choices
+- [ ] install guidance appears for backend + model path
+- [ ] completion state remains `repair` / incomplete until validation succeeds
+
+### A3. API mode with no key
+- [ ] Deepgram path clearly says API key is missing
+- [ ] onboarding does not mislabel API mode as “ready now”
+
+---
+
+## B. Existing installed local model paths
+
+### B1. Backend installed, model already cached
+Example target: `faster-whisper` backend available and chosen model already present.
+- [ ] onboarding highlights the model as **already installed** or equivalent
+- [ ] recommendation prefers the installed model when it matches user goals reasonably well
+- [ ] provisioning does not suggest re-downloading that same model
+- [ ] summary says the model is ready for immediate configuration
+- [ ] runtime validation uses the installed model successfully
+
+### B2. whisper.cpp backend installed with model file already present
+- [ ] onboarding can identify existing whisper.cpp model file
+- [ ] model is marked available without requiring re-download
+- [ ] runtime uses the located model path successfully
+- [ ] doctor output reports the model as found, not merely backend available
+
+### B3. Multiple installed local models
+- [ ] onboarding distinguishes between multiple installed models
+- [ ] recommended option is sensible and clearly justified
+- [ ] user can override recommendation and choose a different installed model
+- [ ] saving the non-default installed model persists correctly
+
+### B4. Installed local model but missing SoX
+- [ ] onboarding correctly says model is ready but recording path still needs SoX
+- [ ] summary distinguishes **model ready** vs **recording dependency missing**
+- [ ] repair state is used instead of complete state until validation passes
+
+---
+
+## C. Backend installed, model missing
+
+### C1. Backend available but requested model not present
+Example: `faster-whisper` available, `medium` not downloaded.
+- [ ] onboarding marks backend as installed but selected model as **download required**
+- [ ] recommendation may prefer an already installed smaller model if appropriate
+- [ ] provisioning suggests only the missing model path, not a full backend reinstall
+- [ ] summary and doctor explain the difference clearly
+
+### C2. whisper.cpp installed but no model file found
+- [ ] onboarding does not say whisper.cpp is fully ready
+- [ ] onboarding explains that model files are missing
+- [ ] doctor separates “install backend” from “obtain model file” if backend already exists
+
+### C3. Partial cache / corrupted model asset
+- [ ] discovery does not falsely mark model as ready if validation fails
+- [ ] onboarding/doctor route user to repair path
+- [ ] runtime does not mark onboarding complete after failed validation
+
+---
+
+## D. Cloud/API branch
+
+### D1. Cloud path with valid key
+- [ ] API mode remains the fastest setup path
+- [ ] onboarding does not incorrectly prefer stale local detection over an explicitly selected API choice
+- [ ] completion state becomes complete after validation succeeds
+
+### D2. Cloud path with installed local model also present
+- [ ] recommendation explains the tradeoff clearly
+- [ ] API path remains selectable even when local is ready
+- [ ] selected API mode is respected and saved
+- [ ] doctor distinguishes current API config from local recommended alternative
+
+---
+
+## E. Migration and persistence
+
+### E1. Existing legacy config + installed local model
+- [ ] migration does not skip onboarding incorrectly for partial legacy configs
+- [ ] onboarding can suggest the already installed model immediately
+- [ ] saved config includes the correct versioned onboarding state
+
+### E2. Reconfigure from API -> installed local model
+- [ ] `/voice reconfigure` detects the local installed model
+- [ ] reconfigure flow offers the installed model without download guidance
+- [ ] config updates correctly
+- [ ] runtime/doctor/info reflect the new local state
+
+### E3. Reconfigure from local -> API
+- [ ] existing local detection does not block switching to API
+- [ ] cloud config saves cleanly
+- [ ] doctor still reports local assets accurately as alternatives
+
+### E4. Scope behavior
+- [ ] global save still writes global settings
+- [ ] project save still writes `.pi/settings.json`
+- [ ] project-level model-aware config overrides global config cleanly
+- [ ] local installed-model detection still behaves correctly under either scope
+
+---
+
+## F. Runtime and daemon regression checks
+
+### F1. Config-scoped socket behavior
+- [ ] switching between projects/scopes/backends/models does not silently reuse stale daemon state
+- [ ] already installed model in one scope does not cause another scope to falsely appear ready unless it truly is
+
+### F2. `/voice info`
+- [ ] reports mode/backend/model/scope accurately
+- [ ] if model-detection metadata is added, it reports installed vs missing status accurately
+
+### F3. `/voice test`
+- [ ] reports installed-model readiness accurately
+- [ ] does not imply success when model is missing
+- [ ] still exercises mic sample flow correctly
+
+### F4. `/voice doctor`
+- [ ] shows current-config repair path first
+- [ ] shows recommended alternative separately
+- [ ] reports model-ready vs model-missing status clearly
+
+### F5. Hold-to-talk regression
+- [ ] hold `Space` still works when editor is empty
+- [ ] `Ctrl+Shift+V` fallback still works
+- [ ] `Ctrl+Shift+B` BTW voice path still works
+
+---
+
+## Suggested execution order
+
+1. **Automated checks**
+   - unit tests for model detection and recommendation changes
+   - `bun run check`
+2. **Fresh install / no model path**
+3. **Installed local model happy path**
+4. **Backend installed / model missing path**
+5. **API branch with and without local alternatives**
+6. **Migration + reconfigure paths**
+7. **Daemon/runtime regressions**
+8. **Docs verification and QA result capture**
+
+---
+
+## Evidence to capture
+
+For each major scenario, capture at least one of:
+- JSON/RPC output snippet
+- screenshot of onboarding step
+- saved config snippet
+- `/voice doctor` output
+- `/voice info` or `/voice test` output
+
+Recommended artifact locations:
+- `docs/qa-results.md` for pass/fail summaries
+- optional raw snippets under `docs/qa-artifacts/`
+
+---
+
+## Signoff criteria
+
+Model-detection-aware onboarding is release-ready when all are true:
+- [ ] already-installed models are surfaced correctly in onboarding
+- [ ] onboarding avoids unnecessary re-download guidance
+- [ ] backend-installed/model-missing states are clearly explained
+- [ ] API/local branches both remain understandable and correct
+- [ ] migration + reconfigure paths remain safe
+- [ ] runtime/daemon behavior matches selected config
+- [ ] docs reflect the new behavior
+- [ ] QA evidence is recorded