@agimon-ai/po-voice 0.0.0-alpha.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vuong Ngo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/NOTICE

@@ -0,0 +1,13 @@
+This package includes portions adapted from OpenAI Codex.
+Copyright 2025 OpenAI
+https://github.com/openai/codex
+Source revision: 5ecb3afd1b
+
+Adapted files:
+- src/services/codexAuth/index.ts, src/services/codexLogin/index.ts, and src/services/codexAuthStorage/index.ts: subscription login, credential storage, and refresh
+- src/services/codexRealtime/index.ts: realtime call creation
+- src/services/realtimeProtocol.ts: Frameless realtime events and context messages
+
+These portions were modified for Po and are licensed under Apache-2.0.
+A copy is included in licenses/Codex-Apache-2.0.txt.
+The package's MIT license continues to apply to its original code.

package/README.md

@@ -0,0 +1,249 @@
+# @agimon-ai/po-voice
+
+## Server and browser configuration
+
+The web Settings page includes Voice under global and workspace settings. It uses the shared
+config API, including file hashes, atomic saves, numeric values, and scope validation. `voice.mode` and `voice.autoCapture`
+remain global settings. Browser input selection is separate from the host ffmpeg
+`voice.recorder.device` option.
+
+The composer Voice button records on whatever input the browser is already using; there is no
+device picker. Autonomous voice asks which input to open only when the browser reports more than
+one, and remembers the answer in that browser's local storage until the set of inputs changes.
+Dismissing that dialog uses the browser's own default input. Each tab keeps its own media
+connection lease. Manual dictation appends to the draft until Send.
+
+Voice owns these authenticated server endpoints:
+
+| Mount                                   | Endpoint         | Purpose                                                  |
+| --------------------------------------- | ---------------- | -------------------------------------------------------- |
+| `/api/plugins/voice`                    | `GET /readiness` | Check configuration and local transcription dependencies |
+| `/api/sessions/:sessionId/plugin/voice` | `GET /status`    | Read lifecycle and media readiness                       |
+| same                                    | `POST /control`  | Send `{action, target?}`                                 |
+
+Control actions are `status`, `manual`, `activate`, `deactivate`, `mute`, `unmute`, `interrupt`,
+and `transfer` (with a target catalog order and its catalog revision). The typed session method `voice.control` uses the
+same operations as commands and minor-mode actions. Browser recording uploads continue to use
+`/api/sessions/:sessionId/plugin/voice-media/manual/transcribe`.
+
+The native server composes Voice controllers with one session media broker. Workers receive PCM
+on a dedicated media port. No terminal editor or Pi ExtensionContext is emulated. Live voice
+still requires its separate subscription credentials; a successful local readiness check does
+not verify a live provider connection.
+
+Capture speech on the client, transcribe it on the host, and play responses through Po Voice mode.
+
+Part of the [Po distribution](https://www.npmjs.com/package/@agimon-ai/po).
+
+Voice can transcribe one recording or keep the currently active session listening. It also gives
+the primary agent a bounded `narrate` tool and accepts narration requests from other extensions.
+
+> **Alpha:** Voice state, tool, and platform support may change between releases.
+
+## Realtime companion
+
+Realtime is an optional conversational interface, not a replacement for the primary Pi
+agent or for half-duplex Voice. Pi keeps its selected model, reasoning, tools, approvals,
+Team and Workflow. The companion handles speech and forwards fresh action requests to Pi;
+it does not execute tools or launch a second coding agent.
+
+The experimental live path connects browser WebRTC, Po-owned subscription login,
+the exclusive media lease, bounded session context, and main-agent requests/results.
+Subscription connectivity and a synthetic echo roundtrip have been exercised. Physical
+playback, echo safety, interruption recovery and microphone usability remain **unqualified**.
+
+For a separately approved trial:
+
+1. Select `voice.mode: live` in global configuration. Repository configuration cannot enable it.
+2. Run `/voice-auto login` and open the displayed URL on the Po host. Credentials stay
+   in Po's private `voice/subscription-auth` directory beside its global configuration,
+   separate from Codex and Pi OAuth. `/voice-auto login-cancel` cancels the attempt.
+3. With separate provider and microphone consent, explicitly activate Voice with a connected
+   WebRTC-capable browser. Selecting live or signing in does not activate capture.
+4. Use browser controls or `/voice-auto mute`, `unmute`, `interrupt`, and `end`.
+
+Pi's settled final text is returned as a correlated companion result, separately from the
+initial submission acknowledgement. Later Pi turns, including turns responding to Team or
+Workflow completion, can supply speakable session updates without another voice request.
+Responses exceeding the bounded result limit stay in the Pi session; Voice reports that it
+cannot summarize contents it did not transmit. Thinking and raw tool output are not forwarded.
+
+Interruption suppresses output for the remainder of that activation; explicitly end and
+reactivate Voice to hear replies again. It does not cancel main-agent work. Exact primary-agent
+`narrate` and its TTS fallback are not enabled in live mode. Voice-only approval answers remain
+unavailable. Reconnect and reload do not resume microphone transmission.
+
+Omit `voice.mode` or select the compatibility value `legacy` for the existing half-duplex
+Voice path. It remains a supported interaction choice. Live capture does not require local
+ASR; the local transcription requirements below apply to half-duplex and manual Voice.
+No API-key fallback is provided if subscription access fails.
+
+See [ARCHITECTURE.md](docs/ARCHITECTURE.md#realtime-port-foundations) for the current boundary.
+Upstream port attribution is in `NOTICE` and `licenses/Codex-Apache-2.0.txt`.
+
+## Requirements
+
+- Node.js 24 or newer
+- A browser with microphone capture and speech synthesis when using `po-web`
+- One supported local transcription engine: `whisper-cli`, `whisper`, or `mlx_whisper` (Apple
+  silicon only)
+- Microphone permission on the client device
+
+## Install
+
+The seeded `modes.yaml` selects Voice in `default.packages`. Keep it there or select it through a
+layer; Voice is not fixed host infrastructure.
+
+## Configure
+
+A complete `~/.po/config.yaml` example using Python Whisper is:
+
+```yaml
+voice:
+  engine: openai-whisper
+  language: auto
+  recorder:
+    device: default
+  adapters:
+    openai-whisper:
+      binary: whisper
+      model:
+        id: base
+  autoCapture:
+    model: provider/model-id
+    startPhrases: []
+    stopPhrases: []
+    composeOpenPhrases: [hey doom]
+    composeSendPhrases: ["that's it"]
+    composeCancelPhrases: [scratch that]
+    utteranceIdleMs: 3000
+    composeUtteranceIdleMs: 1200
+    composeNudgeMs: 10000
+    transcriptionTimeoutMs: 120000
+    tts:
+      engine: macos-say
+      voice: Samantha
+      rate: 190
+```
+
+Replace `provider/model-id` with a model configured in Pi, or omit `autoCapture` when only manual
+transcription is required. Use the `whisper-cpp` engine for `whisper-cli`, `openai-whisper` for
+`whisper`, or `mlx-whisper` for `mlx_whisper`. `utteranceIdleMs` accepts 1,500 to 10,000 ms and
+defaults to 3,000.
+
+A profile can give its persona its own voice. Front-matter in the persona's `profile.md` shadows
+`autoCapture.tts` per field, so a persona setting only `rate` keeps the voice configured here:
+
+```markdown
+---
+name: Rhea
+voice:
+  voice: Karen
+  rate: 175
+---
+```
+
+The `engine` is never overridable, because it selects the synthesiser rather than the persona. The
+spoken voice is read when autonomous voice activates, so a profile switch applies it through the
+reload that switch already performs.
+
+Autonomous voice can also change profile. The `profile` capability on the voice tool facade lists
+the configured personas and queues a switch, which reloads the session with the new persona, its
+voice, and its avatar. See
+[@agimon-ai/po-profile](../../foundations/po-profile/README.md) for authoring personas.
+
+When the agent is launched by `po-server`, capture and narration use the connected media client.
+The protocol is capability-driven rather than browser-specific. A client can own microphone capture,
+adaptive activity detection, endpoint decisions, autonomous capture phases, and playback while streaming
+mono 16 kHz PCM16 through the authenticated session media API. The host supplies capabilities the client
+does not advertise and remains responsible for durable spooling, configured transcription, and Pi or
+model actions. This boundary supports terminal, web, React Native, and future watch adapters without
+putting platform APIs in the shared client runtime.
+
+A browser using the sealed remote-control channel takes the session media lease from a local client, so
+narration and capture follow the user to the remote device. Local clients cannot take that lease back
+while the remote client remains connected. In the browser, `tts.voice` selects an exact
+SpeechSynthesis voice name or URI. If it does not match, the browser uses its default voice. Portable
+clients implement the declared ports through `@agimon-ai/po-voice/client-media`. Browser media ownership is page-global rather than tied to the focused cockpit route, so switching sessions does not disconnect an autonomous capture owned by another session.
+
+For server-streamed narration, the browser uses the exact 16 kHz PCM as a local echo reference while uploading microphone PCM unchanged. A bounded discriminator filters only the local Silero input and reports cumulative trusted speech duration, never reference audio, residual audio, transcript, correlation, or gain. Natural address-free interruption requires 300 ms of trusted residual speech plus semantic novelty. Missing or uncertain alignment, failed local Silero, unsupported clients, and browser SpeechSynthesis all fail closed to the existing intentional-address or exact-stop behavior. The 800 ms echo tail remains active. Acoustic thresholds are provisional until physical speaker-to-microphone qualification passes.
+
+## Commands and tools
+
+- `/voice-auto mute` pauses microphone capture without disabling narration; `/voice-auto unmute` resumes capture.
+- While autonomous capture is active, say `hey doom` at the beginning of a segment to open a long composed prompt. Later finalized segments are buffered until `that's it` submits the combined text or `scratch that` discards it in the example configuration.
+- `describe_voice_tools` returns the active session's spoken capability catalog.
+- `use_voice_tools` executes a bounded ordered batch against that catalog.
+- `narrate` speaks one primary-agent-authored utterance and waits for physical playback.
+
+Composition is for long spoken prompts, so a multi-part prompt is not submitted half-written. A short utterance needs no phrase and is delivered as soon as it finalizes.
+
+Each configured phrase array replaces its defaults rather than extending them. The example above keeps only one phrase per set, so it does not recognize `doom prompt`, `doom send`, or `doom cancel` as composition commands. Omit the arrays to use all defaults:
+
+`composeOpenPhrases` is `hey doom, doom prompt`, `composeSendPhrases` is `that's it, doom send`, and `composeCancelPhrases` is `doom cancel, scratch that`. Matching ignores case, punctuation and apostrophes, and tolerates small transcription differences, so `thats it` and `doom sent` are recognised. Send and cancel act as commands only while a draft is open, and only when the phrase is the whole segment or ends a sentence, so ordinary dictation containing those words stays content. While a draft collects, the endpoint window shortens to `composeUtteranceIdleMs` (default 1,200 ms) so a short command finalizes as its own turn. A composed prompt is queued as a Pi follow-up while Pi is busy.
+
+Drafts are held in memory for the active autonomous session and are limited to 32,768 characters. Worker restarts and five-minute idle capture rotation preserve the draft, but deactivation, extension reload, and process restart discard it. Wait until the status returns to `composing, listening` before speaking the next segment because capture pauses during transcription.
+
+Before a final response, narration contains the complete answer, including every user-relevant
+conclusion, question, warning, result, and next action in the written response. It does not use
+a shorter spoken summary that leaves essential information only in text. Each narration is
+limited to 4,096 characters and returns `completed`, `interrupted`, `superseded`, or `failed`.
+Once confirmed user speech begins, new narration waits in memory until that turn is synchronously delivered to Pi. Multiple waiting requests use the configured `autoCapture.model` to become one bounded spoken update, with one bounded deterministic utterance if compaction fails. Narration that was already playing is not stopped by this queue gate; the existing ranked barge-in policy remains responsible for interruption.
+Narration fails closed while Voice is starting or draining, during shutdown, reload, or
+deactivation, and when the request belongs to a stale session.
+Paired-host targets are discovered through authenticated Voice grants and appear in the same revision-bound transfer catalog. A handoff prepares the target, releases the source, activates the target, and waits for usable media readiness. It does not transfer agent context, transcripts, pending work, or buffered source audio.
+
+If no `narrate` attempt is made before a final response, the active Voice session can produce one
+bounded fallback utterance. Short final responses use deterministic text; longer ones may use
+`autoCapture.model` for one bounded summary and fall back to a deterministic excerpt on failure.
+These optional model calls consume provider quota.
+
+## Data flow and recovery
+
+The selected client captures audio. When it advertises client orchestration, its portable PCM state
+machine performs activity detection and endpointing, then sends phase metadata with the streamed audio.
+Clients without those capabilities use host adaptive and neural VAD as a fallback. PCM validation,
+spooling, WAV creation, normalization, and configured local Whisper execution run on the host in a
+private supervised worker. Private spool directories use mode `0700` and files use `0600`.
+Unacknowledged turns can be rediscovered after a worker restart.
+
+This is not a blanket “nothing leaves the machine” guarantee:
+
+- Client PCM crosses the authenticated cockpit transport, is queued transiently in memory, and is
+  persisted only in the host worker's private spool.
+- Transcript candidates and bounded state strings cross the worker/process boundary.
+- Command correction and long-final fallback text can be sent to the configured model provider.
+- Telemetry may contain bounded operational metadata; review its sink configuration.
+
+Manual recordings and autonomous spool windows are bounded to five minutes. Deactivation or
+cancellation interrupts pending capture and playback. A manual reload does not silently reactivate
+the microphone.
+
+## Architecture
+
+See [Architecture](./docs/ARCHITECTURE.md) for the thread and process topology, the path a single spoken turn takes, the XState lifecycle and how it relates to Pi, the module and cordis service graph, and narration gating and barge-in. [SPEC.md](./docs/SPEC.md) is the normative contract.
+
+## Public API
+
+The root exports audio infrastructure, PCM, VAD, and utterance services, narration and playback
+contracts, command correction, fallback narration, and Voice types. Host adapters should import only declared package exports:
+
+| Export                             | Purpose                                                               |
+| ---------------------------------- | --------------------------------------------------------------------- |
+| `@agimon-ai/po-voice`              | Host audio adapters, narration, PCM, VAD, correction, and Voice types |
+| `@agimon-ai/po-voice/client-media` | Client capture activity and media contracts                           |
+
+## Development
+
+```bash
+pnpm build
+pnpm typecheck
+pnpm test
+pnpm lint
+```
+
+Maintained by [Agimon](https://agimon.ai/about).
+
+## License
+
+MIT