@elizaos/capacitor-talkmode 1.0.0 → 2.0.3-beta.2

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shaw Walters and elizaOS Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,137 @@
+# @elizaos/capacitor-talkmode
+
+Capacitor plugin for full voice conversation sessions: speech-to-text → chat orchestration → text-to-speech.
+
+Runs inside Eliza agent apps on **iOS**, **Android**, **Electrobun (desktop)**, and **browser**. On native platforms, it uses platform STT (AVFoundation / SFSpeechRecognizer on iOS, Android SpeechRecognizer) and ElevenLabs streaming TTS with PCM/MP3 audio playback. On web, it falls back to the Web Speech API for both STT and TTS.
+
+## Capabilities
+
+- **Voice session management** — start/stop a hands-free, push-to-talk, compose, or passive session.
+- **Live transcription** — streaming interim and final transcripts from the microphone.
+- **TTS playback** — speak text via ElevenLabs (native only) or system TTS; supports per-utterance directives (voice, speed, stability, language, seed, latency tier).
+- **Interrupt on speech** — automatically cut TTS playback when the user starts speaking.
+- **Permission handling** — check and request microphone + speech recognition permissions.
+- **Event-driven state machine** — `idle` → `listening` → `processing` → `speaking` → `error` with typed events.
+
+## Installation
+
+```bash
+bun add @elizaos/capacitor-talkmode
+npx cap sync
+```
+
+Peer dependency: `@capacitor/core ^8.3.1`.
+
+On iOS, add the following to your `Info.plist`:
+
+```xml
+<key>NSMicrophoneUsageDescription</key>
+<string>Used for voice conversations</string>
+<key>NSSpeechRecognitionUsageDescription</key>
+<string>Used for speech-to-text transcription</string>
+```
+
+## Usage
+
+```ts
+import { TalkMode } from "@elizaos/capacitor-talkmode";
+
+// Start a voice session with ElevenLabs TTS
+const { started } = await TalkMode.start({
+  config: {
+    mode: "hands-free",
+    tts: {
+      apiKey: process.env.ELEVENLABS_API_KEY,
+      voiceId: "your-voice-id",
+      modelId: "eleven_flash_v2_5",
+      interruptOnSpeech: true,
+    },
+    stt: {
+      language: "en",
+      sampleRate: 16000,
+    },
+    silenceWindowMs: 700,
+  },
+});
+
+// Listen for transcripts
+await TalkMode.addListener("transcript", (event) => {
+  console.log(event.transcript, event.isFinal);
+});
+
+// Listen for state changes
+await TalkMode.addListener("stateChange", (event) => {
+  console.log(event.state, event.statusText);
+});
+
+// Speak a string directly
+const result = await TalkMode.speak({
+  text: "Hello from your Eliza agent.",
+  directive: { speed: 1.1, language: "en-US" },
+});
+
+// Stop the session
+await TalkMode.stop();
+```
+
+## Actions / Methods
+
+| Method | Description |
+|---|---|
+| `start(options?)` | Begin a voice session |
+| `stop()` | End the session and release audio resources |
+| `isEnabled()` | Whether a session is currently active |
+| `getState()` | Current state (`TalkModeState`) and status text |
+| `updateConfig(options)` | Patch configuration mid-session |
+| `speak(options)` | Speak text via TTS; returns completion result |
+| `stopSpeaking()` | Interrupt TTS playback |
+| `isSpeaking()` | Whether TTS is currently playing |
+| `checkPermissions()` | Read microphone + speech recognition permission status |
+| `requestPermissions()` | Prompt for required permissions |
+
+## Events
+
+| Event | Payload type | When fired |
+|---|---|---|
+| `stateChange` | `TalkModeStateEvent` | State machine transition |
+| `transcript` | `TalkModeTranscriptEvent` | Interim or final STT result |
+| `speaking` | `TTSSpeakingEvent` | TTS utterance starts |
+| `speakComplete` | `TTSCompleteEvent` | TTS utterance finishes or is interrupted |
+| `playbackStart` | `TalkModePlaybackStartEvent` | Native PCM/MP3 playback begins |
+| `error` | `TalkModeErrorEvent` | Recoverable or fatal error |
+
+## Configuration reference
+
+All config is passed to `start()` or `updateConfig()` — no process env vars are read by this package.
+
+| Field | Required | Description |
+|---|---|---|
+| `tts.apiKey` | Native TTS only | ElevenLabs API key |
+| `tts.voiceId` | No | ElevenLabs voice ID |
+| `tts.modelId` | No | ElevenLabs model (default: `eleven_flash_v2_5` on iOS) |
+| `tts.outputFormat` | No | e.g. `"pcm_24000"`, `"mp3_44100"` |
+| `tts.interruptOnSpeech` | No | Cut TTS when mic detects speech |
+| `tts.voiceAliases` | No | Name → voiceId mapping |
+| `stt.engine` | No | `"whisper"` or `"web"` |
+| `stt.modelSize` | No | Whisper model size |
+| `stt.language` | No | BCP-47 language code |
+| `stt.sampleRate` | No | Hz, default 16000 |
+| `silenceWindowMs` | No | Silence gap before finalising transcript |
+| `mode` | No | Session mode (`compose`, `push-to-talk`, `hands-free`, `passive`) |
+| `sessionKey` | No | Chat session key for orchestration |
+
+## Platform notes
+
+| Platform | STT | TTS |
+|---|---|---|
+| iOS / Android | Native platform API | ElevenLabs streaming + system TTS fallback |
+| Electrobun (desktop) | Native STT | ElevenLabs streaming + system TTS fallback |
+| Browser | Web Speech API | `SpeechSynthesis` only (ElevenLabs blocked by CORS) |
+
+## Building
+
+```bash
+bun run --cwd plugins/plugin-native-talkmode build
+```
+
+Outputs ESM to `dist/esm/`, CJS to `dist/plugin.cjs.js`, and IIFE to `dist/plugin.js`.

package/android/build.gradle

@@ -6,6 +6,16 @@ ext {
 }
 
 apply plugin: 'com.android.library'
+// Explicitly apply the Kotlin Android plugin. The kotlin-gradle-plugin is on
+// the root buildscript classpath, but without applying it here AGP 8.13 falls
+// back to its "built-in Kotlin" compile path (build/intermediates/
+// built_in_kotlinc), which compiles the .kt sources but does NOT bundle the
+// resulting .class files into the *release* library jar. The app's
+// :app:assembleRelease then links a library AAR with zero plugin classes, so
+// the Capacitor plugin (and any manifest-declared component) is absent from
+// the release dex. Applying the standard Kotlin plugin wires Kotlin
+// compilation into both the debug and release jar-bundling tasks.
+apply plugin: 'org.jetbrains.kotlin.android'
 android {
     namespace = "ai.eliza.plugins.talkmode"
     compileSdk project.hasProperty('compileSdkVersion') ? rootProject.ext.compileSdkVersion : 35
@@ -21,15 +31,19 @@ android {
         }
     }
     compileOptions {
-        sourceCompatibility JavaVersion.VERSION_17
-        targetCompatibility JavaVersion.VERSION_17
+        sourceCompatibility JavaVersion.VERSION_21
+        targetCompatibility JavaVersion.VERSION_21
+    }
+
+    kotlinOptions {
+        jvmTarget = "21"
     }
 }
 
 repositories {
     google()
     maven {
-        url = uri(rootProject.ext.mavenCentralMirrorUrl)
+        url = uri(rootProject.ext.has('mavenCentralMirrorUrl') ? rootProject.ext.mavenCentralMirrorUrl : 'https://repo.maven.apache.org/maven2')
     }
     mavenCentral()
 }