@edkimmel/expo-audio-stream 0.4.2 → 0.6.0

package/NATIVE_EVENTS.md

@@ -9,8 +9,7 @@ including payload shapes, trigger conditions, and recommended JS responses.
 
 ### `AudioData`
 
-Emitted at the configured interval during microphone recording. Also doubles as
-the recording error channel.
+Emitted at the configured interval during microphone recording.
 
 | Field | Type | Notes |
 |---|---|---|
@@ -27,19 +26,21 @@ the recording error channel.
 
 **Error variant** (same event name, different shape):
 
+> Errors are also emitted on the richer `MicrophoneError` event (see below).
+> This variant is kept for backward compatibility.
+
 | Field | Type | Notes |
 |---|---|---|
-| `error` | `string` | Error code: `READ_ERROR`, `RECORDING_CRASH` |
+| `error` | `string` | Error code — see `MicrophoneError` section for full table |
 | `errorMessage` | `string` | Human-readable description |
-| `streamUuid` | `string` | Stream that errored |
+| `streamUuid` | `string` | Stream ID (Android only; `""` on iOS) |
 
 **Platform:** Android, iOS
 
 **JS response:**
 - Forward the base64 PCM to your STT pipeline or WebSocket.
 - Use `soundLevel` for VAD or UI visualisation.
-- Check for the `error` field before assuming the payload is audio data.
-  On error, stop the conversation turn or retry.
+- Prefer subscribing via `addMicrophoneErrorListener` — it receives the structured `MicrophoneError` event with `isFatal` and `autoResuming` fields.
 
 ---
 
@@ -62,6 +63,59 @@ wired headset/headphones, USB headset).
 
 ---
 
+### `MicrophoneError`
+
+Fired on every microphone error alongside the backward-compatible `AudioData`
+error variant. Carries structured `isFatal` and `autoResuming` fields decided
+by native code.
+
+**OTA safe:** apps can subscribe to this event before the native binary
+includes it — old native simply never emits it, so the listener is never
+called. Apps using the `AudioData` error variant continue to work unchanged.
+
+| Field | Type | Notes |
+|---|---|---|
+| `code` | `string` | Error code (see table below) |
+| `message` | `string` | Human-readable description including any underlying system error |
+| `isFatal` | `boolean` | `true` = recording has stopped; caller must call `stopMicrophone()` and reconnect |
+| `autoResuming` | `boolean` | `true` = library will reinstall the tap automatically (`INTERRUPTED` only) |
+
+**Error code reference:**
+
+| Code | Platform | `isFatal` | `autoResuming` | Meaning | Recommended action |
+|------|----------|-----------|----------------|---------|-------------------|
+| `INTERRUPTED` | iOS | false | true | System interrupted the audio session | Show paused UI; do NOT stop — library auto-resumes |
+| `RESUME_FAILED` | iOS | true | false | System allowed resume but engine restart failed | `stopMicrophone()` then reconnect |
+| `RESTART_FAILED` | iOS | true | false | Route-change or engine-rebuild recovery failed | `stopMicrophone()` then reconnect |
+| `ENGINE_DIED` | iOS | true | false | SharedAudioEngine exhausted all recovery | `stopMicrophone()` + `disconnectPipeline()` then reconnect both |
+| `READ_ERROR` | iOS | false | false | Single empty buffer — transient, recording continues | Log for diagnostics; no action |
+| `READ_ERROR` | Android | true | false | `AudioRecord.read()` failed 10× consecutively | `stopMicrophone()` then reconnect |
+| `RECORDING_CRASH` | Android | true | false | Recording thread threw an unexpected exception | `stopMicrophone()` then reconnect |
+
+**Platform:** Android, iOS
+
+**JS response:**
+
+```typescript
+import { addMicrophoneErrorListener } from "@edkimmel/expo-audio-stream"
+
+const sub = addMicrophoneErrorListener((error) => {
+  if (error.isFatal) {
+    stopMicrophone()
+    if (error.code === "ENGINE_DIED") disconnectPipeline()
+    reconnect()
+  } else if (error.autoResuming) {
+    showPausedUI() // INTERRUPTED — library will reinstall tap automatically
+  }
+  // isFatal: false, autoResuming: false → READ_ERROR on iOS: log and ignore
+})
+
+// cleanup
+sub.remove()
+```
+
+---
+
 ## Legacy Playback Events (AudioPlaybackManager)
 
 ### `SoundStarted`
@@ -200,6 +254,43 @@ The pipeline transitions from `draining` to `idle`.
 
 ---
 
+### `PipelinePlaybackStopped`
+
+Fired approximately `outputLatencyMs` after `PipelineDrained` for the same turn —
+i.e., when the last sample physically leaves the speaker. Pairs symmetrically
+with `PipelinePlaybackStarted` (start-of-emission ↔ end-of-emission).
+
+**Distinct from `state: 'idle'`:** that's the pipeline-state-machine value
+reported via `PipelineStateChanged`. `PipelinePlaybackStopped` is a
+physical-world milestone about audible audio, not a state-machine transition.
+
+| Field | Type | Notes |
+|---|---|---|
+| `turnId` | `string` | The turn whose last sample just stopped emitting |
+
+**Cancellation:** the scheduled dispatch is cancelled if a new turn starts,
+`invalidatePipelineTurn` is called, or the pipeline disconnects before the
+delay elapses. Consumers will not receive a late-arriving event mid-stream.
+
+**Approximation:** the delay uses `outputLatencyMs` captured at drain time.
+On iOS the value reflects `AVAudioSession.outputLatency` (total HW output
+latency). On Android it uses `AudioTrack.getTimestamp()` to compute frames
+in-flight, falling back to a HAL-buffer estimate if the timestamp call fails.
+A route change during the latency window (e.g. switch to Bluetooth mid-tail)
+is *not* re-measured — the captured value is used. The error is typically
+single-digit milliseconds and is acceptable for VAD-gating use cases.
+
+**Platform:** Android, iOS
+
+**JS response:**
+- Use this — not `PipelineDrained` — when you need to know when **mic-side
+  echo from the speaker is over**. Useful for voice-agent pipelines that
+  gate server-side VAD on the agent's own playback ending.
+- If you do not care about the physical-world emission boundary, prefer
+  `PipelineDrained` (cheaper, fires earlier).
+
+---
+
 ### `PipelineFrequencyBands`
 
 Fired at the interval configured by `frequencyBandIntervalMs` during pipeline

package/android/.gradle/buildOutputCleanup/cache.properties

@@ -0,0 +1,2 @@
+#Thu Jun 04 15:44:34 EDT 2026
+gradle.version=8.9

package/android/src/main/java/expo/modules/audiostream/AudioEffectsManager.kt

@@ -9,18 +9,13 @@ import android.util.Log
 /**
  * Manages hardware audio effects for voice recording.
  *
- * We use VOICE_RECOGNITION as our audio source. The Android CDD (Section 5.4)
- * mandates that this source delivers unprocessed audio:
- *   [C-1-2] MUST disable noise reduction by default
- *   [C-1-3] MUST disable automatic gain control by default
+ * We use VOICE_COMMUNICATION as our audio source. This source enables
+ * platform-managed AEC at the HAL level automatically. The explicit
+ * AcousticEchoCanceler effect here is belt-and-suspenders for devices
+ * where the platform does not apply it automatically.
  *
- * NS and AGC are therefore off by default to honor the spec. Enabling them
- * re-introduces the processing the CDD explicitly prohibits for this source
- * and can cause low-volume capture on many OEMs.
- *
- * AEC is the one effect the CDD permits for VOICE_RECOGNITION ("expects a
- * stream that has an echo cancellation effect if available"), so it is
- * enabled by default.
+ * NS and AGC remain opt-in — VOICE_COMMUNICATION applies its own
+ * processing and additional effects can cause over-processing on some OEMs.
  */
 class AudioEffectsManager(
     /** Enable hardware noise suppressor. Default false — CDD 5.4 [C-1-2] prohibits it for VOICE_RECOGNITION. */

package/android/src/main/java/expo/modules/audiostream/AudioRecorderManager.kt

@@ -339,23 +339,29 @@ class AudioRecorderManager(
                     consecutiveErrors++
                     if (consecutiveErrors >= 10) {
                         Log.e(Constants.TAG, "Too many consecutive read errors ($consecutiveErrors), stopping")
-                        emitRecordingError("READ_ERROR", "AudioRecord read failed after $consecutiveErrors consecutive errors")
+                        emitRecordingError("READ_ERROR", "AudioRecord read failed after $consecutiveErrors consecutive errors", isFatal = true)
                         break
                     }
                 }
             }
         } catch (e: Exception) {
             Log.e(Constants.TAG, "Recording thread crashed", e)
-            emitRecordingError("RECORDING_CRASH", e.message ?: "Recording thread unexpected error")
+            emitRecordingError("RECORDING_CRASH", e.message ?: "Recording thread unexpected error", isFatal = true)
         }
     }
 
     /**
      * Sends a recording error event to JS so the caller can react.
      */
-    private fun emitRecordingError(code: String, message: String) {
+    private fun emitRecordingError(
+        code: String,
+        message: String,
+        isFatal: Boolean,
+        autoResuming: Boolean = false
+    ) {
         mainHandler.post {
             try {
+                // Backward-compat: keep the error variant on AudioData for existing consumers
                 eventSender.sendExpoEvent(
                     Constants.AUDIO_EVENT_NAME, bundleOf(
                         "error" to code,
@@ -363,6 +369,15 @@ class AudioRecorderManager(
                         "streamUuid" to streamUuid
                     )
                 )
+                // Rich structured channel for new consumers
+                eventSender.sendExpoEvent(
+                    Constants.MICROPHONE_ERROR_EVENT_NAME, bundleOf(
+                        "code" to code,
+                        "message" to message,
+                        "isFatal" to isFatal,
+                        "autoResuming" to autoResuming
+                    )
+                )
             } catch (e: Exception) {
                 Log.e(Constants.TAG, "Failed to send error event", e)
             }
@@ -479,9 +494,11 @@ class AudioRecorderManager(
             return null
         }
 
-        // Use VOICE_RECOGNITION for far-field/speakerphone use — higher mic gain,
-        // no near-field gain reduction. AEC/NS/AGC are applied separately via AudioEffectsManager.
-        val audioSource = MediaRecorder.AudioSource.VOICE_RECOGNITION
+        // VOICE_COMMUNICATION enables platform-managed AEC (echo cancellation happens at
+        // the HAL level, not just via the explicit AcousticEchoCanceler effect).
+        // VOICE_RECOGNITION intentionally bypasses platform AEC per the Android CDD,
+        // making hardware echo cancellation ineffective regardless of AudioEffectsManager.
+        val audioSource = MediaRecorder.AudioSource.VOICE_COMMUNICATION
 
         val record = AudioRecord(
             audioSource,

package/android/src/main/java/expo/modules/audiostream/CommunicationAudioManager.kt

@@ -0,0 +1,155 @@
+package expo.modules.audiostream
+
+import android.app.Activity
+import android.media.AudioDeviceInfo
+import android.media.AudioManager
+import android.os.Build
+import android.util.Log
+import java.util.concurrent.atomic.AtomicInteger
+
+/**
+ * Owns Android communication-mode lifecycle and output device routing for
+ * hands-free voice sessions (microphone + speaker playback with AEC).
+ *
+ * Responsibilities:
+ *   - Set MODE_IN_COMMUNICATION before AudioRecord starts so the HAL echo
+ *     reference path is active when AcousticEchoCanceler initializes.
+ *   - Route output to the best available device: Bluetooth HFP > wired headset
+ *     > built-in speaker. Re-routes automatically when devices connect or
+ *     disconnect.
+ *   - Bind hardware volume buttons to STREAM_VOICE_CALL while a session is
+ *     active so the user's volume buttons control playback volume.
+ *   - Reset everything when all callers have stopped so the phone's audio
+ *     state is clean.
+ *
+ * Reference-counted: mic and pipeline each call startSession/stopSession
+ * independently. Communication mode stays active until both have stopped.
+ *
+ * Usage:
+ *   cam.startSession(activity)  // call before AudioRecord.startRecording() or pipeline connect
+ *   cam.onDeviceChanged()       // call from AudioDeviceCallback
+ *   cam.stopSession(activity)   // call after AudioRecord.stop() or pipeline disconnect
+ */
+class CommunicationAudioManager(private val audioManager: AudioManager) {
+
+    private val refCount = AtomicInteger(0)
+
+    private val sessionActive get() = refCount.get() > 0
+
+    /**
+     * Start (or join) a voice session. Safe to call from multiple owners —
+     * communication mode is activated on the first call.
+     * Pass the current Activity so volume buttons bind to STREAM_VOICE_CALL.
+     */
+    fun startSession(activity: Activity? = null) {
+        if (refCount.getAndIncrement() == 0) {
+            audioManager.mode = AudioManager.MODE_IN_COMMUNICATION
+            activity?.volumeControlStream = AudioManager.STREAM_VOICE_CALL
+            Log.d(TAG, "Session started — MODE_IN_COMMUNICATION")
+        }
+        applyBestRoute()
+    }
+
+    /**
+     * Release this owner's hold on the session. Communication mode is reset
+     * only when all owners have called stopSession.
+     * No-op if the session was already ended via forceReset.
+     */
+    fun stopSession(activity: Activity? = null) {
+        // Decrement only if currently positive — prevents post-forceReset calls
+        // from re-triggering cleanup (getAndUpdate returns the previous value).
+        val prev = refCount.getAndUpdate { if (it > 0) it - 1 else 0 }
+        if (prev <= 0) {
+            Log.d(TAG, "stopSession — already stopped, no-op")
+            return
+        }
+        val remaining = prev - 1
+        if (remaining == 0) {
+            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
+                audioManager.clearCommunicationDevice()
+            } else {
+                @Suppress("DEPRECATION")
+                audioManager.stopBluetoothSco()
+                @Suppress("DEPRECATION")
+                audioManager.isSpeakerphoneOn = false
+            }
+            audioManager.mode = AudioManager.MODE_NORMAL
+            activity?.volumeControlStream = AudioManager.USE_DEFAULT_STREAM_TYPE
+            Log.d(TAG, "Session ended — audio mode reset to NORMAL")
+        } else {
+            Log.d(TAG, "stopSession — $remaining owner(s) still active")
+        }
+    }
+
+    /**
+     * Re-evaluate and apply the best available output route.
+     * Call this from AudioDeviceCallback whenever devices are added or removed.
+     */
+    fun onDeviceChanged() {
+        applyBestRoute()
+    }
+
+    /**
+     * Unconditionally reset all communication audio state regardless of ref count.
+     * Use in OnDestroy and explicit destroy flows where normal lifecycle won't run.
+     */
+    fun forceReset(activity: Activity? = null) {
+        refCount.set(0)
+        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
+            audioManager.clearCommunicationDevice()
+        } else {
+            @Suppress("DEPRECATION")
+            audioManager.stopBluetoothSco()
+            @Suppress("DEPRECATION")
+            audioManager.isSpeakerphoneOn = false
+        }
+        audioManager.mode = AudioManager.MODE_NORMAL
+        activity?.volumeControlStream = AudioManager.USE_DEFAULT_STREAM_TYPE
+        Log.d(TAG, "forceReset — audio mode reset to NORMAL")
+    }
+
+    // ── Private ─────────────────────────────────────────────────────────────
+
+    private fun applyBestRoute() {
+        if (!sessionActive) return
+        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
+            applyBestRouteApi31()
+        } else {
+            applyBestRouteLegacy()
+        }
+    }
+
+    private fun applyBestRouteApi31() {
+        val devices = audioManager.availableCommunicationDevices
+        // Priority: BT HFP > wired headset > built-in speaker
+        val preferred = devices.firstOrNull { it.type == AudioDeviceInfo.TYPE_BLUETOOTH_SCO }
+            ?: devices.firstOrNull { it.type == AudioDeviceInfo.TYPE_WIRED_HEADSET }
+            ?: devices.firstOrNull { it.type == AudioDeviceInfo.TYPE_BUILTIN_SPEAKER }
+        if (preferred != null) {
+            audioManager.setCommunicationDevice(preferred)
+            Log.d(TAG, "Route → ${preferred.productName} (type=${preferred.type})")
+        } else {
+            Log.w(TAG, "No suitable communication device found")
+        }
+    }
+
+    @Suppress("DEPRECATION")
+    private fun applyBestRouteLegacy() {
+        // Detect a connected BT HFP device via the full device list (API 23+).
+        val allDevices = audioManager.getDevices(AudioManager.GET_DEVICES_ALL)
+        val btSco = allDevices.firstOrNull { it.type == AudioDeviceInfo.TYPE_BLUETOOTH_SCO }
+        if (btSco != null) {
+            audioManager.startBluetoothSco()
+            audioManager.isSpeakerphoneOn = false
+            Log.d(TAG, "Route → Bluetooth SCO (${btSco.productName})")
+        } else {
+            audioManager.stopBluetoothSco()
+            audioManager.isSpeakerphoneOn = true
+            Log.d(TAG, "Route → built-in speaker")
+        }
+    }
+
+    companion object {
+        private const val TAG = "CommunicationAudioMgr"
+    }
+}

package/android/src/main/java/expo/modules/audiostream/Constants.kt

@@ -2,6 +2,7 @@ package expo.modules.audiostream
 
 object Constants {
     const val AUDIO_EVENT_NAME = "AudioData"
+    const val MICROPHONE_ERROR_EVENT_NAME = "MicrophoneError"
     const val AUDIO_ANALYSIS_EVENT_NAME = "AudioAnalysis"
     const val DEVICE_RECONNECTED_EVENT_NAME = "DeviceReconnected"
     const val DEFAULT_SAMPLE_RATE = 16000 // Default sample rate for audio recording

package/android/src/main/java/expo/modules/audiostream/ExpoPlayAudioStreamModule.kt

@@ -26,6 +26,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
     private lateinit var audioRecorderManager: AudioRecorderManager
     private lateinit var audioManager: AudioManager
     private lateinit var pipelineIntegration: PipelineIntegration
+    private lateinit var communicationAudioManager: CommunicationAudioManager
 
     // Ensure callbacks are delivered on the main thread
     private val mainHandler by lazy { Handler(Looper.getMainLooper()) }
@@ -62,6 +63,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
             if (firstOfGroup?.isNotEmpty()==true) {
                 val matched = firstOfGroup.map { "${it.productName} (type=${it.type})" }
                 Log.d("ExpoAudioCallback", "AudioDeviceCallback ➜ ADDED (interesting): $matched")
+                communicationAudioManager.onDeviceChanged()
                 pipelineIntegration.logAudioTrackHealth("device_added")
                 val params = Bundle()
                 params.putString("reason", "newDeviceAvailable")
@@ -79,6 +81,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
             if (lastOfGroup?.isNotEmpty() == true) {
                 val matched = lastOfGroup.map { "${it.productName} (type=${it.type})" }
                 Log.d("ExpoAudioCallback", "AudioDeviceCallback ➜ REMOVED (interesting): $matched")
+                communicationAudioManager.onDeviceChanged()
                 pipelineIntegration.logAudioTrackHealth("device_removed")
                 val params = Bundle()
                 params.putString("reason", "oldDeviceUnavailable")
@@ -94,6 +97,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
 
         Events(
             Constants.AUDIO_EVENT_NAME,
+            Constants.MICROPHONE_ERROR_EVENT_NAME,
             Constants.DEVICE_RECONNECTED_EVENT_NAME,
             PipelineIntegration.EVENT_STATE_CHANGED,
             PipelineIntegration.EVENT_PLAYBACK_STARTED,
@@ -101,6 +105,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
             PipelineIntegration.EVENT_ZOMBIE_DETECTED,
             PipelineIntegration.EVENT_UNDERRUN,
             PipelineIntegration.EVENT_DRAINED,
+            PipelineIntegration.EVENT_PLAYBACK_STOPPED,
             PipelineIntegration.EVENT_AUDIO_FOCUS_LOST,
             PipelineIntegration.EVENT_AUDIO_FOCUS_RESUMED,
             PipelineIntegration.EVENT_FREQUENCY_BANDS
@@ -112,26 +117,27 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
 
         OnCreate {
             audioManager = appContext.reactContext?.getSystemService(Context.AUDIO_SERVICE) as AudioManager
+            communicationAudioManager = CommunicationAudioManager(audioManager)
             audioManager.registerAudioDeviceCallback(audioCallCallback, mainHandler)
         }
 
         OnDestroy {
             reportedGroups.clear()
             audioManager.unregisterAudioDeviceCallback(audioCallCallback)
-            // Module is being destroyed (app shutdown)
-            // Just clean up resources without reinitialization
             pipelineIntegration.destroy()
             audioRecorderManager.release()
+            communicationAudioManager.forceReset(appContext.currentActivity)
         }
 
         AsyncFunction("destroy") { promise: Promise ->
-            // User explicitly called destroy - clean up and reinitialize for reuse
             pipelineIntegration.destroy()
             audioRecorderManager.release()
+            communicationAudioManager.forceReset(appContext.currentActivity)
 
             // Reinitialize all managers so the module can be used again
             initializeManager()
             initializePipeline()
+            communicationAudioManager = CommunicationAudioManager(audioManager)
             promise.resolve(null)
         }
 
@@ -152,11 +158,13 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
         }
 
         AsyncFunction("startMicrophone") { options: Map<String, Any?>, promise: Promise ->
+            communicationAudioManager.startSession(appContext.currentActivity)
             audioRecorderManager.startRecording(options, promise)
         }
 
         AsyncFunction("stopMicrophone") { promise: Promise ->
             audioRecorderManager.stopRecording(promise)
+            communicationAudioManager.stopSession(appContext.currentActivity)
         }
 
         Function("toggleSilence") { isSilent: Boolean ->
@@ -167,6 +175,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
         // ── Native Audio Pipeline V3 ────────────────────────────────────
 
         AsyncFunction("connectPipeline") { options: Map<String, Any?>, promise: Promise ->
+            communicationAudioManager.startSession(appContext.currentActivity)
             pipelineIntegration.connect(options, promise)
         }
 
@@ -180,6 +189,7 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
 
         AsyncFunction("disconnectPipeline") { promise: Promise ->
             pipelineIntegration.disconnect(promise)
+            communicationAudioManager.stopSession(appContext.currentActivity)
         }
 
         AsyncFunction("invalidatePipelineTurn") { options: Map<String, Any?>, promise: Promise ->
@@ -194,6 +204,10 @@ class ExpoPlayAudioStreamModule : Module(), EventSender {
             pipelineIntegration.getState()
         }
 
+        Function("getPipelineOutputLatencyMs") {
+            pipelineIntegration.outputLatencyMs()
+        }
+
     }
     private fun initializeManager() {
         val androidContext =