@memberjunction/ai-realtime-client 0.0.1 → 5.41.0

package/README.md

@@ -1,45 +1,165 @@
 # @memberjunction/ai-realtime-client
 
-## ⚠️ IMPORTANT NOTICE ⚠️
+Framework-agnostic **browser-side** abstraction for provider-direct realtime (voice) sessions: the `BaseRealtimeClient` contract plus the four shipped provider drivers (`OpenAIRealtimeClient`, `GeminiRealtimeClient`, `ElevenLabsRealtimeClient`, `AssemblyAIRealtimeClient`) and the shared PCM audio plane (`src/audio/`) the websocket drivers build on.
 
-**This package is created solely for the purpose of setting up OIDC (OpenID Connect) trusted publishing with npm.**
+This package is the **client-side mirror** of the server's `BaseRealtimeModel` pattern (`@memberjunction/ai`). In the **client-direct topology**, the MJ server mints an ephemeral credential + provider-native session config (`ClientRealtimeSessionConfig`) through its server driver, and the browser resolves the matching *client* driver through the MemberJunction `ClassFactory` using the config's `Provider` string as the registration key. The browser owns the provider socket (lowest audio latency — frames never transit the MJ server), while **prompt and tool authority stay server-side**: the client applies the server-built `SessionConfig` verbatim.
 
-This is **NOT** a functional package and contains **NO** code or functionality beyond the OIDC setup configuration.
+For the full architecture — topologies, the co-agent model, channels, narration, security — see **[guides/REALTIME_CO_AGENTS_GUIDE.md](../../../guides/REALTIME_CO_AGENTS_GUIDE.md)**.
 
-## Purpose
+## Installation
 
-This package exists to:
-1. Configure OIDC trusted publishing for the package name `@memberjunction/ai-realtime-client`
-2. Enable secure, token-less publishing from CI/CD workflows
-3. Establish provenance for packages published under this name
+```bash
+npm install @memberjunction/ai-realtime-client
+```
 
-## What is OIDC Trusted Publishing?
+Dependencies are intentionally tiny: `@memberjunction/global` (ClassFactory), `@memberjunction/ai` (the shared `ClientRealtimeSessionConfig` / `JSONObject` types), and `@google/genai` (the Gemini Live SDK). **No Angular, no DOM framework** — the package is plain TypeScript so it can be consumed by any browser host and unit-tested in plain Node.
 
-OIDC trusted publishing allows package maintainers to publish packages directly from their CI/CD workflows without needing to manage npm access tokens. Instead, it uses OpenID Connect to establish trust between the CI/CD provider (like GitHub Actions) and npm.
+## Architecture
 
-## Setup Instructions
+```
+MJ Server                                  Browser
+─────────                                  ───────
+BaseRealtimeModel driver                   BaseRealtimeClient driver
+  .CreateClientSession()                     .Connect(config, micStream)
+        │                                          ▲
+        │  ClientRealtimeSessionConfig             │ ClassFactory.CreateInstance(
+        │  { Provider, Model,                      │   BaseRealtimeClient,
+        │    EphemeralToken, ExpiresAt,  ────────► │   config.Provider)
+        │    SessionConfig (opaque) }              │   // 'openai' | 'gemini' | 'elevenlabs' | 'assemblyai'
+```
 
-To properly configure OIDC trusted publishing for this package:
+**Division of responsibility** (from the `BaseRealtimeClient` doc header):
 
-1. Go to [npmjs.com](https://www.npmjs.com/) and navigate to your package settings
-2. Configure the trusted publisher (e.g., GitHub Actions)
-3. Specify the repository and workflow that should be allowed to publish
-4. Use the configured workflow to publish your actual package
+- **Drivers own ALL provider wire concerns**: transport (WebRTC / WebSocket), event-name translation, the response state machine (a tool-result reply must never collide with an in-flight response), narration-kind tagging, and audible-playback tracking.
+- **Hosts own POLICY**: when to narrate, what instructions to speak, transcript persistence, and UI state. The reference host is `RealtimeSessionService` in `@memberjunction/ng-conversations`.
 
-## DO NOT USE THIS PACKAGE
+## The contract (`BaseRealtimeClient`)
 
-This package is a placeholder for OIDC configuration only. It:
-- Contains no executable code
-- Provides no functionality
-- Should not be installed as a dependency
-- Exists only for administrative purposes
+| Member | Purpose |
+|---|---|
+| `Connect(config, micStream)` | Opens the provider connection with the server-minted ephemeral credential and applies `config.SessionConfig` **verbatim** once the control channel is ready. The *caller* acquires the mic (it owns the permission UX); the driver attaches it and stops its tracks on `Disconnect`. |
+| `SendText(text)` | Injects typed text as a USER turn and asks for a reply through the same collision-safe path tool results use. **SendText implies barge-in**: an active spoken response is cancelled via `CancelActiveResponse` before the text is injected, so the typed turn takes the floor immediately. Must NOT synthesize a user-role transcript echo (the host owns the local echo). |
+| `CancelActiveResponse()` | Cancels the model's ACTIVE spoken response and flushes pending playback so a new user turn can take the floor; no-op when nothing is active. A **floor-control** action only — it must never abort server-side delegated work (hosts do that from `OnInterruption` / their own policy). Leaves `IsBusy` / `IsAudioPlaying` honest afterward. |
+| `SendContextNote(text)` | Injects background context (channel perception deltas, delegated-run progress) **without** forcing a spoken reply. |
+| `RequestSpokenUpdate(instructions)` | Asks for ONE brief interim utterance; the resulting turn's transcripts MUST be tagged `Kind: 'narration'` and must never collide with a pending tool-result reply. |
+| `SendToolResult(callID, outputJson)` | Feeds an executed tool's result back, ensuring the model speaks it ASAP — immediately when idle, otherwise queued behind the in-flight response so the trigger is never dropped. |
+| `SetMuted(muted)` | Toggles mic tracks' `enabled` flag (transport stays up; the provider receives silence). |
+| `Disconnect()` | Tears down everything; emits a final `'closed'` state; safe to call more than once. |
+| `IsBusy` | `true` while a model response is in flight (generation). |
+| `IsAudioPlaying` | `true` while audio is AUDIBLY playing. **Distinct from `IsBusy`** — generation runs ahead of playback; hosts must gate narration on BOTH or queued utterances come out stale. |
+| `OnTranscript / OnToolCall / OnStateChange / OnError / OnInterruption / OnUsage` | Single-handler registration (matching the server `IRealtimeSession` style); registering again replaces the handler. `OnInterruption` fires on **true barge-in only** — user input cut off *active* model output (response in flight or audio audibly playing); a normal turn while the model is idle is not an interruption. Hosts use it per their own policy (the production host cancels pending narration; it deliberately does **not** abort delegated work — that's an explicit user action). |
+| `OnUsage(handler)` | Token-usage telemetry as **deltas** for the response/turn that just completed (`RealtimeClientUsage` — cumulative-only providers must convert in the driver). **Optional capability**: providers without usage events simply never emit (registering is always safe). Emits: OpenAI (`response.done.usage`), Gemini (`usageMetadata`). Never emits: ElevenLabs, AssemblyAI (no wire usage events — ElevenLabs accounts platform-side; AssemblyAI bills flat per session-hour). The production host accumulates deltas and relays them debounced onto the co-agent `AIPromptRun` via the `RelayRealtimeUsage` mutation. |
 
-## More Information
+States (`RealtimeClientState`): `connecting → connected → listening ⇄ speaking → closed | error`. There is deliberately **no `thinking` state** — "the host is executing a tool" is host policy, not wire state.
 
-For more details about npm's trusted publishing feature, see:
-- [npm Trusted Publishing Documentation](https://docs.npmjs.com/generating-provenance-statements)
-- [GitHub Actions OIDC Documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
+Transcripts (`RealtimeClientTranscript`) carry `Role`, `Text` (interim events are incremental **deltas**, finals are the complete turn), `IsFinal`, and `Kind: 'normal' | 'narration'` — narration transcripts are ephemeral by product decision (never captions, never persisted).
 
----
+Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (transport failure, credential expiry) and is also followed by an `'error'` state; `Fatal: false` is a recoverable provider error frame.
 
-**Maintained for OIDC setup purposes only**
+## Drivers
+
+### `OpenAIRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'openai')`
+
+- **Transport**: WebRTC — mic tracks onto a peer connection, remote audio into a hidden `<audio>` sink, the `'oai-events'` data channel for control frames, and the GA SDP handshake. `SessionConfig` is applied via `session.update` when the data channel opens; `'listening'` is reported only after that (obligation #7).
+- **Event translation**: GA *and* beta transcript event names, input-transcription completion, `response.function_call_arguments.done` tool calls, `input_audio_buffer.speech_started` barge-in, `output_audio_buffer.*` playback events, provider error frames.
+- **Response state machine**: `responseActive` set on `response.created`, cleared on `response.done`; tool-result `response.create` triggers are queued while a response is in flight and flushed on `response.done` so the model **always** voices delegated results (obligation #5).
+- **Narration tagging**: `RequestSpokenUpdate` marks the next response so its transcripts emit with `Kind: 'narration'`.
+- **Playback tracking**: `IsAudioPlaying` from the WebRTC `output_audio_buffer` started/stopped events.
+
+### `GeminiRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'gemini')`
+
+- **Transport**: WebSocket via the `@google/genai` Live SDK, authenticated with the server-minted ephemeral token (a `v1alpha` client).
+- **Audio**: client → model is 16-bit PCM @ 16 kHz mono via the shared `createPcmMicCapture` worklet pipeline; model → client is PCM @ 24 kHz, scheduled gaplessly by `GeminiPcmPlayback` (a thin specialization of the shared `RealtimePcmPlayback`), which also backs `IsAudioPlaying` and flushes on barge-in (obligation #3).
+- The server-built `SessionConfig` carries `{ model, config }` (system instruction, tools, transcription, modalities); the client applies it at `live.connect`.
+
+### `ElevenLabsRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'elevenlabs')`
+
+- **Transport**: raw WebSocket against the server-minted **signed URL** — the `EphemeralToken` *is* the `wss://…&token=…` URL (no API key in the browser). Handshake: open → send `conversation_initiation_client_data` carrying the server-authored prompt override (from the `SessionConfig` pact `{ agentId, overrides, config }`) → wait for `conversation_initiation_metadata` → negotiate PCM rates from the metadata's audio-format tags → build the audio plane → `'listening'` (obligation #7). Non-PCM telephony formats (`ulaw_8000`) degrade loudly to the 16 kHz default with a warning.
+- **Audio**: the shared PCM plane (`createPcmMicCapture` up as bare-key `user_audio_chunk` frames, `RealtimePcmPlayback` down from `audio` events) at the **negotiated** rates; `IsAudioPlaying` from the playout clock.
+- **Capability deltas**: transcripts are **finals-only** (no interim deltas; `agent_response_correction` re-finalizes a barged-in turn with what was actually spoken — treat it as the authoritative replacement); `SendContextNote` is **native** (`contextual_update`, sent even mid-response); `RequestSpokenUpdate` is **emulated** as a `user_message` (queued behind in-flight responses; narration kind stamped at send time — there is no `response.created`-style frame to stamp on); there is **no cancel frame** — `CancelActiveResponse` flushes the locally-owned playout (residual server generation is simply never played); **no usage events**; `SendToolResult` is exactly-once (duplicate call ids dropped with a warning).
+- **Busy mapping**: set on the first `audio` / `agent_response` of a turn, cleared on `agent_response_complete` / `interruption` / `client_tool_call` (obligation #2 — no envelope frames exist; state is inferred frame-by-frame).
+
+### `AssemblyAIRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'assemblyai')`
+
+- **Transport**: raw WebSocket to `wss://agents.assemblyai.com/v1/ws?token=…` with the server-minted **one-time** temp token. Handshake: open → send the server-authored `session.update` (the whole session object: prompt, tools, voice, turn detection — from the `SessionConfig` pact `{ session, config }`) as the **first** frame → wait for `session.ready` → audio plane → `'listening'` (obligation #7; audio sent earlier would be dropped).
+- **Audio**: the shared PCM plane at the provider's **fixed 24 kHz** format both directions (`input.audio` up, `reply.audio` down).
+- **Capability deltas**: user transcripts stream as deltas + final, agent transcripts are **final-only** (a barged-in final carries the truncated text — no correction event); `RequestSpokenUpdate` is **native** (`reply.create` per-response instructions, queued behind in-flight replies); `SendText` is **emulated** via `reply.create` (the protocol has no typed-user-input event — best-effort fidelity); `SendContextNote` is emulated via the **mutable `system_prompt`** ("Background updates" section re-sent through `session.update` — a config write that never disturbs generation); **no cancel frame** — `CancelActiveResponse` flushes local playout *and suppresses* residual `reply.audio` of the cancelled reply until the next boundary; **no usage events** (flat session-hour billing).
+- **Barge-in**: `input.speech.started` while output is active is the snappy flush point (~300 ms faster than waiting per the provider's guidance); `reply.done` `status: 'interrupted'` is the authoritative verdict / fallback flush. A speech start while idle is a normal turn, NOT an interruption.
+- **Teardown**: `Disconnect()` sends `session.end` before closing — skipping it leaves a billable 30-second resume hold.
+
+### Shared audio plane (`src/audio/`)
+
+The three websocket drivers (Gemini, ElevenLabs, AssemblyAI — everyone whose audio rides the socket rather than WebRTC) share one browser audio pipeline instead of reimplementing it per provider:
+
+- **`createPcmMicCapture(micStream, sampleRate, onPcmChunk)`** (`micCapture.ts`) — `AudioWorklet`-based mic capture resampled to the requested rate, delivering base64 PCM16 chunks; the worklet is loaded from a Blob URL so the package ships no asset files.
+- **`RealtimePcmPlayback`** (`pcmPlayback.ts`) — gapless playhead-clock scheduling of inbound PCM16, backing `IsAudioPlaying` precisely ("scheduled audio extends beyond the context's current time") with an instant `Flush()` for barge-in / cancel.
+- **`pcmUtils.ts`** — base64 ↔ `ArrayBuffer` and PCM conversion helpers.
+
+Drivers expose these through overridable `protected` creation seams (`createMicCapture` / `createPlayback`), so tests run with no audio hardware.
+
+## Driver-author obligations
+
+`BaseRealtimeClient`'s doc header carries the authoritative client-side **"DRIVER AUTHOR OBLIGATIONS"** block (8 numbered rules, paid for in live debugging — the mirror of the server-side block on `BaseRealtimeModel` in `@memberjunction/ai`). The ones drivers trip over most: leave `'speaking'` *silently* (no state emission) when a tool call is emitted so the host's busy indicator isn't clobbered; release the busy flag at tool-call emission (deadlock guard); flush playback and report `IsAudioPlaying === false` promptly on barge-in *and* on `CancelActiveResponse`; never echo a user transcript for injected text; never drop a tool-result generation trigger (queue behind the in-flight response); surface credential expiry as a `Fatal` error; report `'listening'` only after the session config is applied; treat `SessionConfig` as a private pact between same-keyed driver halves. `RequestSpokenUpdate` has an explicit collision rule: when a response is already in flight the driver must queue or *skip* the update (skipping is fine — narration is disposable by contract); host-side `IsBusy`/`IsAudioPlaying` gating is for timing quality, the driver is the safety net.
+
+## Usage
+
+```typescript
+import { MJGlobal } from '@memberjunction/global';
+import {
+    BaseRealtimeClient,
+    LoadOpenAIRealtimeClient, LoadGeminiRealtimeClient,
+    LoadElevenLabsRealtimeClient, LoadAssemblyAIRealtimeClient
+} from '@memberjunction/ai-realtime-client';
+
+// Tree-shaking prevention — drivers are resolved dynamically, so a static call path
+// must keep their @RegisterClass side effects alive:
+LoadOpenAIRealtimeClient();
+LoadGeminiRealtimeClient();
+LoadElevenLabsRealtimeClient();
+LoadAssemblyAIRealtimeClient();
+
+// 1. The server minted a ClientRealtimeSessionConfig (e.g. via the
+//    StartRealtimeClientSession mutation). Resolve the matching driver:
+const client = MJGlobal.Instance.ClassFactory.CreateInstance<BaseRealtimeClient>(
+    BaseRealtimeClient, startResult.Provider)!;
+
+// 2. Wire policy handlers, then connect with the caller-acquired mic:
+client.OnStateChange(state => updateUI(state));
+client.OnTranscript(t => { if (t.IsFinal && t.Kind === 'normal') persistTurn(t); });
+client.OnToolCall(async call => {
+    const resultJson = await executeTool(call.ToolName, call.ArgumentsJson);
+    client.SendToolResult(call.CallID, resultJson);
+});
+client.OnError(e => { if (e.Fatal) endSession(); });
+
+const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
+await client.Connect(startResult.clientConfig, mic);
+
+// 3. Later:
+client.SendContextNote('[whiteboard] user added a sticky note: "Q3 goals"');
+if (!client.IsBusy && !client.IsAudioPlaying) {
+    client.RequestSpokenUpdate('In one short first-person sentence, say the lookup is still running.');
+}
+await client.Disconnect();
+```
+
+The production host is `RealtimeSessionService` (`packages/Angular/Generic/conversations`) — read it for the full policy layer (caption/transcript routing, prefix-routed client tools, narration pacing, channel plugins).
+
+## Testing seams
+
+All four drivers are written against **structural transport seams** so the full event flow is unit-testable with zero network, zero WebRTC, and zero audio hardware (see `src/__tests__/`, ~4,200 lines of vitest coverage):
+
+- **OpenAI**: `IRealtimePeerConnection`, `IRealtimeDataChannel`, `IRealtimeAudioSink` — created through overridable `protected` factory methods (`createPeerConnection`, `createAudioSink`, …); tests subclass the driver and inject fakes, then drive provider-shaped JSON frames through the data channel.
+- **Gemini**: `GeminiLiveClientSession` (typed subset of the SDK `Session`), `IGeminiMicCapture`, `IGeminiAudioPlayback` — the `connectLiveSession` / capture / playback boundaries are the only things tests replace.
+- **ElevenLabs / AssemblyAI**: `IElevenLabsClientSocket` / `IAssemblyAIClientSocket` (assignable-handler websocket seams behind `createSocket`) plus the shared `createMicCapture` / `createPlayback` seams — tests drive provider-shaped frames straight through the socket fake.
+- Shared fakes live in `src/__tests__/helpers/realtime-fakes.ts`.
+
+If you write a new driver, follow the same shape: every wire/hardware boundary behind a `protected` overridable seam, asserted with scripted provider frames.
+
+## Related
+
+- [`guides/REALTIME_CO_AGENTS_GUIDE.md`](../../../guides/REALTIME_CO_AGENTS_GUIDE.md) — the flagship feature guide
+- `@memberjunction/ai` — `BaseRealtimeModel`, `IRealtimeSession`, `ClientRealtimeSessionConfig`, `RealtimeToolDefinition`
+- `@memberjunction/ai-agents` — `RealtimeSessionRunner`, `RealtimeToolBroker`, `RealtimeClientSessionService`
+- `@memberjunction/ng-conversations` — the Angular host (overlay, channels, session review)
+- `@memberjunction/ng-whiteboard` — the generic whiteboard the Whiteboard channel surfaces

package/dist/audio/audioMeter.d.ts

@@ -0,0 +1,101 @@
+/**
+ * @fileoverview AUDIO ACTIVITY METERING for realtime clients — the Web Audio tap behind
+ * the call UI's audio-reactive visuals (the hero orb that "vibrates like a speaker cone"
+ * and the true-spectrum EQ bars).
+ *
+ * One meter wraps one `AnalyserNode` over either:
+ *  - a `MediaStream` ({@link RealtimeAudioMeter.ForStream} — the mic everywhere; the remote
+ *    WebRTC stream on OpenAI), or
+ *  - a node inside an EXISTING audio graph ({@link RealtimeAudioMeter.ForContextNode} — the
+ *    shared {@link RealtimePcmPlayback} master gain on the client-owned-audio drivers:
+ *    Gemini Live, ElevenLabs Agents, AssemblyAI).
+ *
+ * Construction is DEFENSIVE by contract: in environments without Web Audio (unit tests,
+ * SSR) the factories return `null` and callers degrade to "no metering" — the call UI then
+ * keeps its turn-state-driven animations. The DSP math ({@link ComputeRmsLevel},
+ * {@link BucketizeFrequencyData}) is exported pure so it unit-tests without Web Audio.
+ */
+/** The number of frequency bins the call UI's EQ renders (and meters therefore produce). */
+export declare const REALTIME_AUDIO_BIN_COUNT = 9;
+/**
+ * RMS level (0..1) of byte TIME-DOMAIN samples as `AnalyserNode.getByteTimeDomainData`
+ * delivers them: bytes centered on 128 (silence) spanning 0..255. Pure — unit-testable
+ * without Web Audio. Perceptual boost (×1.6, clamped) keeps normal speech visually alive
+ * without pinning shouts.
+ */
+export declare function ComputeRmsLevel(timeDomainBytes: Uint8Array): number;
+/**
+ * Averages byte FREQUENCY data (`AnalyserNode.getByteFrequencyData`, 0..255 per bin) into
+ * `count` equal buckets normalized 0..1. Only the lower ~70% of the spectrum is used —
+ * voice energy lives there; the top bins are mostly hiss and would flatten the display.
+ * Pure — unit-testable without Web Audio.
+ */
+export declare function BucketizeFrequencyData(frequencyBytes: Uint8Array, count?: number): number[];
+/**
+ * The level/spectrum surface one direction of audio exposes. Narrow interface so unit
+ * tests can substitute fakes for the Web Audio-backed {@link RealtimeAudioMeter}.
+ */
+export interface IRealtimeAudioMeter {
+    /** Instantaneous RMS level, 0..1 (0 = silence). */
+    Level(): number;
+    /** The current spectrum as `count` normalized bins (default {@link REALTIME_AUDIO_BIN_COUNT}). */
+    Bins(count?: number): number[];
+    /** Releases the analyser (and the meter-owned `AudioContext`, when it created one). */
+    Close(): void;
+}
+/**
+ * `AnalyserNode`-backed audio meter. Use the static factories — they are defensive
+ * (return `null` where Web Audio is unavailable) and encode the two ownership modes:
+ * stream meters own a private `AudioContext`; graph meters tap a context the caller owns
+ * (and Close never closes it).
+ */
+export declare class RealtimeAudioMeter implements IRealtimeAudioMeter {
+    private readonly analyser;
+    /** A context this meter created and therefore owns (closed in {@link Close}), or null. */
+    private readonly ownedContext;
+    /** Cloned tracks this meter owns (stopped in {@link Close}); empty when it taps tracks it doesn't own. */
+    private readonly ownedTracks;
+    private readonly timeDomain;
+    private readonly frequency;
+    private closed;
+    private constructor();
+    /**
+     * Shared builder: taps `stream` with a private `AudioContext` + analyser-only sink. A fresh
+     * `AudioContext` starts SUSPENDED and, with no destination route, nothing auto-starts its clock —
+     * so the analyser would read pure silence forever; we resume it (the session always starts from a
+     * user gesture, so this is permitted). `ownedTracks` are clones this meter must stop on Close.
+     */
+    private static buildStreamMeter;
+    /**
+     * Meters a `MediaStream` (e.g. a remote WebRTC stream) via a private `AudioContext`. Returns
+     * `null` when Web Audio / the stream isn't usable (tests, SSR, stopped tracks) — callers treat
+     * null as "no metering available". For the LOCAL microphone use {@link ForMicStream} instead.
+     */
+    static ForStream(stream: MediaStream): RealtimeAudioMeter | null;
+    /**
+     * Meters the LOCAL microphone. Identical to {@link ForStream} EXCEPT it taps a CLONE of the mic's
+     * audio track(s), not the track(s) themselves.
+     *
+     * Why the clone matters: the WebRTC realtime drivers add the mic track to an `RTCPeerConnection`
+     * (`pc.addTrack`). Once a local track feeds the WebRTC pipeline, Chromium reads pure SILENCE from a
+     * parallel `MediaStreamAudioSourceNode` over the SAME track — so the "Listening" meter never moves
+     * while the user speaks even though the mic is live. A cloned track is independent of the PC sender
+     * and meters reliably; it is stopped in {@link Close}. Harmless for the WS drivers (the clone is just
+     * an extra short-lived track). Returns `null` when Web Audio / the stream has no usable audio track.
+     */
+    static ForMicStream(stream: MediaStream): RealtimeAudioMeter | null;
+    /**
+     * Meters a node inside an EXISTING graph (e.g. {@link RealtimePcmPlayback}'s master
+     * gain). The caller keeps ownership of the context — {@link Close} only disconnects
+     * the analyser. Returns `null` when the analyser can't be created.
+     */
+    static ForContextNode(context: AudioContext, source: AudioNode): RealtimeAudioMeter | null;
+    private static createAnalyser;
+    /** @inheritdoc */
+    Level(): number;
+    /** @inheritdoc */
+    Bins(count?: number): number[];
+    /** @inheritdoc */
+    Close(): void;
+}
+//# sourceMappingURL=audioMeter.d.ts.map

package/dist/audio/audioMeter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"audioMeter.d.ts","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,4FAA4F;AAC5F,eAAO,MAAM,wBAAwB,IAAI,CAAC;AAE1C;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,eAAe,EAAE,UAAU,GAAG,MAAM,CAWnE;AAED;;;;;GAKG;AACH,wBAAgB,sBAAsB,CAAC,cAAc,EAAE,UAAU,EAAE,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE,CAiBrH;AAED;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAChC,mDAAmD;IACnD,KAAK,IAAI,MAAM,CAAC;IAChB,kGAAkG;IAClG,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC/B,uFAAuF;IACvF,KAAK,IAAI,IAAI,CAAC;CACjB;AAED;;;;;GAKG;AACH,qBAAa,kBAAmB,YAAW,mBAAmB;IAC1D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAe;IACxC,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAsB;IACnD,0GAA0G;IAC1G,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA0B;IACrD,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA0B;IACpD,OAAO,CAAC,MAAM,CAAS;IAEvB,OAAO;IAQP;;;;;OAKG;IACH,OAAO,CAAC,MAAM,CAAC,gBAAgB;IAW/B;;;;OAIG;WACW,SAAS,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAQvE;;;;;;;;;;OAUG;WACW,YAAY,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAa1E;;;;OAIG;WACW,cAAc,CAAC,OAAO,EAAE,YAAY,EAAE,MAAM,EAAE,SAAS,GAAG,kBAAkB,GAAG,IAAI;IAUjG,OAAO,CAAC,MAAM,CAAC,cAAc;IAO7B,kBAAkB;IACX,KAAK,IAAI,MAAM;IAQtB,kBAAkB;IACX,IAAI,CAAC,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE;IAQ/D,kBAAkB;IACX,KAAK,IAAI,IAAI;CAsBvB"}

package/dist/audio/audioMeter.js

@@ -0,0 +1,193 @@
+/**
+ * @fileoverview AUDIO ACTIVITY METERING for realtime clients — the Web Audio tap behind
+ * the call UI's audio-reactive visuals (the hero orb that "vibrates like a speaker cone"
+ * and the true-spectrum EQ bars).
+ *
+ * One meter wraps one `AnalyserNode` over either:
+ *  - a `MediaStream` ({@link RealtimeAudioMeter.ForStream} — the mic everywhere; the remote
+ *    WebRTC stream on OpenAI), or
+ *  - a node inside an EXISTING audio graph ({@link RealtimeAudioMeter.ForContextNode} — the
+ *    shared {@link RealtimePcmPlayback} master gain on the client-owned-audio drivers:
+ *    Gemini Live, ElevenLabs Agents, AssemblyAI).
+ *
+ * Construction is DEFENSIVE by contract: in environments without Web Audio (unit tests,
+ * SSR) the factories return `null` and callers degrade to "no metering" — the call UI then
+ * keeps its turn-state-driven animations. The DSP math ({@link ComputeRmsLevel},
+ * {@link BucketizeFrequencyData}) is exported pure so it unit-tests without Web Audio.
+ */
+/** The number of frequency bins the call UI's EQ renders (and meters therefore produce). */
+export const REALTIME_AUDIO_BIN_COUNT = 9;
+/**
+ * RMS level (0..1) of byte TIME-DOMAIN samples as `AnalyserNode.getByteTimeDomainData`
+ * delivers them: bytes centered on 128 (silence) spanning 0..255. Pure — unit-testable
+ * without Web Audio. Perceptual boost (×1.6, clamped) keeps normal speech visually alive
+ * without pinning shouts.
+ */
+export function ComputeRmsLevel(timeDomainBytes) {
+    if (timeDomainBytes.length === 0) {
+        return 0;
+    }
+    let sumSquares = 0;
+    for (let i = 0; i < timeDomainBytes.length; i++) {
+        const centered = (timeDomainBytes[i] - 128) / 128;
+        sumSquares += centered * centered;
+    }
+    const rms = Math.sqrt(sumSquares / timeDomainBytes.length);
+    return Math.min(1, rms * 1.6);
+}
+/**
+ * Averages byte FREQUENCY data (`AnalyserNode.getByteFrequencyData`, 0..255 per bin) into
+ * `count` equal buckets normalized 0..1. Only the lower ~70% of the spectrum is used —
+ * voice energy lives there; the top bins are mostly hiss and would flatten the display.
+ * Pure — unit-testable without Web Audio.
+ */
+export function BucketizeFrequencyData(frequencyBytes, count = REALTIME_AUDIO_BIN_COUNT) {
+    const bins = new Array(count).fill(0);
+    if (frequencyBytes.length === 0 || count <= 0) {
+        return bins;
+    }
+    const usable = Math.max(count, Math.floor(frequencyBytes.length * 0.7));
+    const perBucket = usable / count;
+    for (let b = 0; b < count; b++) {
+        const start = Math.floor(b * perBucket);
+        const end = Math.max(start + 1, Math.floor((b + 1) * perBucket));
+        let sum = 0;
+        for (let i = start; i < end && i < frequencyBytes.length; i++) {
+            sum += frequencyBytes[i];
+        }
+        bins[b] = Math.min(1, sum / ((end - start) * 255));
+    }
+    return bins;
+}
+/**
+ * `AnalyserNode`-backed audio meter. Use the static factories — they are defensive
+ * (return `null` where Web Audio is unavailable) and encode the two ownership modes:
+ * stream meters own a private `AudioContext`; graph meters tap a context the caller owns
+ * (and Close never closes it).
+ */
+export class RealtimeAudioMeter {
+    constructor(analyser, ownedContext, ownedTracks = []) {
+        this.closed = false;
+        this.analyser = analyser;
+        this.ownedContext = ownedContext;
+        this.ownedTracks = ownedTracks;
+        this.timeDomain = new Uint8Array(analyser.fftSize);
+        this.frequency = new Uint8Array(analyser.frequencyBinCount);
+    }
+    /**
+     * Shared builder: taps `stream` with a private `AudioContext` + analyser-only sink. A fresh
+     * `AudioContext` starts SUSPENDED and, with no destination route, nothing auto-starts its clock —
+     * so the analyser would read pure silence forever; we resume it (the session always starts from a
+     * user gesture, so this is permitted). `ownedTracks` are clones this meter must stop on Close.
+     */
+    static buildStreamMeter(stream, ownedTracks) {
+        const context = new AudioContext();
+        const source = context.createMediaStreamSource(stream);
+        const analyser = RealtimeAudioMeter.createAnalyser(context);
+        source.connect(analyser);
+        if (context.state === 'suspended') {
+            void context.resume();
+        }
+        return new RealtimeAudioMeter(analyser, context, ownedTracks);
+    }
+    /**
+     * Meters a `MediaStream` (e.g. a remote WebRTC stream) via a private `AudioContext`. Returns
+     * `null` when Web Audio / the stream isn't usable (tests, SSR, stopped tracks) — callers treat
+     * null as "no metering available". For the LOCAL microphone use {@link ForMicStream} instead.
+     */
+    static ForStream(stream) {
+        try {
+            return RealtimeAudioMeter.buildStreamMeter(stream, []);
+        }
+        catch {
+            return null;
+        }
+    }
+    /**
+     * Meters the LOCAL microphone. Identical to {@link ForStream} EXCEPT it taps a CLONE of the mic's
+     * audio track(s), not the track(s) themselves.
+     *
+     * Why the clone matters: the WebRTC realtime drivers add the mic track to an `RTCPeerConnection`
+     * (`pc.addTrack`). Once a local track feeds the WebRTC pipeline, Chromium reads pure SILENCE from a
+     * parallel `MediaStreamAudioSourceNode` over the SAME track — so the "Listening" meter never moves
+     * while the user speaks even though the mic is live. A cloned track is independent of the PC sender
+     * and meters reliably; it is stopped in {@link Close}. Harmless for the WS drivers (the clone is just
+     * an extra short-lived track). Returns `null` when Web Audio / the stream has no usable audio track.
+     */
+    static ForMicStream(stream) {
+        try {
+            const audioTracks = stream.getAudioTracks();
+            if (audioTracks.length === 0) {
+                return null;
+            }
+            const clones = audioTracks.map((t) => t.clone());
+            return RealtimeAudioMeter.buildStreamMeter(new MediaStream(clones), clones);
+        }
+        catch {
+            return null;
+        }
+    }
+    /**
+     * Meters a node inside an EXISTING graph (e.g. {@link RealtimePcmPlayback}'s master
+     * gain). The caller keeps ownership of the context — {@link Close} only disconnects
+     * the analyser. Returns `null` when the analyser can't be created.
+     */
+    static ForContextNode(context, source) {
+        try {
+            const analyser = RealtimeAudioMeter.createAnalyser(context);
+            source.connect(analyser);
+            return new RealtimeAudioMeter(analyser, null);
+        }
+        catch {
+            return null;
+        }
+    }
+    static createAnalyser(context) {
+        const analyser = context.createAnalyser();
+        analyser.fftSize = 256; // 128 frequency bins — plenty for a 9-bar EQ, cheap to read
+        analyser.smoothingTimeConstant = 0.55;
+        return analyser;
+    }
+    /** @inheritdoc */
+    Level() {
+        if (this.closed) {
+            return 0;
+        }
+        this.analyser.getByteTimeDomainData(this.timeDomain);
+        return ComputeRmsLevel(this.timeDomain);
+    }
+    /** @inheritdoc */
+    Bins(count = REALTIME_AUDIO_BIN_COUNT) {
+        if (this.closed) {
+            return new Array(count).fill(0);
+        }
+        this.analyser.getByteFrequencyData(this.frequency);
+        return BucketizeFrequencyData(this.frequency, count);
+    }
+    /** @inheritdoc */
+    Close() {
+        if (this.closed) {
+            return;
+        }
+        this.closed = true;
+        try {
+            this.analyser.disconnect();
+        }
+        catch {
+            /* already disconnected */
+        }
+        // Stop any cloned tracks this meter owns (mic-meter clones) so they don't linger.
+        for (const track of this.ownedTracks) {
+            try {
+                track.stop();
+            }
+            catch {
+                /* already stopped */
+            }
+        }
+        if (this.ownedContext) {
+            void this.ownedContext.close();
+        }
+    }
+}
+//# sourceMappingURL=audioMeter.js.map

package/dist/audio/audioMeter.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"audioMeter.js","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,4FAA4F;AAC5F,MAAM,CAAC,MAAM,wBAAwB,GAAG,CAAC,CAAC;AAE1C;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAAC,eAA2B;IACvD,IAAI,eAAe,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC/B,OAAO,CAAC,CAAC;IACb,CAAC;IACD,IAAI,UAAU,GAAG,CAAC,CAAC;IACnB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC9C,MAAM,QAAQ,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;QAClD,UAAU,IAAI,QAAQ,GAAG,QAAQ,CAAC;IACtC,CAAC;IACD,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,UAAU,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC;IAC3D,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC,CAAC;AAClC,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,sBAAsB,CAAC,cAA0B,EAAE,QAAgB,wBAAwB;IACvG,MAAM,IAAI,GAAa,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACxD,IAAI,cAAc,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC5C,OAAO,IAAI,CAAC;IAChB,CAAC;IACD,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,cAAc,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC;IACxE,MAAM,SAAS,GAAG,MAAM,GAAG,KAAK,CAAC;IACjC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC;QAC7B,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;QACxC,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC,CAAC;QACjE,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,IAAI,CAAC,GAAG,KAAK,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,cAAc,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC5D,GAAG,IAAI,cAAc,CAAC,CAAC,CAAC,CAAC;QAC7B,CAAC;QACD,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC;IACvD,CAAC;IACD,OAAO,IAAI,CAAC;AAChB,CAAC;AAeD;;;;;GAKG;AACH,MAAM,OAAO,kBAAkB;IAU3B,YAAoB,QAAsB,EAAE,YAAiC,EAAE,cAAkC,EAAE;QAF3G,WAAM,GAAG,KAAK,CAAC;QAGnB,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,YAAY,GAAG,YAAY,CAAC;QACjC,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,UAAU,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACnD,IAAI,CAAC,SAAS,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,iBAAiB,CAAC,CAAC;IAChE,CAAC;IAED;;;;;OAKG;IACK,MAAM,CAAC,gBAAgB,CAAC,MAAmB,EAAE,WAA+B;QAChF,MAAM,OAAO,GAAG,IAAI,YAAY,EAAE,CAAC;QACnC,MAAM,MAAM,GAAG,OAAO,CAAC,uBAAuB,CAAC,MAAM,CAAC,CAAC;QACvD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;QAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzB,IAAI,OAAO,CAAC,KAAK,KAAK,WAAW,EAAE,CAAC;YAChC,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;QAC1B,CAAC;QACD,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,OAAO,EAAE,WAAW,CAAC,CAAC;IAClE,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,SAAS,CAAC,MAAmB;QACvC,IAAI,CAAC;YACD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QAC3D,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;;;;;;;OAUG;IACI,MAAM,CAAC,YAAY,CAAC,MAAmB;QAC1C,IAAI,CAAC;YACD,MAAM,WAAW,GAAG,MAAM,CAAC,cAAc,EAAE,CAAC;YAC5C,IAAI,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC;YAChB,CAAC;YACD,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC;YACjD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,IAAI,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAChF,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,cAAc,CAAC,OAAqB,EAAE,MAAiB;QACjE,IAAI,CAAC;YACD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;YAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;YACzB,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAC;QAClD,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAEO,MAAM,CAAC,cAAc,CAAC,OAAqB;QAC/C,MAAM,QAAQ,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;QAC1C,QAAQ,CAAC,OAAO,GAAG,GAAG,CAAC,CAAC,4DAA4D;QACpF,QAAQ,CAAC,qBAAqB,GAAG,IAAI,CAAC;QACtC,OAAO,QAAQ,CAAC;IACpB,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,CAAC,CAAC;QACb,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;QACrD,OAAO,eAAe,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;IAC5C,CAAC;IAED,kBAAkB;IACX,IAAI,CAAC,QAAgB,wBAAwB;QAChD,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAC5C,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,oBAAoB,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;QACnD,OAAO,sBAAsB,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;IACzD,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO;QACX,CAAC;QACD,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC;YACD,IAAI,CAAC,QAAQ,CAAC,UAAU,EAAE,CAAC;QAC/B,CAAC;QAAC,MAAM,CAAC;YACL,0BAA0B;QAC9B,CAAC;QACD,kFAAkF;QAClF,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACnC,IAAI,CAAC;gBACD,KAAK,CAAC,IAAI,EAAE,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACL,qBAAqB;YACzB,CAAC;QACL,CAAC;QACD,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YACpB,KAAK,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QACnC,CAAC;IACL,CAAC;CACJ"}

package/dist/audio/micCapture.d.ts

@@ -0,0 +1,26 @@
+/**
+ * Handle returned by {@link createPcmMicCapture}: the only operation a driver needs is
+ * teardown. Production wraps an `AudioContext` + `AudioWorkletNode` pipeline; tests return a
+ * no-op fake.
+ */
+export interface IPcmMicCapture {
+    /** Stops capture and releases the audio context / worklet resources. */
+    Stop(): void;
+}
+/**
+ * Builds the shared PCM16 mic-capture pipeline for client-owned realtime audio planes:
+ * an `AudioContext` at the requested sample rate, the inline-Blob capture worklet, and a
+ * zero-gain tail that keeps the graph pulled without audible monitoring. Each worklet block
+ * is PCM16-encoded and handed to `onPcmChunk` as base64.
+ *
+ * Extracted from the Gemini client driver (which captured fixed at 16 kHz) and
+ * sample-rate-parameterized so providers that negotiate the input format at session start
+ * (e.g. ElevenLabs' `user_input_audio_format`) capture at the negotiated rate.
+ *
+ * @param micStream The caller-acquired microphone stream (the caller owns the permission UX).
+ * @param sampleRate The PCM16 capture sample rate in Hz (16000 for Gemini Live and the
+ *   ElevenLabs default).
+ * @param onPcmChunk Invoked with each captured block as base64-encoded PCM16.
+ */
+export declare function createPcmMicCapture(micStream: MediaStream, sampleRate: number, onPcmChunk: (base64Pcm16: string) => void): Promise<IPcmMicCapture>;
+//# sourceMappingURL=micCapture.d.ts.map

package/dist/audio/micCapture.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"micCapture.d.ts","sourceRoot":"","sources":["../../src/audio/micCapture.ts"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,WAAW,cAAc;IAC3B,wEAAwE;IACxE,IAAI,IAAI,IAAI,CAAC;CAChB;AAkCD;;;;;;;;;;;;;;GAcG;AACH,wBAAsB,mBAAmB,CACrC,SAAS,EAAE,WAAW,EACtB,UAAU,EAAE,MAAM,EAClB,UAAU,EAAE,CAAC,WAAW,EAAE,MAAM,KAAK,IAAI,GAC1C,OAAO,CAAC,cAAc,CAAC,CAqBzB"}

package/dist/audio/micCapture.js

@@ -0,0 +1,69 @@
+import { encodeFloat32ToPcm16Base64 } from './pcmUtils.js';
+/** Registration name for the inline mic-capture worklet processor. */
+const CAPTURE_WORKLET_NAME = 'mj-realtime-pcm16-capture';
+/**
+ * Inline AudioWorklet processor source (loaded via a Blob URL so the package ships no asset
+ * files). Runs inside the audio rendering thread: forwards each 128-frame mono input block to
+ * the main thread as a copied `Float32Array`. PCM16 conversion + base64 encoding happen on the
+ * main thread to keep the render-thread callback minimal.
+ */
+const CAPTURE_WORKLET_SOURCE = `
+class MJRealtimePcm16Capture extends AudioWorkletProcessor {
+    process(inputs) {
+        const channel = inputs[0] && inputs[0][0];
+        if (channel && channel.length > 0) {
+            this.port.postMessage(channel.slice(0));
+        }
+        return true;
+    }
+}
+registerProcessor('${CAPTURE_WORKLET_NAME}', MJRealtimePcm16Capture);
+`;
+/** Loads the inline capture worklet module from a Blob URL (no asset files shipped). */
+async function loadCaptureWorklet(context) {
+    const blobUrl = URL.createObjectURL(new Blob([CAPTURE_WORKLET_SOURCE], { type: 'application/javascript' }));
+    try {
+        await context.audioWorklet.addModule(blobUrl);
+    }
+    finally {
+        URL.revokeObjectURL(blobUrl);
+    }
+}
+/**
+ * Builds the shared PCM16 mic-capture pipeline for client-owned realtime audio planes:
+ * an `AudioContext` at the requested sample rate, the inline-Blob capture worklet, and a
+ * zero-gain tail that keeps the graph pulled without audible monitoring. Each worklet block
+ * is PCM16-encoded and handed to `onPcmChunk` as base64.
+ *
+ * Extracted from the Gemini client driver (which captured fixed at 16 kHz) and
+ * sample-rate-parameterized so providers that negotiate the input format at session start
+ * (e.g. ElevenLabs' `user_input_audio_format`) capture at the negotiated rate.
+ *
+ * @param micStream The caller-acquired microphone stream (the caller owns the permission UX).
+ * @param sampleRate The PCM16 capture sample rate in Hz (16000 for Gemini Live and the
+ *   ElevenLabs default).
+ * @param onPcmChunk Invoked with each captured block as base64-encoded PCM16.
+ */
+export async function createPcmMicCapture(micStream, sampleRate, onPcmChunk) {
+    const context = new AudioContext({ sampleRate });
+    await loadCaptureWorklet(context);
+    const source = context.createMediaStreamSource(micStream);
+    const worklet = new AudioWorkletNode(context, CAPTURE_WORKLET_NAME);
+    worklet.port.onmessage = (event) => {
+        onPcmChunk(encodeFloat32ToPcm16Base64(event.data));
+    };
+    source.connect(worklet);
+    const muteTail = context.createGain();
+    muteTail.gain.value = 0;
+    worklet.connect(muteTail).connect(context.destination);
+    return {
+        Stop: () => {
+            worklet.port.onmessage = null;
+            source.disconnect();
+            worklet.disconnect();
+            muteTail.disconnect();
+            void context.close();
+        },
+    };
+}
+//# sourceMappingURL=micCapture.js.map

package/dist/audio/micCapture.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"micCapture.js","sourceRoot":"","sources":["../../src/audio/micCapture.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,0BAA0B,EAAE,MAAM,YAAY,CAAC;AAYxD,sEAAsE;AACtE,MAAM,oBAAoB,GAAG,2BAA2B,CAAC;AAEzD;;;;;GAKG;AACH,MAAM,sBAAsB,GAAG;;;;;;;;;;qBAUV,oBAAoB;CACxC,CAAC;AAEF,wFAAwF;AACxF,KAAK,UAAU,kBAAkB,CAAC,OAAqB;IACnD,MAAM,OAAO,GAAG,GAAG,CAAC,eAAe,CAAC,IAAI,IAAI,CAAC,CAAC,sBAAsB,CAAC,EAAE,EAAE,IAAI,EAAE,wBAAwB,EAAE,CAAC,CAAC,CAAC;IAC5G,IAAI,CAAC;QACD,MAAM,OAAO,CAAC,YAAY,CAAC,SAAS,CAAC,OAAO,CAAC,CAAC;IAClD,CAAC;YAAS,CAAC;QACP,GAAG,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC;IACjC,CAAC;AACL,CAAC;AAED;;;;;;;;;;;;;;GAcG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CACrC,SAAsB,EACtB,UAAkB,EAClB,UAAyC;IAEzC,MAAM,OAAO,GAAG,IAAI,YAAY,CAAC,EAAE,UAAU,EAAE,CAAC,CAAC;IACjD,MAAM,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAClC,MAAM,MAAM,GAAG,OAAO,CAAC,uBAAuB,CAAC,SAAS,CAAC,CAAC;IAC1D,MAAM,OAAO,GAAG,IAAI,gBAAgB,CAAC,OAAO,EAAE,oBAAoB,CAAC,CAAC;IACpE,OAAO,CAAC,IAAI,CAAC,SAAS,GAAG,CAAC,KAAiC,EAAE,EAAE;QAC3D,UAAU,CAAC,0BAA0B,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC;IACvD,CAAC,CAAC;IACF,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;IACxB,MAAM,QAAQ,GAAG,OAAO,CAAC,UAAU,EAAE,CAAC;IACtC,QAAQ,CAAC,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC;IACxB,OAAO,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;IACvD,OAAO;QACH,IAAI,EAAE,GAAG,EAAE;YACP,OAAO,CAAC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC;YAC9B,MAAM,CAAC,UAAU,EAAE,CAAC;YACpB,OAAO,CAAC,UAAU,EAAE,CAAC;YACrB,QAAQ,CAAC,UAAU,EAAE,CAAC;YACtB,KAAK,OAAO,CAAC,KAAK,EAAE,CAAC;QACzB,CAAC;KACJ,CAAC;AACN,CAAC"}