@memberjunction/ai-realtime-client 6.2.0-edge.3 → 6.2.0-edge.4

package/README.md

@@ -1,10 +1,10 @@
 # @memberjunction/ai-realtime-client
 
-Framework-agnostic **browser-side** abstraction for provider-direct realtime (voice) sessions: the `BaseRealtimeClient` contract plus the provider drivers (`OpenAIRealtimeClient`, `OpenAILiveClient`, `GeminiRealtimeClient`, `ElevenLabsRealtimeClient`, `AssemblyAIRealtimeClient`, `xAIRealtimeClient`, `HuggingFaceRealtimeClient`) and the shared PCM audio plane (`src/audio/`) the websocket drivers build on.
+Framework-agnostic **browser-side** abstraction for provider-direct realtime (voice and video) sessions: the `BaseRealtimeClient` contract plus the provider drivers (`OpenAIRealtimeClient`, `OpenAILiveClient`, `GeminiRealtimeClient`, `GeminiEnterpriseRealtimeClient`, `ElevenLabsRealtimeClient`, `AssemblyAIRealtimeClient`, `xAIRealtimeClient`, `HuggingFaceRealtimeClient`), the shared PCM audio plane (`src/audio/`) the websocket drivers build on, the browser media code in the `/media` entry (capture, video playout, the media stage) and the video conformance kit in the `/testing` entry.
 
-This package is the **client-side mirror** of the server's `BaseRealtimeModel` pattern (`@memberjunction/ai`). In the **client-direct topology**, the MJ server mints an ephemeral credential + provider-native session config (`ClientRealtimeSessionConfig`) through its server driver, and the browser resolves the matching *client* driver through the MemberJunction `ClassFactory` using the config's `Provider` string as the registration key. The browser owns the provider socket (lowest audio latency — frames never transit the MJ server), while **prompt and tool authority stay server-side**: the client applies the server-built `SessionConfig` verbatim.
+This package is the **client-side mirror** of the server's `BaseRealtimeModel` pattern (`@memberjunction/ai`). In the **client-direct topology**, the MJ server mints a session config (`ClientRealtimeSessionConfig`) through its server driver, and the browser resolves the matching *client* driver through the MemberJunction `ClassFactory` using the config's `Provider` string as the registration key. The browser owns the provider socket (lowest audio latency — frames never transit the MJ server), or, for a provider with no browser-safe credential, a socket to MJAPI's realtime relay. Either way **prompt and tool authority stay server-side**: the client applies the server-built `SessionConfig` verbatim, and a relay session's setup is written by the server.
 
-For the full architecture — topologies, the co-agent model, channels, narration, security — see **[guides/REALTIME_CO_AGENTS_GUIDE.md](../../../guides/REALTIME_CO_AGENTS_GUIDE.md)**.
+For the full architecture — topologies, the co-agent model, channels, narration, security, and adding a video-capable provider — see **[guides/REALTIME_CO_AGENTS_GUIDE.md](../../../guides/REALTIME_CO_AGENTS_GUIDE.md)**.
 
 ## Installation
 
@@ -12,7 +12,17 @@ For the full architecture — topologies, the co-agent model, channels, narratio
 npm install @memberjunction/ai-realtime-client
 ```
 
-Dependencies are intentionally tiny: `@memberjunction/global` (ClassFactory), `@memberjunction/ai` (the shared `ClientRealtimeSessionConfig` / `JSONObject` types), and `@google/genai` (the Gemini Live SDK). **No Angular, no DOM framework** — the package is plain TypeScript so it can be consumed by any browser host and unit-tested in plain Node.
+Dependencies are intentionally tiny: `@memberjunction/global` (ClassFactory), `@memberjunction/ai` (the shared `ClientRealtimeSessionConfig` / `RealtimeVideoFrame` / `JSONObject` types), `rxjs`, and `@google/genai` (the Gemini Live SDK). **No Angular, no DOM framework** — the package is plain TypeScript so it can be consumed by any browser host and unit-tested in plain Node. A consumer that imports only `/media` bundles no driver and no `@google/genai`.
+
+## Entry points
+
+| Import | What it has | Boundary |
+|---|---|---|
+| `@memberjunction/ai-realtime-client` | Everything: `BaseRealtimeClient`, the drivers, the PCM audio plane, and all of `/media` | — |
+| `@memberjunction/ai-realtime-client/media` | The browser media code no driver sits behind: camera, microphone and display capture, frame sampling and pacing, the video source arbiter, video playout and its decoders, the playback clock, the media stage and saved layouts, the audio meter | May import only other `/media` modules, `audio/audioMeter`, `@memberjunction/ai`, `@memberjunction/global` and `rxjs`. `media-entry-boundary.test.ts` fails the build otherwise; its allowlist is where a new `/media` dependency is added |
+| `@memberjunction/ai-realtime-client/testing` | The video provider conformance kit (`RunRealtimeVideoConformance`, `ListRealtimeVideoConformanceChecks`, the harness types, recorders and fixtures) | No driver, no `@google/genai`, no test framework (`testing-entry-boundary.test.ts`). The main entry does not export it, so apps never bundle it |
+
+The LiveKit room (`@memberjunction/livekit-room-core`) and `@memberjunction/ng-realtime-media` import only `/media`.
 
 ## Architecture
 
@@ -20,35 +30,43 @@ Dependencies are intentionally tiny: `@memberjunction/global` (ClassFactory), `@
 MJ Server                                  Browser
 ─────────                                  ───────
 BaseRealtimeModel driver                   BaseRealtimeClient driver
-  .CreateClientSession()                     .Connect(config, micStream)
+  .CreateClientSession()                     .Connect(config, micStream, cameraStream?)
         │                                          ▲
         │  ClientRealtimeSessionConfig             │ ClassFactory.CreateInstance(
         │  { Provider, Model,                      │   BaseRealtimeClient,
         │    EphemeralToken, ExpiresAt,  ────────► │   config.Provider)
-        │    SessionConfig (opaque) }              │   // 'openai' | 'openAILive' | 'gemini' | 'elevenlabs' | 'assemblyai'
+        │    Transport?, RelayUrl?,                │   // 'openai' | 'openai-live' | 'gemini' | 'gemini-enterprise'
+        │    SessionConfig (opaque),               │   // | 'elevenlabs' | 'assemblyai' | 'xai' | 'huggingface'
+        │    AvatarStatus? }                       │
 ```
 
 **Division of responsibility** (from the `BaseRealtimeClient` doc header):
 
-- **Drivers own ALL provider wire concerns**: transport (WebRTC / WebSocket), event-name translation, the response state machine (a tool-result reply must never collide with an in-flight response), narration-kind tagging, and audible-playback tracking.
-- **Hosts own POLICY**: when to narrate, what instructions to speak, transcript persistence, and UI state. The reference host is `RealtimeSessionService` in `@memberjunction/ng-conversations`.
+- **Drivers own ALL provider wire concerns**: transport (WebRTC / WebSocket), event-name translation, the response state machine (a tool-result reply must never collide with an in-flight response), narration-kind tagging, audible-playback tracking, and the agent's video when the model sends one.
+- **Hosts own POLICY**: when to narrate, what instructions to speak, transcript persistence, and UI state. The reference host is `RealtimeSessionRuntime` in `@memberjunction/realtime-runtime`; the Angular `RealtimeSessionService` in `@memberjunction/ng-conversations` is a thin adapter over it.
 
 ## The contract (`BaseRealtimeClient`)
 
 | Member | Purpose |
 |---|---|
-| `Connect(config, micStream)` | Opens the provider connection with the server-minted ephemeral credential and applies `config.SessionConfig` **verbatim** once the control channel is ready. The *caller* acquires the mic (it owns the permission UX); the driver attaches it and stops its tracks on `Disconnect`. |
+| `Connect(config, micStream, cameraStream?)` | Opens the provider connection with the server-minted credential (or, on a relay session, MJAPI's relay URL) and applies `config.SessionConfig` **verbatim** once the control channel is ready. The *caller* acquires the mic (it owns the permission UX); the driver attaches it and stops its tracks on `Disconnect`. A video-capable driver also samples the optional camera into the model. |
 | `SendText(text)` | Injects typed text as a USER turn and asks for a reply through the same collision-safe path tool results use. **SendText implies barge-in**: an active spoken response is cancelled via `CancelActiveResponse` before the text is injected, so the typed turn takes the floor immediately. Must NOT synthesize a user-role transcript echo (the host owns the local echo). |
-| `CancelActiveResponse()` | Cancels the model's ACTIVE spoken response and flushes pending playback so a new user turn can take the floor; no-op when nothing is active. A **floor-control** action only — it must never abort server-side delegated work (hosts do that from `OnInterruption` / their own policy). Leaves `IsBusy` / `IsAudioPlaying` honest afterward. |
+| `CancelActiveResponse()` | Cancels the model's ACTIVE spoken response and flushes pending playback (and the agent's video) so a new user turn can take the floor; no-op when nothing is active. A **floor-control** action only — it must never abort server-side delegated work (hosts do that from `OnInterruption` / their own policy). Leaves `IsBusy` / `IsAudioPlaying` accurate afterward. |
 | `SendContextNote(text)` | Injects background context (channel perception deltas, delegated-run progress) **without** forcing a spoken reply. |
 | `RequestSpokenUpdate(instructions)` | Asks for ONE brief interim utterance; the resulting turn's transcripts MUST be tagged `Kind: 'narration'` and must never collide with a pending tool-result reply. |
 | `SendToolResult(callID, outputJson)` | Feeds an executed tool's result back, ensuring the model speaks it ASAP — immediately when idle, otherwise queued behind the in-flight response so the trigger is never dropped. |
 | `SetMuted(muted)` | Toggles mic tracks' `enabled` flag (transport stays up; the provider receives silence). |
+| `ReplaceMicrophone?(micStream)` | Rebinds what used the old microphone track after a device switch (obligation #10). |
+| `SendVideoFrame?(base64, mime, sourceId?)` | Streams one image frame to a model that takes inbound video. Hosts don't call it directly: `VideoSourceArbiter` does. |
+| `AddTrack(descriptor)` / `RemoveTrack(descriptor)` | Adds or removes a track mid-session, negotiated against what the driver supports (a capture adds its inbound video track when it starts). Audio is not removable. |
+| `SupportsInboundVideo`, `InboundVideoRate`, `MaxInboundVideoStreams`, `EstablishedTracks`, `IsTrackEstablished(modality, direction)` | What the session negotiated: whether the model takes video, at what rate, how many streams. Anything pacing frames reads the rate from here. |
+| `OnRemoteVideo(handler)` | The agent's video, as a `MediaVideoSource`, handed over once per session when the model sends video (a live avatar). One handler, owned by the runtime. |
+| `OnRemoteMediaStream(handler)` | The agent's audio as a `MediaStream`, for a host recorder. |
 | `Disconnect()` | Tears down everything; emits a final `'closed'` state; safe to call more than once. |
 | `IsBusy` | `true` while a model response is in flight (generation). |
-| `IsAudioPlaying` | `true` while audio is AUDIBLY playing. **Distinct from `IsBusy`** — generation runs ahead of playback; hosts must gate narration on BOTH or queued utterances come out stale. |
-| `OnTranscript / OnToolCall / OnStateChange / OnError / OnInterruption / OnUsage` | Single-handler registration (matching the server `IRealtimeSession` style); registering again replaces the handler. `OnInterruption` fires on **true barge-in only** — user input cut off *active* model output (response in flight or audio audibly playing); a normal turn while the model is idle is not an interruption. Hosts use it per their own policy (the production host cancels pending narration; it deliberately does **not** abort delegated work — that's an explicit user action). |
-| `OnUsage(handler)` | Token-usage telemetry as **deltas** for the response/turn that just completed (`RealtimeClientUsage` — cumulative-only providers must convert in the driver). **Optional capability**: providers without usage events simply never emit (registering is always safe). Emits: OpenAI (`response.done.usage`), Gemini (`usageMetadata`). Never emits: ElevenLabs, AssemblyAI (no wire usage events — ElevenLabs accounts platform-side; AssemblyAI bills flat per session-hour). The production host accumulates deltas and relays them debounced onto the co-agent `AIPromptRun` via the `RelayRealtimeUsage` mutation. |
+| `IsAudioPlaying` | `true` while audio is AUDIBLY playing (in an avatar session, while the avatar speaks). **Distinct from `IsBusy`** — generation runs ahead of playback; hosts must gate narration on BOTH or queued utterances come out stale. |
+| `OnTranscript / OnToolCall / OnStateChange / OnError / OnInterruption / OnUsage / OnTrackStateChange` | Single-handler registration (matching the server `IRealtimeSession` style); registering again replaces the handler. `OnInterruption` fires on **true barge-in only** — user input cut off *active* model output (response in flight or audio audibly playing); a normal turn while the model is idle is not an interruption. Hosts use it per their own policy (the production host cancels pending narration; it deliberately does **not** abort delegated work — that's an explicit user action). |
+| `OnUsage(handler)` | Token-usage telemetry as **deltas** for the response/turn that just completed (`RealtimeClientUsage` — cumulative-only providers must convert in the driver), with per-modality detail (`InputTokenDetails`, `OutputTokenDetails`) and the seconds of avatar video generated (`OutputTokenDetails.VideoSeconds`). **Optional capability**: providers without usage events never emit (registering is always safe). Emits: OpenAI (`response.done.usage`), Gemini (`usageMetadata`, reported per turn, plus the avatar's video seconds in updates of their own). Never emits: ElevenLabs, AssemblyAI (no wire usage events — ElevenLabs accounts platform-side; AssemblyAI bills flat per session-hour). The production host accumulates deltas and relays them debounced onto the co-agent `AIPromptRun` via the `RelayRealtimeUsage` mutation. |
 
 States (`RealtimeClientState`): `connecting → connected → listening ⇄ speaking → closed | error`. There is deliberately **no `thinking` state** — "the host is executing a tool" is host policy, not wire state.
 
@@ -56,6 +74,12 @@ Transcripts (`RealtimeClientTranscript`): carry `Role`, `Text` (interim events a
 
 Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (transport failure, credential expiry) and is also followed by an `'error'` state; `Fatal: false` is a recoverable provider error frame.
 
+### Sessions through MJAPI's relay
+
+`ClientRealtimeSessionConfig.Transport` says how the browser reaches the provider. Absent or `'direct'`: the driver opens the provider's socket with `EphemeralToken`. `'relay'`: the provider has no browser-safe credential, so the driver connects to MJAPI's relay at `RelayUrl` (`wss://<mjapi>/realtime/relay/<ticket>`) and `EphemeralToken` is empty. MJAPI holds the provider credential and runs the provider's frame policy between the two sockets.
+
+Every driver calls `AssertTransportSupported(config)` first in `Connect` (obligation #11), so a driver that connects only to its provider refuses a relay session with a clear error. A driver that speaks its provider's protocol through the relay opts in by overriding `SupportsRelayTransport`; today only `GeminiEnterpriseRealtimeClient` does. The relay URL carries the session's ticket, so it is a credential: never log it or put it in an error.
+
 ## Drivers
 
 ### `OpenAIRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'openai')`
@@ -65,8 +89,9 @@ Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (tra
 - **Response state machine**: `responseActive` set on `response.created`, cleared on `response.done`; tool-result `response.create` triggers are queued while a response is in flight and flushed on `response.done` so the model **always** voices delegated results (obligation #5).
 - **Narration tagging**: `RequestSpokenUpdate` marks the next response so its transcripts emit with `Kind: 'narration'`.
 - **Playback tracking**: `IsAudioPlaying` from the WebRTC `output_audio_buffer` started/stopped events.
+- **Video**: none in either direction.
 
-### `OpenAILiveClient` — `@RegisterClass(BaseRealtimeClient, 'openAILive')`
+### `OpenAILiveClient` — `@RegisterClass(BaseRealtimeClient, 'openai-live')`, also `'OpenAILiveRealtime'`
 
 - **Model**: `gpt-live-1` (OpenAI Live API).
 - **Transport**: WebRTC via `/v1/realtime/calls` data channel.
@@ -76,17 +101,25 @@ Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (tra
   - **Tool Call Deduplication**: Tool calls arriving via `output_item.done` or `response.function_call_arguments.done` are deduplicated by `CallID` via an `emittedToolCallIds` Set. The deduplication Set deliberately survives local cancellations so late-arriving frames within an in-flight turn cannot double-emit an already-handled tool call.
   - **Context & Commentary**: `SendContextNote` appends to model thinking via `session.thinking.append` (with `delegation_id: null`). Spoken interim updates use `session.commentary.append`.
   - **Outbound Queuing**: Frames sent while the WebRTC data channel is connecting are buffered in a bounded FIFO queue (`MAX_OUTBOUND_QUEUE_SIZE = 100`, drop-oldest with warnings).
+- **Video**: none in either direction.
 
 ### `GeminiRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'gemini')`
 
 - **Transport**: WebSocket via the `@google/genai` Live SDK, authenticated with the server-minted ephemeral token (a `v1alpha` client).
-- **Audio**: client → model is 16-bit PCM @ 16 kHz mono via the shared `createPcmMicCapture` worklet pipeline; model → client is PCM @ 24 kHz, scheduled gaplessly by `GeminiPcmPlayback` (a thin specialization of the shared `RealtimePcmPlayback`), which also backs `IsAudioPlaying` and flushes on barge-in (obligation #3).
-- The server-built `SessionConfig` carries `{ model, config }` (system instruction, tools, transcription, modalities); the client applies it at `live.connect`.
+- **Audio**: client → model is 16-bit PCM @ 16 kHz mono via the shared `CreatePcmMicCapture` worklet pipeline; model → client is PCM @ 24 kHz, scheduled gaplessly by `GeminiPcmPlayback` (a thin specialization of the shared `RealtimePcmPlayback`), which also backs `IsAudioPlaying` and flushes on barge-in (obligation #3).
+- The server-built `SessionConfig` carries `{ model, config }` (system instruction, tools, transcription, modalities) and the model's facts on its endpoint; the client applies the config at `live.connect`.
+- **Inbound video**: `maxInboundVideoStreams` and `maxInboundVideoRate` come from the minted `SessionConfig` (the model profile: Gemini 3.8 Live and Extended Thinking take one stream at 1 frame per second; a model without video input takes none). `negotiateTracks` keys requested tracks by direction + modality + `SourceID` and marks streams beyond the model's maximum `'unsupported'` with the reason. Frames reach the model only through the session's `VideoSourceArbiter`; a camera passed to `Connect` becomes its `'camera'` source.
+- **Resumption**: when Google announces the connection is ending (`goAway`; on Vertex AI it came about 9 minutes into a connection, with 30 s left) or the socket drops, the session moves to a new connection with Google's resumption handle (`ResumptionHandle`). The avatar's player, its element and the last frame shown carry over.
+- **Agent video (a live avatar)**: when the minted config grants an avatar (its `avatar` block: `output`, `encoding`, `audioMuxed`) and the host shows it (the outbound video track is live), the client creates a player through the `CreateVideoPlayout(options)` seam and hands its `Source` to the host at connect. Model parts are routed by type: a `video/*` part, or one whose bytes open with an MP4 box (`ftyp`, `moov`, `moof`, `styp`) whatever type it names, goes to the player as a `RealtimeVideoFrame`; PCM goes to the voice; anything else is dropped and reported once. The session's first init segment decides whether the MP4 carries the voice; when it does, PCM never plays the voice twice (the turn's first video part flushes the PCM the turn queued, and later PCM in the turn is dropped), and the player's audio is routed into the PCM playback's graph so the meter and the recording carry it. `generationComplete` ends the turn's video (`turnComplete` as the fallback); `interrupted` flushes it and drops the turn's late parts until its `turnComplete`. The seconds of video generated are reported per turn in `OnUsage` updates of their own (`OutputTokenDetails.VideoSeconds`). A host that shows no agent video, or a browser that can't play the avatar's type, gets an audio-only session and one log line (reason `host` or `browser`).
+
+### `GeminiEnterpriseRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'gemini-enterprise')`
+
+Gemini Live on **Gemini Enterprise** (Vertex AI), minted by the server driver `GeminiEnterpriseRealtime` in `@memberjunction/ai-vertex`. Gemini Enterprise has no browser credential, so its sessions are relay sessions only (`Transport: 'relay'`): the client runs the web SDK in Vertex mode with the relay URL as its base URL, and MJAPI holds the Google credential, writes the setup and filters what the browser sends. The client states its response modalities on every connect (VIDEO while the avatar shows), because the relay reads an AUDIO-only setup as a request to drop the avatar. Everything else is `GeminiRealtimeClient`'s, avatar playout and resumption (through the same relay URL) included. Call `LoadGeminiEnterpriseRealtimeClient()` from a static code path.
 
 ### `ElevenLabsRealtimeClient` — `@RegisterClass(BaseRealtimeClient, 'elevenlabs')`
 
 - **Transport**: raw WebSocket against the server-minted **signed URL** — the `EphemeralToken` *is* the `wss://…&token=…` URL (no API key in the browser). Handshake: open → send `conversation_initiation_client_data` carrying the server-authored prompt override (from the `SessionConfig` pact `{ agentId, overrides, config }`) → wait for `conversation_initiation_metadata` → negotiate PCM rates from the metadata's audio-format tags → build the audio plane → `'listening'` (obligation #7). Non-PCM telephony formats (`ulaw_8000`) degrade loudly to the 16 kHz default with a warning.
-- **Audio**: the shared PCM plane (`createPcmMicCapture` up as bare-key `user_audio_chunk` frames, `RealtimePcmPlayback` down from `audio` events) at the **negotiated** rates; `IsAudioPlaying` from the playout clock.
+- **Audio**: the shared PCM plane (`CreatePcmMicCapture` up as bare-key `user_audio_chunk` frames, `RealtimePcmPlayback` down from `audio` events) at the **negotiated** rates; `IsAudioPlaying` from the playout clock.
 - **Capability deltas**: transcripts are **finals-only** (no interim deltas; `agent_response_correction` re-finalizes a barged-in turn with what was actually spoken — treat it as the authoritative replacement); `SendContextNote` is **native** (`contextual_update`, sent even mid-response); `RequestSpokenUpdate` is **emulated** as a `user_message` (queued behind in-flight responses; narration kind stamped at send time — there is no `response.created`-style frame to stamp on); there is **no cancel frame** — `CancelActiveResponse` flushes the locally-owned playout (residual server generation is simply never played); **no usage events**; `SendToolResult` is exactly-once (duplicate call ids dropped with a warning).
 - **Busy mapping**: set on the first `audio` / `agent_response` of a turn, cleared on `agent_response_complete` / `interruption` / `client_tool_call` (obligation #2 — no envelope frames exist; state is inferred frame-by-frame).
 
@@ -98,19 +131,109 @@ Errors (`RealtimeClientError`): `Fatal: true` means the session is unusable (tra
 - **Barge-in**: `input.speech.started` while output is active is the snappy flush point (~300 ms faster than waiting per the provider's guidance); `reply.done` `status: 'interrupted'` is the authoritative verdict / fallback flush. A speech start while idle is a normal turn, NOT an interruption.
 - **Teardown**: `Disconnect()` sends `session.end` before closing — skipping it leaves a billable 30-second resume hold.
 
+`xAIRealtimeClient` (`'xai'`) and `HuggingFaceRealtimeClient` (`'huggingface'`) are thin subclasses of the OpenAI-protocol websocket layer; see [the OpenAI-protocol client family](#openai-protocol-client-family-architecture) below. No driver but the two Gemini clients hands over agent video; `realtime-video-driver-audit.test.ts` keeps that true for every registered driver.
+
 ### Shared audio plane (`src/audio/`)
 
-The three websocket drivers (Gemini, ElevenLabs, AssemblyAI — everyone whose audio rides the socket rather than WebRTC) share one browser audio pipeline instead of reimplementing it per provider:
+The websocket drivers (Gemini, ElevenLabs, AssemblyAI, and the OpenAI-protocol websocket family — everyone whose audio rides the socket rather than WebRTC) share one browser audio pipeline instead of reimplementing it per provider:
 
-- **`createPcmMicCapture(micStream, sampleRate, onPcmChunk)`** (`micCapture.ts`) — `AudioWorklet`-based mic capture resampled to the requested rate, delivering base64 PCM16 chunks; the worklet is loaded from a Blob URL so the package ships no asset files.
-- **`RealtimePcmPlayback`** (`pcmPlayback.ts`) — gapless playhead-clock scheduling of inbound PCM16, backing `IsAudioPlaying` precisely ("scheduled audio extends beyond the context's current time") with an instant `Flush()` for barge-in / cancel.
+- **`CreatePcmMicCapture(micStream, sampleRate, onPcmChunk)`** (`micCapture.ts`) — `AudioWorklet`-based mic capture resampled to the requested rate, delivering base64 PCM16 chunks; the worklet is loaded from a Blob URL so the package ships no asset files. Its `Rebind` follows a replaced microphone track.
+- **`RealtimePcmPlayback`** (`pcmPlayback.ts`) — gapless playhead-clock scheduling of inbound PCM16, backing `IsAudioPlaying` precisely ("scheduled audio extends beyond the context's current time") with an instant `Flush()` for barge-in / cancel. `Enqueue(pcm16, mediaTimeMs?)` takes the chunk's place on the stream's media timeline, and the playback is also an `IPlaybackClock` (`CurrentTimeMs`: the media time of the audio heard now, less the output latency the browser reports), which a video player can follow for lip sync. `CreateMeter()` taps its output for the call's visuals, `GetOutputStream()` hands it to a recorder, and `ConnectMediaElement(element)` plays a media element's audio through the same graph (an avatar's voice is in its video).
 - **`pcmUtils.ts`** — base64 ↔ `ArrayBuffer` and PCM conversion helpers.
 
 Drivers expose these through overridable `protected` creation seams (`createMicCapture` / `createPlayback`), so tests run with no audio hardware.
 
+## Media (`/media`)
+
+The browser media code the realtime call, the LiveKit meeting room and any other host share. None of it starts a device on its own or talks to a vendor.
+
+### Capture
+
+- **`LocalMediaController`** (implements `ILocalMediaController`, which another platform or a test can supply): the user's camera and microphone. `State$` / `State` report each kind's status, device and failure; `RefreshDevices()` lists devices and the controller follows `devicechange`. `Start(kind, deviceId?)` opens a kind (always resolves; a failure is a result that keeps the browser's error name); `SwitchDevice(kind, deviceId)` moves a live kind to another device inside the same `MediaStream`, releasing the old device first (mobile browsers open one camera at a time) and reopening it if the new one fails; a lost device falls back to the default. It never starts on its own, and captures at the device's native rate. A consumer bound to a track rather than the stream (a Web Audio node, a WebRTC sender) rebinds when `State$` reports a new device.
+- **`RequestDisplayCapture(options)`**: asks the browser to share a screen, a window, a tab or one panel of the page, from a user gesture. It resolves `started` (a `DisplayCapture` with its `Stream`, `Track`, actual `Surface`, `Label`, `OnEnded` and `Stop`), `cancelled`, or `failed` (`unsupported`, `panel-unsupported`, `panel-wrong-surface`, `denied`, `error`); it never rejects. The picker hints (`PreferredSurface` and the `selfBrowserSurface` / `surfaceSwitching` / `preferCurrentTab` family) are read only by Chromium browsers. **A panel capture** (`Panel`, with an optional `PanelLabel` the capture carries for the UI and the agent) shares one element: the user picks this tab and the stream is narrowed with Element Capture (Chromium 132+, the element alone) or else Region Capture (Chromium 104+, the element's rectangle). Element Capture needs the element to be its own stacking context, so it gets `isolation: isolate` while shared. Sharing anything other than this tab stops the capture. `GetDisplayCaptureSupport()` says what the browser can do (desktop Firefox and Safari can share, but not a single panel; no mobile browser can share); `CapturedSurfaceOf(track)` reads what a track shares.
+- **`FrameSampler`**: takes still frames from any `MediaStream` at the rate the caller passes (the negotiated `InboundVideoRate`) as base64 JPEG or PNG, with an optional `MaxDimension`. It has no ceiling of its own, so a self-view runs at the camera's native rate while the model gets only the frames it takes.
+- **`MediaPreview`**: a lobby's camera and microphone before joining (a self-view stream, a microphone level, the devices, the user's choices), released before the call or meeting opens its own. The realtime call checks the camera inside the call instead (`RealtimeCaptures` in `@memberjunction/realtime-runtime`).
+- Deprecated wrappers kept for existing callers: `CreateCameraCapture`, `CreateScreenCapture` and `CreateStreamFrameCapture` (`frameCapture.ts`), now thin wrappers over the modules above.
+
+### Showing video
+
+- **`MediaVideoSource`** is what every renderer shows: `{ Kind: 'stream', Stream }` (a camera, a shared screen, a WebRTC track) or `{ Kind: 'element', Attach(element) }` (a player that must own the `<video>` element, such as `VideoPlayout`).
+- **`AttachVideoSource(source, element)`** puts either into a `<video>` and returns a detach function. Streams play muted (a video surface never plays audio; each voice plays once, elsewhere); an element source decides its own audio. Detaching never stops the source's tracks. UI code uses it instead of a vendor's `track.attach`.
+- The UI model types live here too: `MediaParticipant` (identity, role, speaking, videos by kind, `PreferredVideo`, `IsMuted`, `ConnectionQuality`, `GetAudioLevel`, `AgentCanSee`), `MediaDevice`, `MediaDeviceSelection` (including the `speaker` kind), `MediaAgentState`, `MediaConnectionStatus`, `MediaDisconnectReason`, `MediaConnectionQuality`.
+
+### Playing the agent's video: `VideoPlayout`
+
+`VideoPlayout` plays a model's video into one `<video>` element at a time, whatever form it arrives in. A driver creates one when the agent's video is granted and shown, and hands its `Source` to the host once.
+
+- `Append(frame)` takes each `RealtimeVideoFrame` (`@memberjunction/ai`), in order: a piece of fragmented MP4, an encoded chunk or an image.
+- `EndOfTurn()` lets the turn play to its end and hold the last frame; `Flush()` (barge-in) drops everything not yet shown and holds the last frame; between turns, and while a session resumes on a new connection, the element keeps showing the last frame. Across a resume the player keeps its element and decoder, and the next connection's init segment and fragments continue in the same buffer.
+- `IsPlaying` is true while it plays with media ahead of the playhead (an avatar audibly speaking). `CarriesVoice` unmutes the element for an MP4 that carries the voice; `VideoPlayoutOptions.OnElementAttached` lets a driver route the element's audio into its own Web Audio graph.
+- `OnProblem(handler)` reports each kind once: `unsupported`, `no-decoder`, `fragment-before-init`, `append-failed`, `pending-overflow`. `VideoPlayout.IsSupported(mimeType)` says at once whether some decoder plays a type in this browser.
+
+**Decoders.** The player keeps one decoder at a time, chosen per frame type from `VideoFrameDecoderRegistry`:
+
+| Decoder | Plays | How |
+|---|---|---|
+| `'mse-fmp4'` | `video/mp4` fragmented MP4 (a Gemini avatar) | Media Source Extensions, or `ManagedMediaSource` on iOS Safari 17.1+ |
+| `'webcodecs'` | `video/h264` (Annex B, codec string in the `codecs` parameter), `video/vp8`, `video/vp9`, `video/av1` chunks | A WebCodecs `VideoDecoder`, drawn into a `MediaStream` the element shows |
+| `'image'` | `image/jpeg`, `image/png`, `image/webp` | `createImageBitmap`, drawn the same way |
+
+A decoder that gives up (`Failed`) is not chosen again for that type, and the next that can play it takes over. An app adds a decoder with `VideoFrameDecoderRegistry.Instance.Register({ Name, Kind, Priority, CanPlay, Create })` (the same name replaces a built-in); `CanPlay` must answer at once, and `Create` returns an `IVideoFrameDecoder` that plays into the host's element.
+
+**The playback clock.** Pass the voice's playback as `VideoPlayoutOptions.Clock` (an `IPlaybackClock`; `RealtimePcmPlayback` is one) when the driver times its PCM and its frames on one media timeline. The chunk and image decoders then show each frame when the voice reaches its `PresentationTimeMs`: a frame more than 500 ms behind the voice is dropped, one ahead waits, one more than 2 s ahead is dropped, and a new stream waits for its own voice. Before the clock has read anything (at the start, or after a barge-in) frames wait up to 500 ms, then go by their own times; when the voice ends before the video, frames carry on by their own times. MSE ignores the clock: an MP4 that carries the voice keeps face and voice together itself.
+
+### Which source the model sees: `VideoSourceArbiter`
+
+Several things can offer the model pictures at once (a whiteboard, a remote browser, a component, a camera, a shared screen); a model accepts a bounded number of inbound video streams (`MaxInboundVideoStreams`, 1 for Gemini Live today). `VideoSourceArbiter` is the **single writer** of inbound video per connection (`VideoSourceArbiter.ForSink(client)`), so there is exactly one place that decides what the model sees:
+
+* sources `RegisterSource({ SourceID, Label, Kind, ChannelKey, Enabled })` and push frames with `PushFrame(sourceId, base64, mime)`; the arbiter forwards up to the model's stream count (N-stream models pass every enabled source through) and **paces** to the negotiated rate with jitter headroom (`MinVideoFrameSpacingMs`). A source registered with `Enabled: false` starts off, and the model is not told about it.
+* with more enabled sources than streams it picks by a **policy that is data** (`DEFAULT_VIDEO_SOURCE_POLICY`: the user's pick, the most recently started capture, the focused surface, the newest), and tells the model in the second person on every switch (`[You can now see: <label>]`), when a source is turned off (`[You can no longer see: <label> (the user turned it off)]`) and when it is turned back on. The wording can be replaced through the arbiter's options.
+* `SelectSource`, `SetFocusedSource` / `SetFocusedChannel`, `SetSourceEnabled(id, enabled, notify)`, `GetSources()`, `OnChange()` back the "agent can see" control.
+* it talks to a narrow `IVideoFrameSink` (stream count, rate, `SendVideoFrame(data, mime, sourceId?)`, `SendContextNote`), so a driver other than Gemini adopts it by implementing four members.
+
+`ChannelInboundVideoBridge` registers a channel as a source and polls its frames at the negotiated rate; its `Rate` option is deprecated. The pacing math (`NominalVideoFrameIntervalMs`, `MinVideoFrameSpacingMs`) wraps `@memberjunction/ai`'s, which the meeting bot uses too.
+
+### Layout: the media stage
+
+`LayoutMediaStage(input)` is a headless layout model: participants, placeable surfaces (`MediaSurface`: a channel's surface or a participant's video, with its `DefaultPlacement` and `AllowedPlacements`) and the user's moves go in; one stage, picture-in-picture tiles (newest first), tabs, hidden surfaces, the other participants and a screen-share split come out. The rules:
+
+- a surface goes where the user last moved it, if that placement is allowed, else to its default;
+- **one stage**: the latest move to the stage wins it, and a surface it displaces goes to its default, or to picture-in-picture when its default is also the stage;
+- with no surface on the stage, the spotlight participant holds it: a pin, the active speaker, a speaking participant, the agent, the first other participant, the user; a participant already shown through a surface (the avatar, the user's camera) is not repeated, and one in a picture-in-picture box doesn't take the spotlight;
+- the stage goes to the newest move there, so withdrawing that move hands the stage to an earlier one; a host whose "unpin" should return the stage to its default drops the stage moves first (`WithoutStageMoves`).
+
+A host that lets the user move participants' tiles passes each as a surface (`ParticipantTileSurface`, `ParticipantTileKey`, `StageParticipantIdentity`), as the meeting room does; the realtime call passes its channel surfaces. `PlacementOffStage(surface)` says where a surface goes when it leaves the stage.
+
+**Saved layouts.** `mediaLayout.ts` keeps a host's layout in a per-user settings store (`MediaLayoutSettings`: `GetSetting` / `SetSettingDebounced`, which `UserInfoEngine` fits): the moves in order (`RecordPlacementMove`, `ParsePlacementMoves` / `SerializePlacementMoves`), the picture-in-picture boxes as fractions of the stage (`MediaPipRect`, `RecordPipRect`, `ParsePipRects` / `SerializePipRects`) and, optionally, whether the user hid their self-view (`ParseSelfViewHidden` / `SerializeSelfViewHidden`; only a saved `true` hides it). `MediaLayoutPrefs` reads and writes all three under the host's `MediaLayoutKeys` (`Moves`, `PipRects`, optional `SelfViewHidden`), keeps the newest 50 moves and boxes, and treats a store that isn't ready as nothing saved. The realtime call saves under `mj.realtime.placement.v1` and `mj.realtime.pip.v1`, the meeting room under its own `mj.livekit.*` keys.
+
+### Audio meter
+
+`RealtimeAudioMeter` taps a `MediaStream` (`ForStream`, `ForMicStream`) or a node in an existing graph (`ForContextNode`) and reports a level and 9 frequency bins; it returns `null` where Web Audio is missing. The smoothing every meter display shares is pure: `GateAudioLevel`, `SmoothAudioLevel`, `SmoothAudioBars`, `SynthesizeAudioBars`, and `AudioLevelSmoother`, which puts them together for one source; each display passes its own attack and decay.
+
+## Testing kit (`/testing`)
+
+`@memberjunction/ai-realtime-client/testing` is the video provider conformance kit: a provider's test runs it against the provider's real client driver, with a fake transport, to prove the driver follows the realtime video rules (obligation #12). `RunRealtimeVideoConformance(driverFactory)` resolves to one result per check (VC01-VC15, VF01-VF02), and `ListRealtimeVideoConformanceChecks(driverFactory)` gives any test runner one test per check:
+
+```ts
+import { ListRealtimeVideoConformanceChecks } from '@memberjunction/ai-realtime-client/testing';
+
+for (const check of ListRealtimeVideoConformanceChecks(() => new MyProviderHarness())) {
+    (check.SkipReason ? it.skip : it)(`${check.Id} ${check.Title}`, () => check.Run());
+}
+```
+
+The provider writes the harness (`IRealtimeVideoConformanceHarness`): what its server mints, its client driver with its creation seams wired to the kit's recorders (`RecordingVideoPlayout`, `RecordingPcmPlayback`), and the model's side of the wire, sending video as `RealtimeVideoFrame`s of the provider's kind. The harness contract, the traits and the check table are in the [Co-Agents guide §12](../../../guides/REALTIME_CO_AGENTS_GUIDE.md#12-adding-a-video-capable-realtime-provider). `src/__tests__/video-conformance/` runs the kit against Gemini (Developer API and Gemini Enterprise), OpenAI and a synthetic raw-frame provider; the synthetic one is the worked example.
+
 ## Driver-author obligations
 
-`BaseRealtimeClient`'s doc header carries the authoritative client-side **"DRIVER AUTHOR OBLIGATIONS"** block (8 numbered rules, paid for in live debugging — the mirror of the server-side block on `BaseRealtimeModel` in `@memberjunction/ai`). The ones drivers trip over most: leave `'speaking'` *silently* (no state emission) when a tool call is emitted so the host's busy indicator isn't clobbered; release the busy flag at tool-call emission (deadlock guard); flush playback and report `IsAudioPlaying === false` promptly on barge-in *and* on `CancelActiveResponse`; never echo a user transcript for injected text; never drop a tool-result generation trigger (queue behind the in-flight response); surface credential expiry as a `Fatal` error; report `'listening'` only after the session config is applied; treat `SessionConfig` as a private pact between same-keyed driver halves. `RequestSpokenUpdate` has an explicit collision rule: when a response is already in flight the driver must queue or *skip* the update (skipping is fine — narration is disposable by contract); host-side `IsBusy`/`IsAudioPlaying` gating is for timing quality, the driver is the safety net.
+`BaseRealtimeClient`'s doc header carries the authoritative client-side **"DRIVER AUTHOR OBLIGATIONS"** block (12 numbered rules, paid for in live debugging — rules 1-8 mirror the server-side block on `BaseRealtimeModel` in `@memberjunction/ai`). The ones drivers trip over most: leave `'speaking'` *silently* (no state emission) when a tool call is emitted so the host's busy indicator isn't clobbered; release the busy flag at tool-call emission (deadlock guard); flush playback and report `IsAudioPlaying === false` promptly on barge-in *and* on `CancelActiveResponse`; never echo a user transcript for injected text; never drop a tool-result generation trigger (queue behind the in-flight response); surface credential expiry as a `Fatal` error; report `'listening'` only after the session config is applied; treat `SessionConfig` as a private pact between same-keyed driver halves. `RequestSpokenUpdate` has an explicit collision rule: when a response is already in flight the driver must queue or *skip* the update (skipping is fine — narration is disposable by contract); host-side `IsBusy`/`IsAudioPlaying` gating is for timing quality, the driver is the safety net.
+
+The client block adds four:
+
+9. **Audio metering is a capability**: wire the meters for the audio planes the driver owns, and release them on disconnect.
+10. **Follow a replaced microphone track** (`ReplaceMicrophone`): `replaceTrack` on WebRTC senders, `IPcmMicCapture.Rebind` on PCM capture, a new input meter; mute follows the track's `enabled`.
+11. **Refuse a transport you don't speak**: `AssertTransportSupported` first in `Connect`; opt in to relay sessions with `SupportsRelayTransport`.
+12. **The agent's video keeps the voice's rules**: hand it over once, only while the outbound video track is live; give the player each frame as the model sent it; barge-in flushes it with the voice and the cut turn's late media never plays; a turn's end lets it play out; a video that carries the voice never plays it twice; each generated second is reported once; a resumed session keeps its video; `Disconnect` releases it; a driver whose voice and video share a media timeline gives the player its voice playback as the clock. The `/testing` kit checks each rule.
 
 ## Usage
 
@@ -119,7 +242,7 @@ import { MJGlobal } from '@memberjunction/global';
 import {
     BaseRealtimeClient,
     LoadOpenAIRealtimeClient, LoadOpenAILiveClient, LoadGeminiRealtimeClient,
-    LoadElevenLabsRealtimeClient, LoadAssemblyAIRealtimeClient
+    LoadGeminiEnterpriseRealtimeClient, LoadElevenLabsRealtimeClient, LoadAssemblyAIRealtimeClient
 } from '@memberjunction/ai-realtime-client';
 
 // Tree-shaking prevention — drivers are resolved dynamically, so a static call path
@@ -127,6 +250,7 @@ import {
 LoadOpenAIRealtimeClient();
 LoadOpenAILiveClient();
 LoadGeminiRealtimeClient();
+LoadGeminiEnterpriseRealtimeClient();
 LoadElevenLabsRealtimeClient();
 LoadAssemblyAIRealtimeClient();
 
@@ -142,6 +266,7 @@ client.OnToolCall(async call => {
     const resultJson = await executeTool(call.ToolName, call.ArgumentsJson);
     client.SendToolResult(call.CallID, resultJson);
 });
+client.OnRemoteVideo(video => showAgentVideo(video)); // a MediaVideoSource; render it with AttachVideoSource
 client.OnError(e => { if (e.Fatal) endSession(); });
 
 const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
@@ -155,24 +280,26 @@ if (!client.IsBusy && !client.IsAudioPlaying) {
 await client.Disconnect();
 ```
 
-The production host is `RealtimeSessionService` (`packages/Angular/Generic/conversations`) — read it for the full policy layer (caption/transcript routing, prefix-routed client tools, narration pacing, channel plugins).
+The production host is `RealtimeSessionRuntime` (`@memberjunction/realtime-runtime`), adapted for Angular by `RealtimeSessionService` (`packages/Angular/Generic/conversations`) — read it for the full policy layer (caption/transcript routing, prefix-routed client tools, narration pacing, channel plugins, captures, the agent's video).
 
 ## Testing seams
 
-All four drivers are written against **structural transport seams** so the full event flow is unit-testable with zero network, zero WebRTC, and zero audio hardware (see `src/__tests__/`, ~4,200 lines of vitest coverage):
+Every driver is written against **structural transport seams** so the full event flow is unit-testable with zero network, zero WebRTC, and zero audio or video hardware (see `src/__tests__/`):
 
 - **OpenAI**: `IRealtimePeerConnection`, `IRealtimeDataChannel`, `IRealtimeAudioSink` — created through overridable `protected` factory methods (`createPeerConnection`, `createAudioSink`, …); tests subclass the driver and inject fakes, then drive provider-shaped JSON frames through the data channel.
-- **Gemini**: `GeminiLiveClientSession` (typed subset of the SDK `Session`), `IGeminiMicCapture`, `IGeminiAudioPlayback` — the `connectLiveSession` / capture / playback boundaries are the only things tests replace.
+- **Gemini**: `GeminiLiveClientSession` (typed subset of the SDK `Session`), `IGeminiMicCapture`, `IGeminiAudioPlayback`, `IAvatarVideoPlayout` — the `connectLiveSession` / capture / playback / `CreateVideoPlayout` boundaries are the only things tests replace. The Gemini Enterprise tests run the real web SDK over a fake relay socket.
 - **ElevenLabs / AssemblyAI**: `IElevenLabsClientSocket` / `IAssemblyAIClientSocket` (assignable-handler websocket seams behind `createSocket`) plus the shared `createMicCapture` / `createPlayback` seams — tests drive provider-shaped frames straight through the socket fake.
-- Shared fakes live in `src/__tests__/helpers/realtime-fakes.ts`.
+- Shared fakes live in `src/__tests__/helpers/` (`realtime-fakes.ts`, and fakes for media devices, display capture, Media Source, WebCodecs and Web Audio); fMP4 fixtures are in `src/__tests__/fixtures/`.
 
-If you write a new driver, follow the same shape: every wire/hardware boundary behind a `protected` overridable seam, asserted with scripted provider frames.
+If you write a new driver, follow the same shape: every wire/hardware boundary behind a `protected` overridable seam, asserted with scripted provider frames. A driver that plays the agent's video creates its player through such a seam, so the conformance kit can record it.
 
 ## Related
 
-- [`guides/REALTIME_CO_AGENTS_GUIDE.md`](../../../guides/REALTIME_CO_AGENTS_GUIDE.md) — the flagship feature guide
-- `@memberjunction/ai` — `BaseRealtimeModel`, `IRealtimeSession`, `ClientRealtimeSessionConfig`, `RealtimeToolDefinition`
+- [`guides/REALTIME_CO_AGENTS_GUIDE.md`](../../../guides/REALTIME_CO_AGENTS_GUIDE.md) — the flagship feature guide, including §12 on video-capable providers
+- `@memberjunction/ai` — `BaseRealtimeModel`, `IRealtimeSession`, `ClientRealtimeSessionConfig`, `RealtimeVideoFrame`, the fMP4 reader, `RealtimeToolDefinition`
 - `@memberjunction/ai-agents` — `RealtimeSessionRunner`, `RealtimeToolBroker`, `RealtimeClientSessionService`
+- `@memberjunction/realtime-runtime` — `RealtimeSessionRuntime`, the host-side session (channels, captures, the agent's video)
+- `@memberjunction/ng-realtime-media` — the Angular media widgets that render `/media` models
 - `@memberjunction/ng-conversations` — the Angular host (overlay, channels, session review)
 - `@memberjunction/ng-whiteboard` — the generic whiteboard the Whiteboard channel surfaces
 

package/dist/audio/audioMeter.d.ts

@@ -14,6 +14,12 @@
  * SSR) the factories return `null` and callers degrade to "no metering" — the call UI then
  * keeps its turn-state-driven animations. The DSP math ({@link ComputeRmsLevel},
  * {@link BucketizeFrequencyData}) is exported pure so it unit-tests without Web Audio.
+ *
+ * The second half of the file is the SMOOTHING every meter display shares, also pure: a noise gate
+ * ({@link GateAudioLevel}), the attack/decay step ({@link SmoothAudioLevel}, {@link SmoothAudioBars}),
+ * bars synthesized from a level for sources with no spectrum ({@link SynthesizeAudioBars}), and
+ * {@link AudioLevelSmoother}, which puts them together for one source. Each display passes its own attack
+ * and decay, so unifying the code changed nothing on screen.
  */
 /** The number of frequency bins the call UI's EQ renders (and meters therefore produce). */
 export declare const REALTIME_AUDIO_BIN_COUNT = 9;
@@ -98,4 +104,69 @@ export declare class RealtimeAudioMeter implements IRealtimeAudioMeter {
     /** @inheritdoc */
     Close(): void;
 }
+/** Attack the call overlay uses: how far a rising level moves toward its target in one step. */
+export declare const DEFAULT_AUDIO_ATTACK = 0.5;
+/** Decay the call overlay uses: how far a falling level moves toward its target in one step. */
+export declare const DEFAULT_AUDIO_DECAY = 0.12;
+/**
+ * Soft-knee noise gate: a level at or below `gate` is silence (0); above it, the rest of the range rescales
+ * to 0..1, so the gate never visibly clips the bottom off real speech. Pure.
+ */
+export declare function GateAudioLevel(level: number, gate: number): number;
+/**
+ * One smoothing step from `previous` toward `next`: by `attack` while rising, so speech onset reads at once,
+ * and by `decay` while falling, so the level rings down instead of snapping shut. Clamped to 0..1. Pure.
+ */
+export declare function SmoothAudioLevel(previous: number, next: number, attack?: number, decay?: number): number;
+/**
+ * One smoothing step for every bar: toward its target, or toward silence when there is no target. Pure.
+ *
+ * @param previous The bars now.
+ * @param target The bars to move toward (a missing entry counts as 0), or `null` to fall toward silence.
+ */
+export declare function SmoothAudioBars(previous: readonly number[], target: readonly number[] | null, attack?: number, decay?: number): number[];
+/**
+ * Bars for a source that reports only a level, such as a LiveKit participant: center bars taller than the
+ * edges, with a little per-bar variation so they don't move in lockstep. Pure.
+ */
+export declare function SynthesizeAudioBars(level: number, count: number): number[];
+export interface AudioLevelSmootherOptions {
+    /** How many bars each frame carries. */
+    BarCount: number;
+    /** See {@link SmoothAudioLevel}. Defaults to {@link DEFAULT_AUDIO_ATTACK}. */
+    Attack?: number;
+    /** See {@link SmoothAudioLevel}. Defaults to {@link DEFAULT_AUDIO_DECAY}. */
+    Decay?: number;
+    /** A smoothed level below this reads as silence: the frame's level is 0 and the bars fall. Defaults to 0. */
+    SilenceFloor?: number;
+}
+/** One smoothed frame for a level display. */
+export interface AudioLevelFrame {
+    /** The smoothed level, 0..1; 0 while silent. */
+    Level: number;
+    /** The smoothed bars, each 0..1. */
+    Bars: number[];
+    /** Whether the smoothed level is under the silence floor. */
+    IsSilent: boolean;
+}
+/**
+ * Smooths one audio source for display. Feed it the latest level (0..1) each animation frame, with the
+ * source's spectrum bars when it has them (an analyser tap); without them, bars are synthesized from the
+ * level ({@link SynthesizeAudioBars}). Pure: no Web Audio, no clock.
+ */
+export declare class AudioLevelSmoother {
+    private readonly options;
+    private level;
+    private bars;
+    constructor(options: AudioLevelSmootherOptions);
+    /**
+     * Folds in the latest level, and the source's spectrum bars when it has them.
+     *
+     * @param level The latest raw level, 0..1 (clamped).
+     * @param spectrum The source's current spectrum bars, or nothing to synthesize bars from the level.
+     */
+    Next(level: number, spectrum?: readonly number[] | null): AudioLevelFrame;
+    /** Returns to silence. */
+    Reset(): void;
+}
 //# sourceMappingURL=audioMeter.d.ts.map

package/dist/audio/audioMeter.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"audioMeter.d.ts","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,4FAA4F;AAC5F,eAAO,MAAM,wBAAwB,IAAI,CAAC;AAE1C;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,eAAe,EAAE,UAAU,GAAG,MAAM,CAWnE;AAED;;;;;GAKG;AACH,wBAAgB,sBAAsB,CAAC,cAAc,EAAE,UAAU,EAAE,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE,CAiBrH;AAED;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAChC,mDAAmD;IACnD,KAAK,IAAI,MAAM,CAAC;IAChB,kGAAkG;IAClG,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC/B,uFAAuF;IACvF,KAAK,IAAI,IAAI,CAAC;CACjB;AAED;;;;;GAKG;AACH,qBAAa,kBAAmB,YAAW,mBAAmB;IAC1D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAe;IACxC,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAsB;IACnD,0GAA0G;IAC1G,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA0B;IACrD,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA0B;IACpD,OAAO,CAAC,MAAM,CAAS;IAEvB,OAAO;IAQP;;;;;OAKG;IACH,OAAO,CAAC,MAAM,CAAC,gBAAgB;IAW/B;;;;OAIG;WACW,SAAS,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAQvE;;;;;;;;;;OAUG;WACW,YAAY,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAa1E;;;;OAIG;WACW,cAAc,CAAC,OAAO,EAAE,YAAY,EAAE,MAAM,EAAE,SAAS,GAAG,kBAAkB,GAAG,IAAI;IAUjG,OAAO,CAAC,MAAM,CAAC,cAAc;IAO7B,kBAAkB;IACX,KAAK,IAAI,MAAM;IAQtB,kBAAkB;IACX,IAAI,CAAC,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE;IAQ/D,kBAAkB;IACX,KAAK,IAAI,IAAI;CAsBvB"}
+{"version":3,"file":"audioMeter.d.ts","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,4FAA4F;AAC5F,eAAO,MAAM,wBAAwB,IAAI,CAAC;AAE1C;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,eAAe,EAAE,UAAU,GAAG,MAAM,CAWnE;AAED;;;;;GAKG;AACH,wBAAgB,sBAAsB,CAAC,cAAc,EAAE,UAAU,EAAE,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE,CAiBrH;AAED;;;GAGG;AACH,MAAM,WAAW,mBAAmB;IAChC,mDAAmD;IACnD,KAAK,IAAI,MAAM,CAAC;IAChB,kGAAkG;IAClG,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC/B,uFAAuF;IACvF,KAAK,IAAI,IAAI,CAAC;CACjB;AAED;;;;;GAKG;AACH,qBAAa,kBAAmB,YAAW,mBAAmB;IAC1D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAe;IACxC,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAsB;IACnD,0GAA0G;IAC1G,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA0B;IACrD,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA0B;IACpD,OAAO,CAAC,MAAM,CAAS;IAEvB,OAAO;IAQP;;;;;OAKG;IACH,OAAO,CAAC,MAAM,CAAC,gBAAgB;IAW/B;;;;OAIG;WACW,SAAS,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAQvE;;;;;;;;;;OAUG;WACW,YAAY,CAAC,MAAM,EAAE,WAAW,GAAG,kBAAkB,GAAG,IAAI;IAa1E;;;;OAIG;WACW,cAAc,CAAC,OAAO,EAAE,YAAY,EAAE,MAAM,EAAE,SAAS,GAAG,kBAAkB,GAAG,IAAI;IAUjG,OAAO,CAAC,MAAM,CAAC,cAAc;IAO7B,kBAAkB;IACX,KAAK,IAAI,MAAM;IAQtB,kBAAkB;IACX,IAAI,CAAC,KAAK,GAAE,MAAiC,GAAG,MAAM,EAAE;IAQ/D,kBAAkB;IACX,KAAK,IAAI,IAAI;CAsBvB;AAID,gGAAgG;AAChG,eAAO,MAAM,oBAAoB,MAAM,CAAC;AAExC,gGAAgG;AAChG,eAAO,MAAM,mBAAmB,OAAO,CAAC;AAExC;;;GAGG;AACH,wBAAgB,cAAc,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,MAAM,CAKlE;AAED;;;GAGG;AACH,wBAAgB,gBAAgB,CAAC,QAAQ,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,GAAE,MAA6B,EAAE,KAAK,GAAE,MAA4B,GAAG,MAAM,CAGnJ;AAED;;;;;GAKG;AACH,wBAAgB,eAAe,CAC3B,QAAQ,EAAE,SAAS,MAAM,EAAE,EAC3B,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,IAAI,EAChC,MAAM,GAAE,MAA6B,EACrC,KAAK,GAAE,MAA4B,GACpC,MAAM,EAAE,CAEV;AAED;;;GAGG;AACH,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE,CAQ1E;AAED,MAAM,WAAW,yBAAyB;IACtC,wCAAwC;IACxC,QAAQ,EAAE,MAAM,CAAC;IACjB,8EAA8E;IAC9E,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,6EAA6E;IAC7E,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,6GAA6G;IAC7G,YAAY,CAAC,EAAE,MAAM,CAAC;CACzB;AAED,8CAA8C;AAC9C,MAAM,WAAW,eAAe;IAC5B,gDAAgD;IAChD,KAAK,EAAE,MAAM,CAAC;IACd,oCAAoC;IACpC,IAAI,EAAE,MAAM,EAAE,CAAC;IACf,6DAA6D;IAC7D,QAAQ,EAAE,OAAO,CAAC;CACrB;AAED;;;;GAIG;AACH,qBAAa,kBAAkB;IAIf,OAAO,CAAC,QAAQ,CAAC,OAAO;IAHpC,OAAO,CAAC,KAAK,CAAK;IAClB,OAAO,CAAC,IAAI,CAAW;gBAEM,OAAO,EAAE,yBAAyB;IAI/D;;;;;OAKG;IACI,IAAI,CAAC,KAAK,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,SAAS,MAAM,EAAE,GAAG,IAAI,GAAG,eAAe;IAUhF,0BAA0B;IACnB,KAAK,IAAI,IAAI;CAIvB"}

package/dist/audio/audioMeter.js

@@ -14,6 +14,12 @@
  * SSR) the factories return `null` and callers degrade to "no metering" — the call UI then
  * keeps its turn-state-driven animations. The DSP math ({@link ComputeRmsLevel},
  * {@link BucketizeFrequencyData}) is exported pure so it unit-tests without Web Audio.
+ *
+ * The second half of the file is the SMOOTHING every meter display shares, also pure: a noise gate
+ * ({@link GateAudioLevel}), the attack/decay step ({@link SmoothAudioLevel}, {@link SmoothAudioBars}),
+ * bars synthesized from a level for sources with no spectrum ({@link SynthesizeAudioBars}), and
+ * {@link AudioLevelSmoother}, which puts them together for one source. Each display passes its own attack
+ * and decay, so unifying the code changed nothing on screen.
  */
 /** The number of frequency bins the call UI's EQ renders (and meters therefore produce). */
 export const REALTIME_AUDIO_BIN_COUNT = 9;
@@ -190,4 +196,81 @@ export class RealtimeAudioMeter {
         }
     }
 }
+// ── Smoothing: the pure math every meter display shares ─────────────────────────────────────
+/** Attack the call overlay uses: how far a rising level moves toward its target in one step. */
+export const DEFAULT_AUDIO_ATTACK = 0.5;
+/** Decay the call overlay uses: how far a falling level moves toward its target in one step. */
+export const DEFAULT_AUDIO_DECAY = 0.12;
+/**
+ * Soft-knee noise gate: a level at or below `gate` is silence (0); above it, the rest of the range rescales
+ * to 0..1, so the gate never visibly clips the bottom off real speech. Pure.
+ */
+export function GateAudioLevel(level, gate) {
+    if (!Number.isFinite(level) || level <= gate) {
+        return 0;
+    }
+    return Math.min(1, (level - gate) / (1 - gate));
+}
+/**
+ * One smoothing step from `previous` toward `next`: by `attack` while rising, so speech onset reads at once,
+ * and by `decay` while falling, so the level rings down instead of snapping shut. Clamped to 0..1. Pure.
+ */
+export function SmoothAudioLevel(previous, next, attack = DEFAULT_AUDIO_ATTACK, decay = DEFAULT_AUDIO_DECAY) {
+    const factor = next > previous ? attack : decay;
+    return Math.min(1, Math.max(0, previous + (next - previous) * factor));
+}
+/**
+ * One smoothing step for every bar: toward its target, or toward silence when there is no target. Pure.
+ *
+ * @param previous The bars now.
+ * @param target The bars to move toward (a missing entry counts as 0), or `null` to fall toward silence.
+ */
+export function SmoothAudioBars(previous, target, attack = DEFAULT_AUDIO_ATTACK, decay = DEFAULT_AUDIO_DECAY) {
+    return previous.map((bar, i) => SmoothAudioLevel(bar, target?.[i] ?? 0, attack, decay));
+}
+/**
+ * Bars for a source that reports only a level, such as a LiveKit participant: center bars taller than the
+ * edges, with a little per-bar variation so they don't move in lockstep. Pure.
+ */
+export function SynthesizeAudioBars(level, count) {
+    const center = (count - 1) / 2;
+    return Array.from({ length: count }, (_, i) => {
+        const distance = center > 0 ? Math.abs(i - center) / center : 0;
+        const shape = 1 - distance * 0.55;
+        const variation = 0.85 + 0.15 * Math.sin(i * 1.7 + level * 6);
+        return level * shape * variation;
+    });
+}
+/**
+ * Smooths one audio source for display. Feed it the latest level (0..1) each animation frame, with the
+ * source's spectrum bars when it has them (an analyser tap); without them, bars are synthesized from the
+ * level ({@link SynthesizeAudioBars}). Pure: no Web Audio, no clock.
+ */
+export class AudioLevelSmoother {
+    constructor(options) {
+        this.options = options;
+        this.level = 0;
+        this.bars = new Array(options.BarCount).fill(0);
+    }
+    /**
+     * Folds in the latest level, and the source's spectrum bars when it has them.
+     *
+     * @param level The latest raw level, 0..1 (clamped).
+     * @param spectrum The source's current spectrum bars, or nothing to synthesize bars from the level.
+     */
+    Next(level, spectrum) {
+        const attack = this.options.Attack ?? DEFAULT_AUDIO_ATTACK;
+        const decay = this.options.Decay ?? DEFAULT_AUDIO_DECAY;
+        this.level = SmoothAudioLevel(this.level, Math.max(0, Math.min(1, level)), attack, decay);
+        const isSilent = this.level < (this.options.SilenceFloor ?? 0);
+        const target = isSilent ? null : (spectrum ?? SynthesizeAudioBars(this.level, this.options.BarCount));
+        this.bars = SmoothAudioBars(this.bars, target, attack, decay);
+        return { Level: isSilent ? 0 : this.level, Bars: [...this.bars], IsSilent: isSilent };
+    }
+    /** Returns to silence. */
+    Reset() {
+        this.level = 0;
+        this.bars = new Array(this.options.BarCount).fill(0);
+    }
+}
 //# sourceMappingURL=audioMeter.js.map

package/dist/audio/audioMeter.js.map

@@ -1 +1 @@
-{"version":3,"file":"audioMeter.js","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,4FAA4F;AAC5F,MAAM,CAAC,MAAM,wBAAwB,GAAG,CAAC,CAAC;AAE1C;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAAC,eAA2B;IACvD,IAAI,eAAe,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC/B,OAAO,CAAC,CAAC;IACb,CAAC;IACD,IAAI,UAAU,GAAG,CAAC,CAAC;IACnB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC9C,MAAM,QAAQ,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;QAClD,UAAU,IAAI,QAAQ,GAAG,QAAQ,CAAC;IACtC,CAAC;IACD,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,UAAU,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC;IAC3D,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC,CAAC;AAClC,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,sBAAsB,CAAC,cAA0B,EAAE,QAAgB,wBAAwB;IACvG,MAAM,IAAI,GAAa,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACxD,IAAI,cAAc,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC5C,OAAO,IAAI,CAAC;IAChB,CAAC;IACD,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,cAAc,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC;IACxE,MAAM,SAAS,GAAG,MAAM,GAAG,KAAK,CAAC;IACjC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC;QAC7B,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;QACxC,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC,CAAC;QACjE,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,IAAI,CAAC,GAAG,KAAK,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,cAAc,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC5D,GAAG,IAAI,cAAc,CAAC,CAAC,CAAC,CAAC;QAC7B,CAAC;QACD,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC;IACvD,CAAC;IACD,OAAO,IAAI,CAAC;AAChB,CAAC;AAeD;;;;;GAKG;AACH,MAAM,OAAO,kBAAkB;IAU3B,YAAoB,QAAsB,EAAE,YAAiC,EAAE,cAAkC,EAAE;QAF3G,WAAM,GAAG,KAAK,CAAC;QAGnB,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,YAAY,GAAG,YAAY,CAAC;QACjC,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,UAAU,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACnD,IAAI,CAAC,SAAS,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,iBAAiB,CAAC,CAAC;IAChE,CAAC;IAED;;;;;OAKG;IACK,MAAM,CAAC,gBAAgB,CAAC,MAAmB,EAAE,WAA+B;QAChF,MAAM,OAAO,GAAG,IAAI,YAAY,EAAE,CAAC;QACnC,MAAM,MAAM,GAAG,OAAO,CAAC,uBAAuB,CAAC,MAAM,CAAC,CAAC;QACvD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;QAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzB,IAAI,OAAO,CAAC,KAAK,KAAK,WAAW,EAAE,CAAC;YAChC,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;QAC1B,CAAC;QACD,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,OAAO,EAAE,WAAW,CAAC,CAAC;IAClE,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,SAAS,CAAC,MAAmB;QACvC,IAAI,CAAC;YACD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QAC3D,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;;;;;;;OAUG;IACI,MAAM,CAAC,YAAY,CAAC,MAAmB;QAC1C,IAAI,CAAC;YACD,MAAM,WAAW,GAAG,MAAM,CAAC,cAAc,EAAE,CAAC;YAC5C,IAAI,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC;YAChB,CAAC;YACD,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC;YACjD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,IAAI,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAChF,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,cAAc,CAAC,OAAqB,EAAE,MAAiB;QACjE,IAAI,CAAC;YACD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;YAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;YACzB,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAC;QAClD,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAEO,MAAM,CAAC,cAAc,CAAC,OAAqB;QAC/C,MAAM,QAAQ,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;QAC1C,QAAQ,CAAC,OAAO,GAAG,GAAG,CAAC,CAAC,4DAA4D;QACpF,QAAQ,CAAC,qBAAqB,GAAG,IAAI,CAAC;QACtC,OAAO,QAAQ,CAAC;IACpB,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,CAAC,CAAC;QACb,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;QACrD,OAAO,eAAe,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;IAC5C,CAAC;IAED,kBAAkB;IACX,IAAI,CAAC,QAAgB,wBAAwB;QAChD,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAC5C,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,oBAAoB,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;QACnD,OAAO,sBAAsB,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;IACzD,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO;QACX,CAAC;QACD,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC;YACD,IAAI,CAAC,QAAQ,CAAC,UAAU,EAAE,CAAC;QAC/B,CAAC;QAAC,MAAM,CAAC;YACL,0BAA0B;QAC9B,CAAC;QACD,kFAAkF;QAClF,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACnC,IAAI,CAAC;gBACD,KAAK,CAAC,IAAI,EAAE,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACL,qBAAqB;YACzB,CAAC;QACL,CAAC;QACD,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YACpB,KAAK,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QACnC,CAAC;IACL,CAAC;CACJ"}
+{"version":3,"file":"audioMeter.js","sourceRoot":"","sources":["../../src/audio/audioMeter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,4FAA4F;AAC5F,MAAM,CAAC,MAAM,wBAAwB,GAAG,CAAC,CAAC;AAE1C;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAAC,eAA2B;IACvD,IAAI,eAAe,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC/B,OAAO,CAAC,CAAC;IACb,CAAC;IACD,IAAI,UAAU,GAAG,CAAC,CAAC;IACnB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,eAAe,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QAC9C,MAAM,QAAQ,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;QAClD,UAAU,IAAI,QAAQ,GAAG,QAAQ,CAAC;IACtC,CAAC;IACD,MAAM,GAAG,GAAG,IAAI,CAAC,IAAI,CAAC,UAAU,GAAG,eAAe,CAAC,MAAM,CAAC,CAAC;IAC3D,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC,CAAC;AAClC,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,sBAAsB,CAAC,cAA0B,EAAE,QAAgB,wBAAwB;IACvG,MAAM,IAAI,GAAa,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACxD,IAAI,cAAc,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,EAAE,CAAC;QAC5C,OAAO,IAAI,CAAC;IAChB,CAAC;IACD,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,CAAC,KAAK,CAAC,cAAc,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC;IACxE,MAAM,SAAS,GAAG,MAAM,GAAG,KAAK,CAAC;IACjC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC;QAC7B,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;QACxC,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC,CAAC;QACjE,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,IAAI,CAAC,GAAG,KAAK,EAAE,CAAC,GAAG,GAAG,IAAI,CAAC,GAAG,cAAc,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC5D,GAAG,IAAI,cAAc,CAAC,CAAC,CAAC,CAAC;QAC7B,CAAC;QACD,IAAI,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,GAAG,CAAC,CAAC,GAAG,GAAG,KAAK,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC;IACvD,CAAC;IACD,OAAO,IAAI,CAAC;AAChB,CAAC;AAeD;;;;;GAKG;AACH,MAAM,OAAO,kBAAkB;IAU3B,YAAoB,QAAsB,EAAE,YAAiC,EAAE,cAAkC,EAAE;QAF3G,WAAM,GAAG,KAAK,CAAC;QAGnB,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,YAAY,GAAG,YAAY,CAAC;QACjC,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,UAAU,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACnD,IAAI,CAAC,SAAS,GAAG,IAAI,UAAU,CAAC,QAAQ,CAAC,iBAAiB,CAAC,CAAC;IAChE,CAAC;IAED;;;;;OAKG;IACK,MAAM,CAAC,gBAAgB,CAAC,MAAmB,EAAE,WAA+B;QAChF,MAAM,OAAO,GAAG,IAAI,YAAY,EAAE,CAAC;QACnC,MAAM,MAAM,GAAG,OAAO,CAAC,uBAAuB,CAAC,MAAM,CAAC,CAAC;QACvD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;QAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACzB,IAAI,OAAO,CAAC,KAAK,KAAK,WAAW,EAAE,CAAC;YAChC,KAAK,OAAO,CAAC,MAAM,EAAE,CAAC;QAC1B,CAAC;QACD,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,OAAO,EAAE,WAAW,CAAC,CAAC;IAClE,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,SAAS,CAAC,MAAmB;QACvC,IAAI,CAAC;YACD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;QAC3D,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;;;;;;;OAUG;IACI,MAAM,CAAC,YAAY,CAAC,MAAmB;QAC1C,IAAI,CAAC;YACD,MAAM,WAAW,GAAG,MAAM,CAAC,cAAc,EAAE,CAAC;YAC5C,IAAI,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;gBAC3B,OAAO,IAAI,CAAC;YAChB,CAAC;YACD,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC;YACjD,OAAO,kBAAkB,CAAC,gBAAgB,CAAC,IAAI,WAAW,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,CAAC;QAChF,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAED;;;;OAIG;IACI,MAAM,CAAC,cAAc,CAAC,OAAqB,EAAE,MAAiB;QACjE,IAAI,CAAC;YACD,MAAM,QAAQ,GAAG,kBAAkB,CAAC,cAAc,CAAC,OAAO,CAAC,CAAC;YAC5D,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;YACzB,OAAO,IAAI,kBAAkB,CAAC,QAAQ,EAAE,IAAI,CAAC,CAAC;QAClD,CAAC;QAAC,MAAM,CAAC;YACL,OAAO,IAAI,CAAC;QAChB,CAAC;IACL,CAAC;IAEO,MAAM,CAAC,cAAc,CAAC,OAAqB;QAC/C,MAAM,QAAQ,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;QAC1C,QAAQ,CAAC,OAAO,GAAG,GAAG,CAAC,CAAC,4DAA4D;QACpF,QAAQ,CAAC,qBAAqB,GAAG,IAAI,CAAC;QACtC,OAAO,QAAQ,CAAC;IACpB,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,CAAC,CAAC;QACb,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,qBAAqB,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;QACrD,OAAO,eAAe,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;IAC5C,CAAC;IAED,kBAAkB;IACX,IAAI,CAAC,QAAgB,wBAAwB;QAChD,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO,IAAI,KAAK,CAAS,KAAK,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QAC5C,CAAC;QACD,IAAI,CAAC,QAAQ,CAAC,oBAAoB,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;QACnD,OAAO,sBAAsB,CAAC,IAAI,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;IACzD,CAAC;IAED,kBAAkB;IACX,KAAK;QACR,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YACd,OAAO;QACX,CAAC;QACD,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC;YACD,IAAI,CAAC,QAAQ,CAAC,UAAU,EAAE,CAAC;QAC/B,CAAC;QAAC,MAAM,CAAC;YACL,0BAA0B;QAC9B,CAAC;QACD,kFAAkF;QAClF,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACnC,IAAI,CAAC;gBACD,KAAK,CAAC,IAAI,EAAE,CAAC;YACjB,CAAC;YAAC,MAAM,CAAC;gBACL,qBAAqB;YACzB,CAAC;QACL,CAAC;QACD,IAAI,IAAI,CAAC,YAAY,EAAE,CAAC;YACpB,KAAK,IAAI,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QACnC,CAAC;IACL,CAAC;CACJ;AAED,+FAA+F;AAE/F,gGAAgG;AAChG,MAAM,CAAC,MAAM,oBAAoB,GAAG,GAAG,CAAC;AAExC,gGAAgG;AAChG,MAAM,CAAC,MAAM,mBAAmB,GAAG,IAAI,CAAC;AAExC;;;GAGG;AACH,MAAM,UAAU,cAAc,CAAC,KAAa,EAAE,IAAY;IACtD,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,IAAI,EAAE,CAAC;QAC3C,OAAO,CAAC,CAAC;IACb,CAAC;IACD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;AACpD,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,gBAAgB,CAAC,QAAgB,EAAE,IAAY,EAAE,SAAiB,oBAAoB,EAAE,QAAgB,mBAAmB;IACvI,MAAM,MAAM,GAAG,IAAI,GAAG,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC;IAChD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,GAAG,CAAC,IAAI,GAAG,QAAQ,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC;AAC3E,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,eAAe,CAC3B,QAA2B,EAC3B,MAAgC,EAChC,SAAiB,oBAAoB,EACrC,QAAgB,mBAAmB;IAEnC,OAAO,QAAQ,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,gBAAgB,CAAC,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;AAC5F,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,mBAAmB,CAAC,KAAa,EAAE,KAAa;IAC5D,MAAM,MAAM,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC;IAC/B,OAAO,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;QAC1C,MAAM,QAAQ,GAAG,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC;QAChE,MAAM,KAAK,GAAG,CAAC,GAAG,QAAQ,GAAG,IAAI,CAAC;QAClC,MAAM,SAAS,GAAG,IAAI,GAAG,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,GAAG,KAAK,GAAG,CAAC,CAAC,CAAC;QAC9D,OAAO,KAAK,GAAG,KAAK,GAAG,SAAS,CAAC;IACrC,CAAC,CAAC,CAAC;AACP,CAAC;AAuBD;;;;GAIG;AACH,MAAM,OAAO,kBAAkB;IAI3B,YAA6B,OAAkC;QAAlC,YAAO,GAAP,OAAO,CAA2B;QAHvD,UAAK,GAAG,CAAC,CAAC;QAId,IAAI,CAAC,IAAI,GAAG,IAAI,KAAK,CAAS,OAAO,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAC5D,CAAC;IAED;;;;;OAKG;IACI,IAAI,CAAC,KAAa,EAAE,QAAmC;QAC1D,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,IAAI,oBAAoB,CAAC;QAC3D,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,KAAK,IAAI,mBAAmB,CAAC;QACxD,IAAI,CAAC,KAAK,GAAG,gBAAgB,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;QAC1F,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,CAAC,CAAC,CAAC;QAC/D,MAAM,MAAM,GAAG,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,QAAQ,IAAI,mBAAmB,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC;QACtG,IAAI,CAAC,IAAI,GAAG,eAAe,CAAC,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,KAAK,CAAC,CAAC;QAC9D,OAAO,EAAE,KAAK,EAAE,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,EAAE,QAAQ,EAAE,QAAQ,EAAE,CAAC;IAC1F,CAAC;IAED,0BAA0B;IACnB,KAAK;QACR,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,IAAI,KAAK,CAAS,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACjE,CAAC;CACJ"}