@voltkit-rtc/client 0.7.3 → 0.9.0

package/CHANGELOG.md

@@ -1,5 +1,91 @@
 # Changelog
 
+## 0.9.0
+
+- **AI noise suppression: `@voltkit-rtc/client/noise-suppression`** (Agora AI Denoiser, LiveKit Krisp, ZEGOCLOUD
+  AI noise suppression). `createNoiseSuppressor({ workletUrl?, wasmUrl?, simdWasmUrl? })` is RNNoise in an
+  AudioWorklet (`@sapphi-red/web-noise-suppressor` 0.4.1, MIT, loaded only when used); files from jsDelivr at the
+  pinned version unless self-hosted. In Chromium, rumble-like noise went down by more than 60 dB.
+- **`setAudioProcessor(processor | null)` and `audioProcessorInUse`** (LiveKit `setProcessor`): the microphone goes
+  through an `AudioTrackProcessor` (`{ name, process(track), destroy() }`) before it is sent. Swapped in place while
+  live; kept across device switches, a recovered microphone, reconnects and audio-quality changes. A failing
+  processor sends the microphone unprocessed and emits `error`. `mic-silent` is not checked while one is on.
+- **Virtual background and beauty: `@voltkit-rtc/client/video-effects`** (LiveKit track processors, Agora/ZEGOCLOUD
+  virtual background and beauty, TRTC video effects). `createVideoEffects({ background: 'none' | 'blur' | { image },
+  blur, beauty, wasmPath?, modelUrl?, maxFrameRate?, onOverload? })`, `update()` live, `isVideoEffectsSupported()`.
+  WebGL2: mask-weighted blur (no halo), edges refined against the picture, cover-fitted images, beauty without a
+  face model. MediaPipe's selfie segmenter (landscape model for wide pictures, square for tall), from the optional
+  peer `@mediapipe/tasks-vision` at exactly 0.10.35 (later versions report usage to Google); nothing behind the
+  person shows until it is found. Insertable streams in Chrome/Edge, a hidden video on a worker timer elsewhere;
+  Chime-style adaptive segmentation on slow devices.
+- **`setVideoProcessor(processor | null)` and `videoProcessorInUse`**: the camera goes through a `VideoTrackProcessor`
+  (`{ name, process(track), destroy() }`, same size out) before it is sent; the processed picture is also what `local`
+  shows. Swapped in place while live; kept across `switchDevice`, `replaceVideoTrack`, reconnects and quality changes,
+  without restarting the processor. The camera behind it is the one switched, constrained, watched and released.
+  A failing processor sends the camera unprocessed and emits `error`. `camera-fps-low` is not checked while one is on.
+- **A faster first picture** (Agora/TRTC fast first frame). The join asks for the media transports and uses the ones
+  the server made with it (`join {transports}`, controllers from 2026-10-20; older ones are asked as before). The
+  DTLS parameters are sent without waiting for the reply, since mediasoup takes them after its handshake has begun:
+  ICE starts a round trip sooner, and a refusal rejoins. A video's first layers no longer wait before its resume.
+  Up to three round trips fewer before the first publish or the first video.
+
+## 0.8.0
+
+- **Media recovery:** send and receive transports restart ICE after failure or
+  3 s disconnected, before a full rejoin. The entire recovery is bounded to
+  10 s; unsupported servers rejoin. `mediaReconnecting` reports recovery.
+- **Audio-only fallback and `streamFallback` (Agora AUDIO_ONLY, Chime received-audio health, Stream Video's
+  paused tracks).** When the received sound breaks up badly (25 % of the audio lost over ~15 s), remote cameras are
+  paused at the server so the audio keeps going (screen shares stay), and come back after 20 s of recovery (longer
+  after a relapse). The media server also reports videos it cut for this viewer's bandwidth. Both arrive as
+  `streamFallback {producerId, identity, source, fallback, reason: 'bandwidth' | 'audio-protection'}`. On by
+  default: `ClientOptions.audioOnlyFallback: false` or `setAudioOnlyFallback(false)`; `audioOnlyFallbackActive`.
+- **`getDiagnostics()` and `SDK_VERSION`**: a JSON snapshot for a support ticket (session, transport paths, network,
+  published/received, the last 200 notable events); no token, no media.
+- **`play(element)` and `autoplayFailed {element, resume}`** (Agora autoplay-failed, TRTC AUTOPLAY_FAILED): a refused
+  play is reported and retried on the viewer's next tap, click or key press.
+- **`setPlayoutDelay(ms | null)`** (TRTC playoutDelay): `RTCRtpReceiver.jitterBufferTarget` on every remote track,
+  now and later; 300-500 ms for an audience watching a stream, null (default) for conversation.
+- **Faster joins.** The producers already live are subscribed to together on join (as `newProducer` events and the
+  Flutter SDK already were), and `StreamClient.prepareConnection(wsUrl)` warms DNS and TLS before `connect()`.
+- **`StreamClient.preflight(wsUrl, token, {media?, durationMs?})`** (Twilio runPreflight, LiveKit ConnectionCheck):
+  joins a test room, publishes a synthetic tone and picture (or the app's media, cloned) for a few seconds, leaves,
+  and reports steps, media path (UDP/TCP/relayed), round trip, loss, upload and warnings. Never rejects.
+- **`mediaWarning {code, active}`** (Agora exception/recover pairs, LiveKit silence and system-mute handling):
+  `mic-silent`, `camera-fps-low` (under 15 fps when 24+ asked; usually low light), `camera-interrupted`,
+  `mic-interrupted`; each once when it starts and once when it clears; checked every 2 s while listened to.
+- **`audioLevels` and `activeSpeakers`** (Agora volume-indicator, TRTC AUDIO_VOLUME, Amazon Chime's active speaker
+  ranking): levels every ~250 ms while listened to (remote audio from `getSynchronizationSources`, the own microphone
+  from its media-source stats; `speaking` above -35 dBov, `SPEAKING_LEVEL`), and the speaker order when it changes
+  (microphones only). Nothing is read while nobody listens.
+- **Cameras and microphones.** `listDevices(kind?)`, `switchDevice('videoinput' | 'audioinput', deviceId)` (same
+  constraints, the replaced device released, a reconnect keeps the choice; refuses a canvas/processed video), and
+  `devicesChanged {devices, added, removed}` while listened to. A published device that stops from outside raises
+  `localTrackEnded {kind, deviceId, recovered}`: a microphone comes back from the system default, a camera is
+  switched off and never swapped on its own (Amazon Chime's and LiveKit's behaviour).
+- **Publisher-side audio-only fallback (opt-in)**: `setPublishFallback(true)` pauses the camera while the server reports
+  this client's own sound breaking up (Agora's setLocalPublishFallbackOption), and resumes it after recovery;
+  `publishFallback {fallback}`. A camera switched off by the user or a moderator meanwhile stays off.
+- **Socket liveness:** ping after 15 s idle, reconnect after 8 s without any
+  response (checked every 5 s). Browser `online` wakes a pending backoff.
+  Late restart/setup results cannot change a newer session; close cancels watches.
+
+Performance on busy devices (owner's demo test, 2026-10-19: three cameras with effects and reactions hung phones
+and made laptops stutter). Ideas from LiveKit (adaptive stream, dynacast, CpuConstrained), Tencent TRTC (small
+stream, receiveWhenViewVisible) and Agora (automatic low stream). Every 0.7 call works as before.
+
+- **Adaptive stream: `attachVideo(producerId, element)`.** A remote video is received at the size its element is
+  shown (smallest layer that fills it sharply, devicePixelRatio up to 2) and paused at the server while the element
+  cannot be seen (off screen, `display: none`, background tab). Bigger at once, smaller after ~1 s, paused after
+  1.5 s hidden. `setReceiveQuality`/`setPreferredLayers` stay a ceiling. Returns `detach()`. A server without
+  `pauseConsumer` gets the smallest layer instead of a pause.
+- **Dynacast.** The server's new `producerLayers` event says which camera layers any viewer needs; the SDK
+  switches the unused encodings off (the smallest stays on). Nothing to call. Older servers never send it.
+- **`cpuConstrained` event and `prioritizePerformance(on = true)`.** `cpuConstrained` reports when the camera is
+  CPU-limited (held ~4 s) and when that clears; `prioritizePerformance()` stops encoding the camera's largest
+  layer until `prioritizePerformance(false)`, kept across publishes and reconnects. `performancePrioritized`.
+- New wire calls (additive): `pauseConsumer`; new wire event: `producerLayers`.
+
 ## 0.7.3
 
 Bug fixes found by checking this SDK for the bug classes a frontend tester

package/README.md

@@ -122,6 +122,34 @@ modified: the client keeps one of its own (`client.local`, announced by
 track you replaced is yours again: it is not stopped, and if pausing the camera
 had switched it off, it is switched back on.
 
+### Cameras and microphones (0.8.0)
+
+```ts
+const cameras = await client.listDevices('videoinput');      // also 'audioinput', 'audiooutput'
+await client.switchDevice('videoinput', cameras[1].deviceId);
+await client.switchDevice('audioinput', headsetId);
+
+client.on('devicesChanged', ({ added, removed }) => refreshDevicePicker());
+client.on('localTrackEnded', ({ kind, recovered }) => {
+  if (kind === 'video') showCameraPicker();   // the camera was switched off: offer another, then setCamEnabled(true)
+});
+```
+
+`switchDevice` captures the chosen device with the constraints `publish` was
+given (size, frame rate, echo cancellation), swaps it in (viewers see no pause
+for a camera of the same size) and stops the one it replaces. A reconnect
+captures the chosen device again. If you publish a canvas or another processed
+track, capture the new camera yourself and pass your track to
+`replaceVideoTrack`; `switchDevice` refuses rather than replace your picture
+with the raw camera.
+
+When a published device stops from outside (unplugged, a Bluetooth headset
+switched off, the permission revoked), as Amazon Chime and LiveKit do: the
+**microphone** is captured again from the system default and keeps publishing
+(`recovered: true`), or is switched off if there is none; the **camera** is
+switched off, and never swapped for another camera on its own. Labels in
+`listDevices` are empty until the page has camera or microphone access.
+
 ### Video quality
 
 Choose 1080p, 720p or 480p when publishing, or change it while live:
@@ -217,6 +245,102 @@ off only for headphones or line-in music, or listeners hear their own echo.
 Music (`publishMusic`) is a separate 128 kbps stereo track and is not affected:
 it is HD and free whatever the microphone's quality (see [Music](#music)).
 
+### Noise suppression (0.9.0)
+
+AI noise suppression for the microphone, as Agora's AI Denoiser, LiveKit's
+Krisp filter and ZEGOCLOUD's AI noise suppression do: RNNoise, a small neural
+network, runs in the browser's audio thread and takes fans, traffic, air
+conditioning and room rumble out of the voice. It is a separate import, so apps
+that do not use it never download it:
+
+```ts
+import { createNoiseSuppressor } from '@voltkit-rtc/client/noise-suppression';
+
+await client.setAudioProcessor(createNoiseSuppressor());   // before or after publish
+client.audioProcessorInUse;                                 // the processor, or null
+await client.setAudioProcessor(null);                       // send the microphone as it is again
+```
+
+- It applies to the microphone published now (swapped in place: listeners hear
+  no gap) and to every one after it: `switchDevice('audioinput', …)`, a
+  microphone recovered after unplugging, a reconnect, `setAudioQuality`.
+  Music and screen-share audio are not processed.
+- The worklet (64 kB) and the model (150 kB) load the first time it runs, from
+  jsDelivr at the exact version this SDK was tested with. Under a strict
+  Content-Security-Policy, copy `rnnoise/workletProcessor.js`, `rnnoise.wasm`
+  and `rnnoise_simd.wasm` from `node_modules/@sapphi-red/web-noise-suppressor/dist`
+  to your site and pass their URLs:
+  `createNoiseSuppressor({ workletUrl, wasmUrl, simdWasmUrl })`.
+- If it cannot start (the files are blocked, an old browser), the microphone
+  still goes out, unprocessed, and `error` says why. The next microphone tries
+  again.
+- Turn it on from a click (a toggle, the join button): browsers start audio
+  processing only after the page has been touched, and the SDK resumes it on
+  the first tap or key press.
+- `mic-silent` is not reported while a processor is on: a quiet room is
+  near-silence after denoising. Your own level in `audioLevels` is what
+  listeners hear.
+- Measured in Chromium: rumble-like noise (fans, traffic, AC) went down by more
+  than 60 dB; steady white hiss, which it was not trained on, by 7-12 dB. Keep
+  the browser's own `echoCancellation`; its `noiseSuppression` can stay on too.
+
+Your own processor works the same way: anything with `name`,
+`process(track) → Promise<MediaStreamTrack>` and `destroy()`
+(`AudioTrackProcessor`), such as a licensed Krisp build.
+
+### Virtual background and beauty (0.9.0)
+
+A blurred or replaced background and beauty for the camera, as LiveKit's track
+processors, Agora's and ZEGOCLOUD's virtual background and beauty, and TRTC's
+video effects do: in the browser, on the GPU. A separate import, so apps that do
+not use it never download it:
+
+```ts
+import { createVideoEffects, isVideoEffectsSupported } from '@voltkit-rtc/client/video-effects';
+
+if (isVideoEffectsSupported()) {
+  const effects = createVideoEffects({ background: 'blur', beauty: true });
+  await client.setVideoProcessor(effects);                           // before or after publish
+  client.videoProcessorInUse;                                        // the processor, or null
+
+  await effects.update({ background: { image: '/office.jpg' } });   // live: no pause for viewers
+  await effects.update({ blur: 8, beauty: { smooth: 0.7, brighten: 0.4 } });
+  await client.setVideoProcessor(null);                              // the camera as it is again
+}
+```
+
+- `background`: `'none'`, `'blur'` (`blur` 1-10, default 5) or `{ image }` (a
+  URL, Blob, image, canvas or ImageBitmap), fitted like `object-fit: cover`.
+- `beauty`: `true`, or `smooth`, `brighten`, `redness`, `sharpen` (0-1;
+  defaults 0.5, 0.3, 0.1, 0.3). Skin is smoothed with an edge-preserving filter
+  (eyes, lips and teeth stay sharp) and lightly brightened.
+- What you see in `local` is what viewers get. Switching cameras
+  (`switchDevice`), `replaceVideoTrack`, reconnects and quality changes keep the
+  effects without restarting them. A paused camera costs no processing.
+- **Background effects need MediaPipe**: `npm install @mediapipe/tasks-vision@0.10.35`
+  (exactly that version: later ones report usage to Google). Its wasm (3 MB
+  compressed) and the person model (250 kB) load the first time a background is
+  used, from jsDelivr and Google. To serve them yourself (a strict CSP), copy
+  `node_modules/@mediapipe/tasks-vision/wasm` to your site and pass `wasmPath`,
+  and `modelUrl` for the model. Beauty alone needs none of this.
+- Nothing behind you shows while the model loads: the background is fully
+  blurred or replaced until you are found. If the effects cannot start, the
+  camera goes out as it is and `error` says why, so watch `error` when privacy
+  matters.
+- Webcams (wide pictures) use MediaPipe's landscape model and phones (tall
+  pictures) the square one, which is what each traces best.
+- A slow device finds you only every few frames and reuses the outline (the
+  picture keeps the camera's frame rate). `onOverload` tells you when even that
+  is not enough: turn the background off or lower the camera's size.
+- Chrome and Edge process each frame as it arrives and keep going in a
+  background tab. Firefox and Safari draw from a hidden video on a timer, which
+  a background tab can slow down.
+
+Your own processor works the same way: anything with `name`,
+`process(track) → Promise<MediaStreamTrack>` (same size and frame rate) and
+`destroy()` (`VideoTrackProcessor`). The SDK stops a picture it no longer sends;
+stop drawing it once its `readyState` is `'ended'`.
+
 ### Project media settings
 
 A project's owner chooses what it may publish: audio and video on or off, the
@@ -297,11 +421,61 @@ for their own connection; if every viewer is, look at the publisher's
 The [troubleshooting guide](https://voltkit.in/docs/troubleshooting) lists the
 usual causes and what to do about each.
 
+### Smooth on busy devices (0.8.0)
+
+A room with several cameras, effects and reactions can overload a phone or a
+modest laptop: every viewer decoding every co-host's full-size picture, every
+publisher encoding layers nobody watches. Three things keep it smooth (the
+ideas behind LiveKit's adaptive stream and dynacast, Tencent TRTC's small
+stream with `receiveWhenViewVisible`, and Agora's automatic low stream):
+
+**Adaptive stream: each video at the size it is shown.** Attach the element
+that shows a remote video:
+
+```ts
+client.on('track', ({ kind, producerId, track }) => {
+  if (kind !== 'video') return;
+  const video = showVideo(producerId, track);            // your <video> element
+  const detach = client.attachVideo(producerId, video);  // call detach() when the element goes away
+});
+```
+
+The SDK then receives the smallest layer that fills the element sharply (its
+size on screen × devicePixelRatio, at most 2) and pauses the video at the
+server while the element cannot be seen (scrolled away, `display: none`, the tab
+in the background): no bandwidth, no decoding. Bigger is applied at once,
+smaller after about a second, and a hidden video pauses after 1.5 s, so
+scrolling past a tile does not stop it. `setReceiveQuality` and
+`setPreferredLayers` still apply as a ceiling.
+
+**Dynacast: publishers stop encoding layers nobody receives.** Nothing to
+call: the server tells the publishing client which of its camera layers any
+viewer needs, and the SDK switches the others off (the smallest always stays
+on, so a new viewer starts at once). With adaptive stream on the viewers, a host
+shown in small tiles stops spending its CPU on its largest layer.
+
+**When the publisher's device itself is overloaded:**
+
+```ts
+client.on('cpuConstrained', ({ constrained }) => {
+  if (constrained) {
+    void client.prioritizePerformance();  // the camera's largest layer stops being encoded
+    turnOffEffects();                     // and your own savings: effects off, smaller remote videos
+  }
+});
+await client.prioritizePerformance(false); // later, when the user wants full quality back
+```
+
+`cpuConstrained` fires when the browser reports the camera limited by the CPU
+for about 4 seconds (`videoLimited` with `cpu`), and again when that clears.
+`performancePrioritized` tells whether the lighter mode is on.
+
 ### Small tiles
 
 Each viewer gets the biggest layer their connection carries, which a thumbnail
-or a row of co-host tiles does not need. Ask for a smaller one with the
-`producerId` from the `track` event:
+or a row of co-host tiles does not need. `attachVideo` (above) chooses for you;
+to choose yourself, ask for a smaller one with the `producerId` from the `track`
+event:
 
 ```ts
 client.on('track', ({ kind, producerId, track }) => {
@@ -526,6 +700,176 @@ apps. The room closes with `roomClosed { reason: 'quota-exhausted' }` when the
 funds reach zero unless the wallet is recharged first; a recharge sends
 `walletLow { cleared: true }`, which also cancels the `usageLimit` countdown.
 
+### Media and socket recovery (0.8.0)
+
+Recovery runs automatically. A send or receive transport that fails, or stays
+disconnected for 3 seconds, requests new ICE credentials and restarts on the
+same transport. A recovery still unfinished after 10 seconds (including the
+request and browser call), or an older server without `restartIce`, uses the
+normal reconnect and republish path.
+
+`mediaReconnecting` carries `{ transportId, reconnecting }`: `true` when ICE
+recovery starts and `false` when that recovery ends or its transport is discarded.
+Use `connectionStateChanged` to distinguish recovery from a full reconnect.
+
+After 15 seconds without any socket frame the SDK sends `ping`. If no frame
+arrives for 8 seconds after it, the next 5-second check reconnects. Any reply
+counts, including an older server's `unknown-method`. Background tabs may
+delay timers. A browser `online` event wakes a pending reconnect immediately;
+while connected it probes the socket. Closing releases the timers and listener.
+
+### Diagnostics for support (0.8.0)
+
+```ts
+const report = await client.getDiagnostics();
+sendToSupport(JSON.stringify(report));
+```
+
+What Agora's log upload and Twilio's Insights are used for, kept on the device:
+the SDK version (`SDK_VERSION`) and browser, the room, identity, role and
+connection state, how each media transport is connected (UDP, TCP or relayed),
+the round trip and upload now, what is published and received, and the last
+200 notable events (state changes, errors, recoveries, fallbacks, warnings) with
+their times. It holds no token and no media.
+
+### Sound that browsers block until a tap (0.8.0)
+
+```ts
+client.on('track', ({ kind, track }) => {
+  const el = kind === 'audio' ? new Audio() : videoFor(track);
+  el.srcObject = new MediaStream([track]);
+  void client.play(el);                       // instead of el.play()
+});
+client.on('autoplayFailed', ({ resume }) => showTapToUnmute(resume)); // optional
+```
+
+Browsers often refuse sound until the viewer interacts with the page. `play()`
+reports that (`autoplayFailed`, like Agora's and Tencent TRTC's), and plays
+every refused element by itself on the viewer's next tap, click or key press,
+so a viewer who touches anything hears the room.
+
+### Smoother watching on shaky networks (0.8.0)
+
+```ts
+client.setPlayoutDelay(400); // an audience watching a stream: a little later, far fewer freezes
+client.setPlayoutDelay(null); // people talking with each other: as little delay as the browser can keep
+```
+
+Like Tencent TRTC's `playoutDelay`: every remote track is buffered about that
+long before it plays, which rides out a mobile network's jitter. It applies to
+what is received now and later; 0 to 4000 ms. Browsers before Chrome 114 keep
+their own buffer.
+
+### Faster joins (0.8.0)
+
+Everything already live in the room is subscribed to at once when the client
+joins (it used to be one after another), so a room with six cameras shows them
+together. Warm the connection while the user is still on the way in, as
+LiveKit's `prepareConnection` does:
+
+```ts
+StreamClient.prepareConnection(wsUrl); // on the room screen; never rejects
+// ... later
+await client.connect();
+```
+
+From 0.9.0 the first picture comes sooner, with nothing to change in your app:
+the join brings the media transports with it (controllers from 2026-10-20), and
+neither the DTLS handshake nor a video's first layer waits for an
+acknowledgement any more. Each is a round trip saved, 50-150 ms on a mobile
+network, for the first publish and the first video alike.
+
+### Check before going live (0.8.0)
+
+```ts
+// Your backend mints a host token for a test room nobody else is in.
+const { wsUrl, token } = await fetch('/api/precall-token').then((r) => r.json());
+const report = await StreamClient.preflight(wsUrl, token);           // or { media: myStream, durationMs: 5000 }
+if (!report.ok) showProblem(report.steps.find((s) => !s.ok)?.error);
+else if (report.path?.relayed || report.path?.protocol === 'tcp') showHint('A firewall slows your connection');
+else if (report.warnings.includes('mic-silent')) showHint('Your microphone sends no sound');
+```
+
+Like Twilio's `runPreflight` and LiveKit's `ConnectionCheck`, it joins the
+room, publishes for a few seconds (a tone and a moving picture it makes, or your
+`media`, cloned so your tracks keep running) and leaves. The report has each
+step (`join`, `media`) with its time or error, how media travelled (`path`: UDP
+is best; TCP or a relay means a firewall in the way), `rttMs`, `lossPct`,
+`uploadKbps`, `level`, and any camera or microphone `warnings`. It never
+rejects. The session is real and billed like any other for its few seconds.
+
+### Camera and microphone warnings (0.8.0)
+
+```ts
+client.on('mediaWarning', ({ code, active }) => {
+  if (code === 'mic-silent') showBanner(active, 'Your microphone sends no sound: check its mute switch');
+  if (code === 'camera-fps-low') showBanner(active, 'Your camera is dark: add light in front of you');
+});
+```
+
+Like Agora's exception events, each warning comes once when it starts
+(`active: true`) and once when it clears:
+
+| `code` | When |
+| --- | --- |
+| `mic-silent` | the microphone is on but sends exact silence for about 6 s: a mute switch, a broken or wrong device (a quiet room still has a noise floor) |
+| `camera-fps-low` | the camera delivers under 15 fps for about 6 s when 24 or more were asked: usually a dark room, where cameras expose each frame longer. 20 fps is not a warning |
+| `camera-interrupted`, `mic-interrupted` | the system took the device for more than 2 s: a phone call, another app |
+
+Checked every 2 s while listened to.
+
+### Who is speaking (0.8.0)
+
+```ts
+client.on('audioLevels', (levels) => {
+  for (const { identity, speaking } of levels) tileFor(identity).ring(speaking);
+});
+client.on('activeSpeakers', ({ speakers }) => showLarge(speakers[0])); // most active first
+```
+
+`audioLevels` comes about four times a second while listened to, for every
+remote microphone, music and shared audio this client receives and its own
+microphone while on: `{ identity, producerId, source, level, speaking, local }`.
+`level` is WebRTC's audio level (0 silence, 1 full scale); `speaking` is above
+-35 dBov (`SPEAKING_LEVEL`). `activeSpeakers` ranks microphones the way Amazon
+Chime does, so a short cough does not steal the large tile and the person who
+keeps talking does; music never counts. Both are read in the browser from what
+it receives, and nothing is read while nobody listens.
+
+### Poor connections: audio keeps going (0.8.0)
+
+When a viewer's connection cannot carry video, the audio comes first, as in
+Agora's audio-only stream fallback:
+
+- **The media server cuts a video** this viewer's bandwidth cannot carry even
+  at its smallest size, while the sound keeps flowing, and says so.
+- **The SDK pauses remote cameras** when the received sound itself breaks up
+  badly (a quarter of it lost over about 15 seconds), so what bandwidth there is
+  goes to the audio. Screen shares stay. The cameras come back once the
+  connection has recovered for 20 seconds, longer if it falls back again soon.
+
+Either way the app is told, per video, so it can show the person's name instead
+of a frozen picture:
+
+```ts
+client.on('streamFallback', ({ producerId, identity, fallback, reason }) => {
+  tileFor(producerId).showPoorConnection(fallback); // reason: 'bandwidth' or 'audio-protection'
+});
+```
+
+The pause is on by default. Turn it off with `new StreamClient(url, token,
+{ audioOnlyFallback: false })` or `client.setAudioOnlyFallback(false)`, which
+also brings back any camera held at that moment.
+
+**The host's side (opt-in).** `client.setPublishFallback(true)` does the same
+for what this client sends (Agora's `setLocalPublishFallbackOption`): when the
+server reports this client's own sound breaking up on the way in, its camera is
+paused (viewers see it paused) so the upload carries the voice, and resumed once
+it recovers. `publishFallback { fallback }` says when. A camera the user or a
+moderator switches off meanwhile stays off. `audioOnlyFallbackActive`
+says whether it is holding cameras now. The server's bandwidth reports need the
+controller and media node from 2026-10-20; older ones never send them.
+
 ### Network quality
 
 ```ts

package/dist/cjs/adaptive.d.ts

@@ -0,0 +1,104 @@
+/**
+ * Adaptive stream: receive each remote video at the size it is shown, and not
+ * at all while nobody can see it.
+ *
+ * The app attaches the element that shows a video (`attachVideo`). From that
+ * element's size on screen (CSS pixels × devicePixelRatio, at most 2) the
+ * smallest simulcast layer that fills it sharply is chosen; while no attached
+ * element is visible (scrolled away, `display: none`, the tab in the
+ * background) the video is paused at the media server, so it costs no
+ * bandwidth and no decoding. This is what LiveKit calls adaptiveStream and
+ * Tencent TRTC "small stream + receiveWhenViewVisible": the single biggest
+ * saving on a viewer with several tiles, where every tile otherwise decodes
+ * the publisher's full-size layer.
+ *
+ * Pure decisions are exported for tests; the browser parts (observers,
+ * timers) are injected.
+ */
+import type { VideoLayer } from './quality.js';
+/** Where a video is drawn, in CSS pixels, and how it is fitted into that box. */
+export interface ViewBox {
+    width: number;
+    height: number;
+    fit: 'cover' | 'contain';
+}
+/** Never ask for more than twice the CSS size: a 3× phone screen gains nothing visible from it. */
+export declare const MAX_PIXEL_RATIO = 2;
+/** A layer shown at most this much bigger than it is still looks sharp: avoids a whole layer for a few pixels. */
+export declare const MAX_UPSCALE = 1.15;
+/**
+ * The smallest layer that fills `box` sharply, or null when the layers are
+ * not known (an older publisher): then the server chooses, as before.
+ *
+ * With `cover` the picture is scaled until it fills the box (a part is
+ * cropped), with `contain` until it fits. Either way the scale is measured
+ * against each layer's own size, so a portrait layer list works too.
+ */
+export declare function layerForBox(layers: readonly VideoLayer[], box: ViewBox, pixelRatio?: number): number | null;
+/** What one video should get now: paused, or a layer (null = let the server choose). */
+export interface AdaptiveWant {
+    visible: boolean;
+    spatialLayer: number | null;
+}
+/**
+ * The largest view among the visible ones decides; any visible view keeps the video flowing. A view not measured
+ * yet (just attached, before layout) counts as shown at the server's choice, so attaching never pauses a video
+ * for the moment before the browser has sized the element.
+ */
+export declare function wantFor(views: readonly {
+    box: ViewBox;
+    visible: boolean;
+    measured?: boolean;
+}[], layers: readonly VideoLayer[], pixelRatio: number, pageVisible: boolean): AdaptiveWant;
+/** Pause a hidden video only after this long, so scrolling past or a quick tab switch does not stop it. */
+export declare const PAUSE_AFTER_MS = 1500;
+/** Go down a layer only after this long (going up is immediate): a tile shrinking for a moment keeps its picture. */
+export declare const LOWER_AFTER_MS = 1200;
+export interface AdaptiveHost {
+    /** The layers this producer's publisher declared (smallest first or any order); [] when unknown. */
+    layers(producerId: string): readonly VideoLayer[];
+    /** Apply a decision for one video. */
+    apply(producerId: string, want: AdaptiveWant): void;
+    /** For tests: the environment. */
+    pixelRatio?(): number;
+    pageVisible?(): boolean;
+    now?(): number;
+    setTimer?(fn: () => void, ms: number): unknown;
+    clearTimer?(handle: unknown): void;
+    /** For tests: stand-ins for ResizeObserver / IntersectionObserver / document visibility. */
+    observe?(element: Element, onChange: (box: ViewBox | null, visible: boolean | null) => void): () => void;
+    onPageVisibility?(listener: () => void): () => void;
+}
+/**
+ * The videos being shown, by producer id. Decisions are applied through the
+ * host; going up (or becoming visible) is applied at once, going down (or
+ * hiding) after a short delay.
+ */
+export declare class AdaptiveVideos {
+    private readonly host;
+    private readonly entries;
+    private stopPage;
+    constructor(host: AdaptiveHost);
+    /** Show `producerId` in `element`; returns the function that stops. */
+    attach(producerId: string, element: Element): () => void;
+    detach(producerId: string, element: Element): void;
+    /** Whether the app shows this video through attachVideo (its decisions then apply to new consumers too). */
+    has(producerId: string): boolean;
+    /** The decision in force, for a consumer made (or made again) after it: null when the app does not attach it. */
+    current(producerId: string): AdaptiveWant | null;
+    /**
+     * A new consumer was made with `current()` applied (its layers, and paused when hidden): record that as applied,
+     * so nothing is sent twice, and drop a pending change made for the old one.
+     */
+    consumed(producerId: string): void;
+    /** Stop everything (the session closed). */
+    dispose(): void;
+    private decide;
+    private update;
+    private clear;
+    private observe;
+    private watchPage;
+}
+/** The page is visible (not a background tab). */
+export declare function pageIsVisible(): boolean;
+//# sourceMappingURL=adaptive.d.ts.map

package/dist/cjs/adaptive.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"adaptive.d.ts","sourceRoot":"","sources":["../../src/adaptive.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAE/C,iFAAiF;AACjF,MAAM,WAAW,OAAO;IACtB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,OAAO,GAAG,SAAS,CAAC;CAC1B;AAED,mGAAmG;AACnG,eAAO,MAAM,eAAe,IAAI,CAAC;AACjC,kHAAkH;AAClH,eAAO,MAAM,WAAW,OAAO,CAAC;AAEhC;;;;;;;GAOG;AACH,wBAAgB,WAAW,CAAC,MAAM,EAAE,SAAS,UAAU,EAAE,EAAE,GAAG,EAAE,OAAO,EAAE,UAAU,SAAI,GAAG,MAAM,GAAG,IAAI,CAYtG;AAED,wFAAwF;AACxF,MAAM,WAAW,YAAY;IAC3B,OAAO,EAAE,OAAO,CAAC;IACjB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;CAC7B;AAED;;;;GAIG;AACH,wBAAgB,OAAO,CAAC,KAAK,EAAE,SAAS;IAAE,GAAG,EAAE,OAAO,CAAC;IAAC,OAAO,EAAE,OAAO,CAAC;IAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;CAAE,EAAE,EAAE,MAAM,EAAE,SAAS,UAAU,EAAE,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,GAAG,YAAY,CAWvL;AAED,2GAA2G;AAC3G,eAAO,MAAM,cAAc,OAAO,CAAC;AACnC,qHAAqH;AACrH,eAAO,MAAM,cAAc,OAAO,CAAC;AAEnC,MAAM,WAAW,YAAY;IAC3B,oGAAoG;IACpG,MAAM,CAAC,UAAU,EAAE,MAAM,GAAG,SAAS,UAAU,EAAE,CAAC;IAClD,sCAAsC;IACtC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,IAAI,EAAE,YAAY,GAAG,IAAI,CAAC;IACpD,kCAAkC;IAClC,UAAU,CAAC,IAAI,MAAM,CAAC;IACtB,WAAW,CAAC,IAAI,OAAO,CAAC;IACxB,GAAG,CAAC,IAAI,MAAM,CAAC;IACf,QAAQ,CAAC,CAAC,EAAE,EAAE,MAAM,IAAI,EAAE,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC;IAC/C,UAAU,CAAC,CAAC,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACnC,4FAA4F;IAC5F,OAAO,CAAC,CAAC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC,GAAG,EAAE,OAAO,GAAG,IAAI,EAAE,OAAO,EAAE,OAAO,GAAG,IAAI,KAAK,IAAI,GAAG,MAAM,IAAI,CAAC;IACzG,gBAAgB,CAAC,CAAC,QAAQ,EAAE,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC;CACrD;AAkBD;;;;GAIG;AACH,qBAAa,cAAc;IAIb,OAAO,CAAC,QAAQ,CAAC,IAAI;IAHjC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAA4B;IACpD,OAAO,CAAC,QAAQ,CAA6B;gBAEhB,IAAI,EAAE,YAAY;IAE/C,uEAAuE;IACvE,MAAM,CAAC,UAAU,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,MAAM,IAAI;IAsBxD,MAAM,CAAC,UAAU,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,IAAI;IAoBlD,4GAA4G;IAC5G,GAAG,CAAC,UAAU,EAAE,MAAM,GAAG,OAAO;IAIhC,iHAAiH;IACjH,OAAO,CAAC,UAAU,EAAE,MAAM,GAAG,YAAY,GAAG,IAAI;IAKhD;;;OAGG;IACH,QAAQ,CAAC,UAAU,EAAE,MAAM,GAAG,IAAI;IAOlC,4CAA4C;IAC5C,OAAO,IAAI,IAAI;IAUf,OAAO,CAAC,MAAM;IAId,OAAO,CAAC,MAAM;IAiCd,OAAO,CAAC,KAAK;IAKb,OAAO,CAAC,OAAO;IAKf,OAAO,CAAC,SAAS;CAMlB;AAoCD,kDAAkD;AAClD,wBAAgB,aAAa,IAAI,OAAO,CAEvC"}