@voltkit-rtc/client 0.7.2 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +89 -0
- package/README.md +255 -4
- package/dist/cjs/adaptive.d.ts +104 -0
- package/dist/cjs/adaptive.d.ts.map +1 -0
- package/dist/cjs/adaptive.js +263 -0
- package/dist/cjs/adaptive.js.map +1 -0
- package/dist/cjs/devices.d.ts +42 -0
- package/dist/cjs/devices.d.ts.map +1 -0
- package/dist/cjs/devices.js +56 -0
- package/dist/cjs/devices.js.map +1 -0
- package/dist/cjs/diagnostics.d.ts +65 -0
- package/dist/cjs/diagnostics.d.ts.map +1 -0
- package/dist/cjs/diagnostics.js +69 -0
- package/dist/cjs/diagnostics.js.map +1 -0
- package/dist/cjs/fallback.d.ts +81 -0
- package/dist/cjs/fallback.d.ts.map +1 -0
- package/dist/cjs/fallback.js +148 -0
- package/dist/cjs/fallback.js.map +1 -0
- package/dist/cjs/health.d.ts +70 -0
- package/dist/cjs/health.d.ts.map +1 -0
- package/dist/cjs/health.js +116 -0
- package/dist/cjs/health.js.map +1 -0
- package/dist/cjs/index.d.ts +350 -0
- package/dist/cjs/index.d.ts.map +1 -1
- package/dist/cjs/index.js +1201 -36
- package/dist/cjs/index.js.map +1 -1
- package/dist/cjs/link.d.ts +99 -0
- package/dist/cjs/link.d.ts.map +1 -0
- package/dist/cjs/link.js +219 -0
- package/dist/cjs/link.js.map +1 -0
- package/dist/cjs/preflight.d.ts +79 -0
- package/dist/cjs/preflight.d.ts.map +1 -0
- package/dist/cjs/preflight.js +159 -0
- package/dist/cjs/preflight.js.map +1 -0
- package/dist/cjs/recording.d.ts +14 -0
- package/dist/cjs/recording.d.ts.map +1 -1
- package/dist/cjs/recording.js +51 -5
- package/dist/cjs/recording.js.map +1 -1
- package/dist/cjs/speakers.d.ts +34 -0
- package/dist/cjs/speakers.d.ts.map +1 -0
- package/dist/cjs/speakers.js +71 -0
- package/dist/cjs/speakers.js.map +1 -0
- package/dist/esm/adaptive.d.ts +104 -0
- package/dist/esm/adaptive.d.ts.map +1 -0
- package/dist/esm/adaptive.js +256 -0
- package/dist/esm/adaptive.js.map +1 -0
- package/dist/esm/devices.d.ts +42 -0
- package/dist/esm/devices.d.ts.map +1 -0
- package/dist/esm/devices.js +50 -0
- package/dist/esm/devices.js.map +1 -0
- package/dist/esm/diagnostics.d.ts +65 -0
- package/dist/esm/diagnostics.d.ts.map +1 -0
- package/dist/esm/diagnostics.js +65 -0
- package/dist/esm/diagnostics.js.map +1 -0
- package/dist/esm/fallback.d.ts +81 -0
- package/dist/esm/fallback.d.ts.map +1 -0
- package/dist/esm/fallback.js +142 -0
- package/dist/esm/fallback.js.map +1 -0
- package/dist/esm/health.d.ts +70 -0
- package/dist/esm/health.d.ts.map +1 -0
- package/dist/esm/health.js +111 -0
- package/dist/esm/health.js.map +1 -0
- package/dist/esm/index.d.ts +350 -0
- package/dist/esm/index.d.ts.map +1 -1
- package/dist/esm/index.js +1199 -36
- package/dist/esm/index.js.map +1 -1
- package/dist/esm/link.d.ts +99 -0
- package/dist/esm/link.d.ts.map +1 -0
- package/dist/esm/link.js +214 -0
- package/dist/esm/link.js.map +1 -0
- package/dist/esm/preflight.d.ts +79 -0
- package/dist/esm/preflight.d.ts.map +1 -0
- package/dist/esm/preflight.js +153 -0
- package/dist/esm/preflight.js.map +1 -0
- package/dist/esm/recording.d.ts +14 -0
- package/dist/esm/recording.d.ts.map +1 -1
- package/dist/esm/recording.js +51 -5
- package/dist/esm/recording.js.map +1 -1
- package/dist/esm/speakers.d.ts +34 -0
- package/dist/esm/speakers.d.ts.map +1 -0
- package/dist/esm/speakers.js +67 -0
- package/dist/esm/speakers.js.map +1 -0
- package/package.json +1 -1
- package/src/adaptive.ts +295 -0
- package/src/devices.ts +70 -0
- package/src/diagnostics.ts +94 -0
- package/src/fallback.ts +169 -0
- package/src/health.ts +127 -0
- package/src/index.ts +1192 -36
- package/src/link.ts +255 -0
- package/src/preflight.ts +191 -0
- package/src/recording.ts +59 -21
- package/src/speakers.ts +66 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,94 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.8.0
|
|
4
|
+
|
|
5
|
+
- **Media recovery:** send and receive transports restart ICE after failure or
|
|
6
|
+
3 s disconnected, before a full rejoin. The entire recovery is bounded to
|
|
7
|
+
10 s; unsupported servers rejoin. `mediaReconnecting` reports recovery.
|
|
8
|
+
- **Audio-only fallback and `streamFallback` (Agora AUDIO_ONLY, Chime received-audio health, Stream Video's
|
|
9
|
+
paused tracks).** When the received sound breaks up badly (25 % of the audio lost over ~15 s), remote cameras are
|
|
10
|
+
paused at the server so the audio keeps going (screen shares stay), and come back after 20 s of recovery (longer
|
|
11
|
+
after a relapse). The media server also reports videos it cut for this viewer's bandwidth. Both arrive as
|
|
12
|
+
`streamFallback {producerId, identity, source, fallback, reason: 'bandwidth' | 'audio-protection'}`. On by
|
|
13
|
+
default: `ClientOptions.audioOnlyFallback: false` or `setAudioOnlyFallback(false)`; `audioOnlyFallbackActive`.
|
|
14
|
+
- **`getDiagnostics()` and `SDK_VERSION`**: a JSON snapshot for a support ticket (session, transport paths, network,
|
|
15
|
+
published/received, the last 200 notable events); no token, no media.
|
|
16
|
+
- **`play(element)` and `autoplayFailed {element, resume}`** (Agora autoplay-failed, TRTC AUTOPLAY_FAILED): a refused
|
|
17
|
+
play is reported and retried on the viewer's next tap, click or key press.
|
|
18
|
+
- **`setPlayoutDelay(ms | null)`** (TRTC playoutDelay): `RTCRtpReceiver.jitterBufferTarget` on every remote track,
|
|
19
|
+
now and later; 300-500 ms for an audience watching a stream, null (default) for conversation.
|
|
20
|
+
- **Faster joins.** The producers already live are subscribed to together on join (as `newProducer` events and the
|
|
21
|
+
Flutter SDK already were), and `StreamClient.prepareConnection(wsUrl)` warms DNS and TLS before `connect()`.
|
|
22
|
+
- **`StreamClient.preflight(wsUrl, token, {media?, durationMs?})`** (Twilio runPreflight, LiveKit ConnectionCheck):
|
|
23
|
+
joins a test room, publishes a synthetic tone and picture (or the app's media, cloned) for a few seconds, leaves,
|
|
24
|
+
and reports steps, media path (UDP/TCP/relayed), round trip, loss, upload and warnings. Never rejects.
|
|
25
|
+
- **`mediaWarning {code, active}`** (Agora exception/recover pairs, LiveKit silence and system-mute handling):
|
|
26
|
+
`mic-silent`, `camera-fps-low` (under 15 fps when 24+ asked; usually low light), `camera-interrupted`,
|
|
27
|
+
`mic-interrupted`; each once when it starts and once when it clears; checked every 2 s while listened to.
|
|
28
|
+
- **`audioLevels` and `activeSpeakers`** (Agora volume-indicator, TRTC AUDIO_VOLUME, Amazon Chime's active speaker
|
|
29
|
+
ranking): levels every ~250 ms while listened to (remote audio from `getSynchronizationSources`, the own microphone
|
|
30
|
+
from its media-source stats; `speaking` above -35 dBov, `SPEAKING_LEVEL`), and the speaker order when it changes
|
|
31
|
+
(microphones only). Nothing is read while nobody listens.
|
|
32
|
+
- **Cameras and microphones.** `listDevices(kind?)`, `switchDevice('videoinput' | 'audioinput', deviceId)` (same
|
|
33
|
+
constraints, the replaced device released, a reconnect keeps the choice; refuses a canvas/processed video), and
|
|
34
|
+
`devicesChanged {devices, added, removed}` while listened to. A published device that stops from outside raises
|
|
35
|
+
`localTrackEnded {kind, deviceId, recovered}`: a microphone comes back from the system default, a camera is
|
|
36
|
+
switched off and never swapped on its own (Amazon Chime's and LiveKit's behaviour).
|
|
37
|
+
- **Publisher-side audio-only fallback (opt-in)**: `setPublishFallback(true)` pauses the camera while the server reports
|
|
38
|
+
this client's own sound breaking up (Agora's setLocalPublishFallbackOption), and resumes it after recovery;
|
|
39
|
+
`publishFallback {fallback}`. A camera switched off by the user or a moderator meanwhile stays off.
|
|
40
|
+
- **Socket liveness:** ping after 15 s idle, reconnect after 8 s without any
|
|
41
|
+
response (checked every 5 s). Browser `online` wakes a pending backoff.
|
|
42
|
+
Late restart/setup results cannot change a newer session; close cancels watches.
|
|
43
|
+
|
|
44
|
+
Performance on busy devices (owner's demo test, 2026-10-19: three cameras with effects and reactions hung phones
|
|
45
|
+
and made laptops stutter). Ideas from LiveKit (adaptive stream, dynacast, CpuConstrained), Tencent TRTC (small
|
|
46
|
+
stream, receiveWhenViewVisible) and Agora (automatic low stream). Every 0.7 call works as before.
|
|
47
|
+
|
|
48
|
+
- **Adaptive stream: `attachVideo(producerId, element)`.** A remote video is received at the size its element is
|
|
49
|
+
shown (smallest layer that fills it sharply, devicePixelRatio up to 2) and paused at the server while the element
|
|
50
|
+
cannot be seen (off screen, `display: none`, background tab). Bigger at once, smaller after ~1 s, paused after
|
|
51
|
+
1.5 s hidden. `setReceiveQuality`/`setPreferredLayers` stay a ceiling. Returns `detach()`. A server without
|
|
52
|
+
`pauseConsumer` gets the smallest layer instead of a pause.
|
|
53
|
+
- **Dynacast.** The server's new `producerLayers` event says which camera layers any viewer needs; the SDK
|
|
54
|
+
switches the unused encodings off (the smallest stays on). Nothing to call. Older servers never send it.
|
|
55
|
+
- **`cpuConstrained` event and `prioritizePerformance(on = true)`.** `cpuConstrained` reports when the camera is
|
|
56
|
+
CPU-limited (held ~4 s) and when that clears; `prioritizePerformance()` stops encoding the camera's largest
|
|
57
|
+
layer until `prioritizePerformance(false)`, kept across publishes and reconnects. `performancePrioritized`.
|
|
58
|
+
- New wire calls (additive): `pauseConsumer`; new wire event: `producerLayers`.
|
|
59
|
+
|
|
60
|
+
## 0.7.3
|
|
61
|
+
|
|
62
|
+
Bug fixes found by checking this SDK for the bug classes a frontend tester
|
|
63
|
+
reported in the Flutter SDK 0.7.2 (2026-10-19, V072-01…V072-08). Each has a
|
|
64
|
+
test that failed before its fix (`test/parity-v072.test.mjs`).
|
|
65
|
+
|
|
66
|
+
- **V072-01: a second `publish()` no longer orphans the first stream.**
|
|
67
|
+
Publishing a microphone stream and then a camera stream replaced `local`
|
|
68
|
+
with the second: `unpublish()`/`close()` left the first stream's tracks
|
|
69
|
+
capturing and a reconnect published only the camera. `local` is now one
|
|
70
|
+
stream with every live track (the newer track first for a kind published
|
|
71
|
+
twice; `localStreamChanged` fires), all are stopped with it, a reconnect
|
|
72
|
+
publishes both, and the constraints for capturing again cover both kinds.
|
|
73
|
+
- **V072-06: an unmute answered late cannot undo a newer forced mute.**
|
|
74
|
+
A `forceMuted` arriving while `setMicEnabled(true)`/`setCamEnabled(true)`
|
|
75
|
+
waited was reversed by the older reply. That switch now leaves it off,
|
|
76
|
+
tells the server so, and rejects with `force-muted`. Microphone and camera
|
|
77
|
+
are tracked separately.
|
|
78
|
+
- **V072-03: a switch outliving `unpublish()` or `close()` resolves** instead
|
|
79
|
+
of rejecting with `not-connected`.
|
|
80
|
+
- **V072-05: opening the socket has a deadline.** A socket that does not open
|
|
81
|
+
within 15 s fails `connect()` (or the reconnect attempt) with `timeout` and
|
|
82
|
+
is closed; the join already had its 15 s deadline. A reconnect attempt whose
|
|
83
|
+
join failed lets its socket go at once instead of keeping it until the next
|
|
84
|
+
attempt.
|
|
85
|
+
- **V072-04: finalising a recording asks again when the answer is unclear.**
|
|
86
|
+
A lost, unreadable, 5xx or 408/409/425/429 answer to `/complete` was a
|
|
87
|
+
plain failure; it is asked again twice (the service's completion is
|
|
88
|
+
idempotent), and if still unclear rejects saying the recording may already
|
|
89
|
+
be saved. An accepted reply without a recording rejects saying it
|
|
90
|
+
completed. Neither aborts; a readable refusal is final.
|
|
91
|
+
|
|
3
92
|
## 0.7.2
|
|
4
93
|
|
|
5
94
|
Bug fixes found by checking this SDK for the bug classes a frontend tester
|
package/README.md
CHANGED
|
@@ -79,6 +79,14 @@ client.publishing; client.micEnabled; client.camEnabled; client.local; // curr
|
|
|
79
79
|
A `viewer` cannot publish. That is enforced at the media server, not in this
|
|
80
80
|
package — a tampered client gets an error, not a stream.
|
|
81
81
|
|
|
82
|
+
Calling `publish()` again adds to what is published (a microphone stream
|
|
83
|
+
first, a camera stream later): `local` becomes one stream carrying both
|
|
84
|
+
(`localStreamChanged` says so), and `unpublish()`, `close()` and a reconnect
|
|
85
|
+
handle every track. A switch still waiting when `unpublish()` or `close()` runs
|
|
86
|
+
resolves without doing more. `setMicEnabled(true)` / `setCamEnabled(true)`
|
|
87
|
+
rejects with `force-muted` while a moderator holds that device off, including
|
|
88
|
+
when the moderator decided while the call was waiting; it then stays off.
|
|
89
|
+
|
|
82
90
|
`publish(media, constraints, options)` takes these options. The defaults suit
|
|
83
91
|
phones and computers alike; they are kept for a reconnect's republish.
|
|
84
92
|
|
|
@@ -114,6 +122,34 @@ modified: the client keeps one of its own (`client.local`, announced by
|
|
|
114
122
|
track you replaced is yours again: it is not stopped, and if pausing the camera
|
|
115
123
|
had switched it off, it is switched back on.
|
|
116
124
|
|
|
125
|
+
### Cameras and microphones (0.8.0)
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
const cameras = await client.listDevices('videoinput'); // also 'audioinput', 'audiooutput'
|
|
129
|
+
await client.switchDevice('videoinput', cameras[1].deviceId);
|
|
130
|
+
await client.switchDevice('audioinput', headsetId);
|
|
131
|
+
|
|
132
|
+
client.on('devicesChanged', ({ added, removed }) => refreshDevicePicker());
|
|
133
|
+
client.on('localTrackEnded', ({ kind, recovered }) => {
|
|
134
|
+
if (kind === 'video') showCameraPicker(); // the camera was switched off: offer another, then setCamEnabled(true)
|
|
135
|
+
});
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
`switchDevice` captures the chosen device with the constraints `publish` was
|
|
139
|
+
given (size, frame rate, echo cancellation), swaps it in (viewers see no pause
|
|
140
|
+
for a camera of the same size) and stops the one it replaces. A reconnect
|
|
141
|
+
captures the chosen device again. If you publish a canvas or another processed
|
|
142
|
+
track, capture the new camera yourself and pass your track to
|
|
143
|
+
`replaceVideoTrack`; `switchDevice` refuses rather than replace your picture
|
|
144
|
+
with the raw camera.
|
|
145
|
+
|
|
146
|
+
When a published device stops from outside (unplugged, a Bluetooth headset
|
|
147
|
+
switched off, the permission revoked), as Amazon Chime and LiveKit do: the
|
|
148
|
+
**microphone** is captured again from the system default and keeps publishing
|
|
149
|
+
(`recovered: true`), or is switched off if there is none; the **camera** is
|
|
150
|
+
switched off, and never swapped for another camera on its own. Labels in
|
|
151
|
+
`listDevices` are empty until the page has camera or microphone access.
|
|
152
|
+
|
|
117
153
|
### Video quality
|
|
118
154
|
|
|
119
155
|
Choose 1080p, 720p or 480p when publishing, or change it while live:
|
|
@@ -289,11 +325,61 @@ for their own connection; if every viewer is, look at the publisher's
|
|
|
289
325
|
The [troubleshooting guide](https://voltkit.in/docs/troubleshooting) lists the
|
|
290
326
|
usual causes and what to do about each.
|
|
291
327
|
|
|
328
|
+
### Smooth on busy devices (0.8.0)
|
|
329
|
+
|
|
330
|
+
A room with several cameras, effects and reactions can overload a phone or a
|
|
331
|
+
modest laptop: every viewer decoding every co-host's full-size picture, every
|
|
332
|
+
publisher encoding layers nobody watches. Three things keep it smooth (the
|
|
333
|
+
ideas behind LiveKit's adaptive stream and dynacast, Tencent TRTC's small
|
|
334
|
+
stream with `receiveWhenViewVisible`, and Agora's automatic low stream):
|
|
335
|
+
|
|
336
|
+
**Adaptive stream: each video at the size it is shown.** Attach the element
|
|
337
|
+
that shows a remote video:
|
|
338
|
+
|
|
339
|
+
```ts
|
|
340
|
+
client.on('track', ({ kind, producerId, track }) => {
|
|
341
|
+
if (kind !== 'video') return;
|
|
342
|
+
const video = showVideo(producerId, track); // your <video> element
|
|
343
|
+
const detach = client.attachVideo(producerId, video); // call detach() when the element goes away
|
|
344
|
+
});
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The SDK then receives the smallest layer that fills the element sharply (its
|
|
348
|
+
size on screen × devicePixelRatio, at most 2) and pauses the video at the
|
|
349
|
+
server while the element cannot be seen (scrolled away, `display: none`, the tab
|
|
350
|
+
in the background): no bandwidth, no decoding. Bigger is applied at once,
|
|
351
|
+
smaller after about a second, and a hidden video pauses after 1.5 s, so
|
|
352
|
+
scrolling past a tile does not stop it. `setReceiveQuality` and
|
|
353
|
+
`setPreferredLayers` still apply as a ceiling.
|
|
354
|
+
|
|
355
|
+
**Dynacast: publishers stop encoding layers nobody receives.** Nothing to
|
|
356
|
+
call: the server tells the publishing client which of its camera layers any
|
|
357
|
+
viewer needs, and the SDK switches the others off (the smallest always stays
|
|
358
|
+
on, so a new viewer starts at once). With adaptive stream on the viewers, a host
|
|
359
|
+
shown in small tiles stops spending its CPU on its largest layer.
|
|
360
|
+
|
|
361
|
+
**When the publisher's device itself is overloaded:**
|
|
362
|
+
|
|
363
|
+
```ts
|
|
364
|
+
client.on('cpuConstrained', ({ constrained }) => {
|
|
365
|
+
if (constrained) {
|
|
366
|
+
void client.prioritizePerformance(); // the camera's largest layer stops being encoded
|
|
367
|
+
turnOffEffects(); // and your own savings: effects off, smaller remote videos
|
|
368
|
+
}
|
|
369
|
+
});
|
|
370
|
+
await client.prioritizePerformance(false); // later, when the user wants full quality back
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
`cpuConstrained` fires when the browser reports the camera limited by the CPU
|
|
374
|
+
for about 4 seconds (`videoLimited` with `cpu`), and again when that clears.
|
|
375
|
+
`performancePrioritized` tells whether the lighter mode is on.
|
|
376
|
+
|
|
292
377
|
### Small tiles
|
|
293
378
|
|
|
294
379
|
Each viewer gets the biggest layer their connection carries, which a thumbnail
|
|
295
|
-
or a row of co-host tiles does not need.
|
|
296
|
-
`producerId` from the `track`
|
|
380
|
+
or a row of co-host tiles does not need. `attachVideo` (above) chooses for you;
|
|
381
|
+
to choose yourself, ask for a smaller one with the `producerId` from the `track`
|
|
382
|
+
event:
|
|
297
383
|
|
|
298
384
|
```ts
|
|
299
385
|
client.on('track', ({ kind, producerId, track }) => {
|
|
@@ -518,6 +604,170 @@ apps. The room closes with `roomClosed { reason: 'quota-exhausted' }` when the
|
|
|
518
604
|
funds reach zero unless the wallet is recharged first; a recharge sends
|
|
519
605
|
`walletLow { cleared: true }`, which also cancels the `usageLimit` countdown.
|
|
520
606
|
|
|
607
|
+
### Media and socket recovery (0.8.0)
|
|
608
|
+
|
|
609
|
+
Recovery runs automatically. A send or receive transport that fails, or stays
|
|
610
|
+
disconnected for 3 seconds, requests new ICE credentials and restarts on the
|
|
611
|
+
same transport. A recovery still unfinished after 10 seconds (including the
|
|
612
|
+
request and browser call), or an older server without `restartIce`, uses the
|
|
613
|
+
normal reconnect and republish path.
|
|
614
|
+
|
|
615
|
+
`mediaReconnecting` carries `{ transportId, reconnecting }`: `true` when ICE
|
|
616
|
+
recovery starts and `false` when that recovery ends or its transport is discarded.
|
|
617
|
+
Use `connectionStateChanged` to distinguish recovery from a full reconnect.
|
|
618
|
+
|
|
619
|
+
After 15 seconds without any socket frame the SDK sends `ping`. If no frame
|
|
620
|
+
arrives for 8 seconds after it, the next 5-second check reconnects. Any reply
|
|
621
|
+
counts, including an older server's `unknown-method`. Background tabs may
|
|
622
|
+
delay timers. A browser `online` event wakes a pending reconnect immediately;
|
|
623
|
+
while connected it probes the socket. Closing releases the timers and listener.
|
|
624
|
+
|
|
625
|
+
### Diagnostics for support (0.8.0)
|
|
626
|
+
|
|
627
|
+
```ts
|
|
628
|
+
const report = await client.getDiagnostics();
|
|
629
|
+
sendToSupport(JSON.stringify(report));
|
|
630
|
+
```
|
|
631
|
+
|
|
632
|
+
What Agora's log upload and Twilio's Insights are used for, kept on the device:
|
|
633
|
+
the SDK version (`SDK_VERSION`) and browser, the room, identity, role and
|
|
634
|
+
connection state, how each media transport is connected (UDP, TCP or relayed),
|
|
635
|
+
the round trip and upload now, what is published and received, and the last
|
|
636
|
+
200 notable events (state changes, errors, recoveries, fallbacks, warnings) with
|
|
637
|
+
their times. It holds no token and no media.
|
|
638
|
+
|
|
639
|
+
### Sound that browsers block until a tap (0.8.0)
|
|
640
|
+
|
|
641
|
+
```ts
|
|
642
|
+
client.on('track', ({ kind, track }) => {
|
|
643
|
+
const el = kind === 'audio' ? new Audio() : videoFor(track);
|
|
644
|
+
el.srcObject = new MediaStream([track]);
|
|
645
|
+
void client.play(el); // instead of el.play()
|
|
646
|
+
});
|
|
647
|
+
client.on('autoplayFailed', ({ resume }) => showTapToUnmute(resume)); // optional
|
|
648
|
+
```
|
|
649
|
+
|
|
650
|
+
Browsers often refuse sound until the viewer interacts with the page. `play()`
|
|
651
|
+
reports that (`autoplayFailed`, like Agora's and Tencent TRTC's), and plays
|
|
652
|
+
every refused element by itself on the viewer's next tap, click or key press,
|
|
653
|
+
so a viewer who touches anything hears the room.
|
|
654
|
+
|
|
655
|
+
### Smoother watching on shaky networks (0.8.0)
|
|
656
|
+
|
|
657
|
+
```ts
|
|
658
|
+
client.setPlayoutDelay(400); // an audience watching a stream: a little later, far fewer freezes
|
|
659
|
+
client.setPlayoutDelay(null); // people talking with each other: as little delay as the browser can keep
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
Like Tencent TRTC's `playoutDelay`: every remote track is buffered about that
|
|
663
|
+
long before it plays, which rides out a mobile network's jitter. It applies to
|
|
664
|
+
what is received now and later; 0 to 4000 ms. Browsers before Chrome 114 keep
|
|
665
|
+
their own buffer.
|
|
666
|
+
|
|
667
|
+
### Faster joins (0.8.0)
|
|
668
|
+
|
|
669
|
+
Everything already live in the room is subscribed to at once when the client
|
|
670
|
+
joins (it used to be one after another), so a room with six cameras shows them
|
|
671
|
+
together. Warm the connection while the user is still on the way in, as
|
|
672
|
+
LiveKit's `prepareConnection` does:
|
|
673
|
+
|
|
674
|
+
```ts
|
|
675
|
+
StreamClient.prepareConnection(wsUrl); // on the room screen; never rejects
|
|
676
|
+
// ... later
|
|
677
|
+
await client.connect();
|
|
678
|
+
```
|
|
679
|
+
|
|
680
|
+
### Check before going live (0.8.0)
|
|
681
|
+
|
|
682
|
+
```ts
|
|
683
|
+
// Your backend mints a host token for a test room nobody else is in.
|
|
684
|
+
const { wsUrl, token } = await fetch('/api/precall-token').then((r) => r.json());
|
|
685
|
+
const report = await StreamClient.preflight(wsUrl, token); // or { media: myStream, durationMs: 5000 }
|
|
686
|
+
if (!report.ok) showProblem(report.steps.find((s) => !s.ok)?.error);
|
|
687
|
+
else if (report.path?.relayed || report.path?.protocol === 'tcp') showHint('A firewall slows your connection');
|
|
688
|
+
else if (report.warnings.includes('mic-silent')) showHint('Your microphone sends no sound');
|
|
689
|
+
```
|
|
690
|
+
|
|
691
|
+
Like Twilio's `runPreflight` and LiveKit's `ConnectionCheck`, it joins the
|
|
692
|
+
room, publishes for a few seconds (a tone and a moving picture it makes, or your
|
|
693
|
+
`media`, cloned so your tracks keep running) and leaves. The report has each
|
|
694
|
+
step (`join`, `media`) with its time or error, how media travelled (`path`: UDP
|
|
695
|
+
is best; TCP or a relay means a firewall in the way), `rttMs`, `lossPct`,
|
|
696
|
+
`uploadKbps`, `level`, and any camera or microphone `warnings`. It never
|
|
697
|
+
rejects. The session is real and billed like any other for its few seconds.
|
|
698
|
+
|
|
699
|
+
### Camera and microphone warnings (0.8.0)
|
|
700
|
+
|
|
701
|
+
```ts
|
|
702
|
+
client.on('mediaWarning', ({ code, active }) => {
|
|
703
|
+
if (code === 'mic-silent') showBanner(active, 'Your microphone sends no sound: check its mute switch');
|
|
704
|
+
if (code === 'camera-fps-low') showBanner(active, 'Your camera is dark: add light in front of you');
|
|
705
|
+
});
|
|
706
|
+
```
|
|
707
|
+
|
|
708
|
+
Like Agora's exception events, each warning comes once when it starts
|
|
709
|
+
(`active: true`) and once when it clears:
|
|
710
|
+
|
|
711
|
+
| `code` | When |
|
|
712
|
+
| --- | --- |
|
|
713
|
+
| `mic-silent` | the microphone is on but sends exact silence for about 6 s: a mute switch, a broken or wrong device (a quiet room still has a noise floor) |
|
|
714
|
+
| `camera-fps-low` | the camera delivers under 15 fps for about 6 s when 24 or more were asked: usually a dark room, where cameras expose each frame longer. 20 fps is not a warning |
|
|
715
|
+
| `camera-interrupted`, `mic-interrupted` | the system took the device for more than 2 s: a phone call, another app |
|
|
716
|
+
|
|
717
|
+
Checked every 2 s while listened to.
|
|
718
|
+
|
|
719
|
+
### Who is speaking (0.8.0)
|
|
720
|
+
|
|
721
|
+
```ts
|
|
722
|
+
client.on('audioLevels', (levels) => {
|
|
723
|
+
for (const { identity, speaking } of levels) tileFor(identity).ring(speaking);
|
|
724
|
+
});
|
|
725
|
+
client.on('activeSpeakers', ({ speakers }) => showLarge(speakers[0])); // most active first
|
|
726
|
+
```
|
|
727
|
+
|
|
728
|
+
`audioLevels` comes about four times a second while listened to, for every
|
|
729
|
+
remote microphone, music and shared audio this client receives and its own
|
|
730
|
+
microphone while on: `{ identity, producerId, source, level, speaking, local }`.
|
|
731
|
+
`level` is WebRTC's audio level (0 silence, 1 full scale); `speaking` is above
|
|
732
|
+
-35 dBov (`SPEAKING_LEVEL`). `activeSpeakers` ranks microphones the way Amazon
|
|
733
|
+
Chime does, so a short cough does not steal the large tile and the person who
|
|
734
|
+
keeps talking does; music never counts. Both are read in the browser from what
|
|
735
|
+
it receives, and nothing is read while nobody listens.
|
|
736
|
+
|
|
737
|
+
### Poor connections: audio keeps going (0.8.0)
|
|
738
|
+
|
|
739
|
+
When a viewer's connection cannot carry video, the audio comes first, as in
|
|
740
|
+
Agora's audio-only stream fallback:
|
|
741
|
+
|
|
742
|
+
- **The media server cuts a video** this viewer's bandwidth cannot carry even
|
|
743
|
+
at its smallest size, while the sound keeps flowing, and says so.
|
|
744
|
+
- **The SDK pauses remote cameras** when the received sound itself breaks up
|
|
745
|
+
badly (a quarter of it lost over about 15 seconds), so what bandwidth there is
|
|
746
|
+
goes to the audio. Screen shares stay. The cameras come back once the
|
|
747
|
+
connection has recovered for 20 seconds, longer if it falls back again soon.
|
|
748
|
+
|
|
749
|
+
Either way the app is told, per video, so it can show the person's name instead
|
|
750
|
+
of a frozen picture:
|
|
751
|
+
|
|
752
|
+
```ts
|
|
753
|
+
client.on('streamFallback', ({ producerId, identity, fallback, reason }) => {
|
|
754
|
+
tileFor(producerId).showPoorConnection(fallback); // reason: 'bandwidth' or 'audio-protection'
|
|
755
|
+
});
|
|
756
|
+
```
|
|
757
|
+
|
|
758
|
+
The pause is on by default. Turn it off with `new StreamClient(url, token,
|
|
759
|
+
{ audioOnlyFallback: false })` or `client.setAudioOnlyFallback(false)`, which
|
|
760
|
+
also brings back any camera held at that moment.
|
|
761
|
+
|
|
762
|
+
**The host's side (opt-in).** `client.setPublishFallback(true)` does the same
|
|
763
|
+
for what this client sends (Agora's `setLocalPublishFallbackOption`): when the
|
|
764
|
+
server reports this client's own sound breaking up on the way in, its camera is
|
|
765
|
+
paused (viewers see it paused) so the upload carries the voice, and resumed once
|
|
766
|
+
it recovers. `publishFallback { fallback }` says when. A camera the user or a
|
|
767
|
+
moderator switches off meanwhile stays off. `audioOnlyFallbackActive`
|
|
768
|
+
says whether it is holding cameras now. The server's bandwidth reports need the
|
|
769
|
+
controller and media node from 2026-10-20; older ones never send them.
|
|
770
|
+
|
|
521
771
|
### Network quality
|
|
522
772
|
|
|
523
773
|
```ts
|
|
@@ -564,8 +814,9 @@ dropped and the slot is kept for them for about a minute. A refused
|
|
|
564
814
|
|
|
565
815
|
### Reconnection
|
|
566
816
|
|
|
567
|
-
Handled for you. A dropped socket is retried with backoff
|
|
568
|
-
|
|
817
|
+
Handled for you. A dropped socket is retried with backoff (each attempt's
|
|
818
|
+
socket must open, and its join be answered, within 15 s; a stalled attempt is
|
|
819
|
+
closed and the next one follows), the session is rejoined, and if you were publishing, your media is published again (with the
|
|
569
820
|
same quality, frame rate and audio quality). `closed` with `gave-up` means it
|
|
570
821
|
stopped trying: after eight attempts, or at once for a join refused with a code
|
|
571
822
|
retrying cannot fix (`isTerminalError`: the room ended, the token is wrong, you
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Adaptive stream: receive each remote video at the size it is shown, and not
|
|
3
|
+
* at all while nobody can see it.
|
|
4
|
+
*
|
|
5
|
+
* The app attaches the element that shows a video (`attachVideo`). From that
|
|
6
|
+
* element's size on screen (CSS pixels × devicePixelRatio, at most 2) the
|
|
7
|
+
* smallest simulcast layer that fills it sharply is chosen; while no attached
|
|
8
|
+
* element is visible (scrolled away, `display: none`, the tab in the
|
|
9
|
+
* background) the video is paused at the media server, so it costs no
|
|
10
|
+
* bandwidth and no decoding. This is what LiveKit calls adaptiveStream and
|
|
11
|
+
* Tencent TRTC "small stream + receiveWhenViewVisible": the single biggest
|
|
12
|
+
* saving on a viewer with several tiles, where every tile otherwise decodes
|
|
13
|
+
* the publisher's full-size layer.
|
|
14
|
+
*
|
|
15
|
+
* Pure decisions are exported for tests; the browser parts (observers,
|
|
16
|
+
* timers) are injected.
|
|
17
|
+
*/
|
|
18
|
+
import type { VideoLayer } from './quality.js';
|
|
19
|
+
/** Where a video is drawn, in CSS pixels, and how it is fitted into that box. */
|
|
20
|
+
export interface ViewBox {
|
|
21
|
+
width: number;
|
|
22
|
+
height: number;
|
|
23
|
+
fit: 'cover' | 'contain';
|
|
24
|
+
}
|
|
25
|
+
/** Never ask for more than twice the CSS size: a 3× phone screen gains nothing visible from it. */
|
|
26
|
+
export declare const MAX_PIXEL_RATIO = 2;
|
|
27
|
+
/** A layer shown at most this much bigger than it is still looks sharp: avoids a whole layer for a few pixels. */
|
|
28
|
+
export declare const MAX_UPSCALE = 1.15;
|
|
29
|
+
/**
|
|
30
|
+
* The smallest layer that fills `box` sharply, or null when the layers are
|
|
31
|
+
* not known (an older publisher): then the server chooses, as before.
|
|
32
|
+
*
|
|
33
|
+
* With `cover` the picture is scaled until it fills the box (a part is
|
|
34
|
+
* cropped), with `contain` until it fits. Either way the scale is measured
|
|
35
|
+
* against each layer's own size, so a portrait layer list works too.
|
|
36
|
+
*/
|
|
37
|
+
export declare function layerForBox(layers: readonly VideoLayer[], box: ViewBox, pixelRatio?: number): number | null;
|
|
38
|
+
/** What one video should get now: paused, or a layer (null = let the server choose). */
|
|
39
|
+
export interface AdaptiveWant {
|
|
40
|
+
visible: boolean;
|
|
41
|
+
spatialLayer: number | null;
|
|
42
|
+
}
|
|
43
|
+
/**
|
|
44
|
+
* The largest view among the visible ones decides; any visible view keeps the video flowing. A view not measured
|
|
45
|
+
* yet (just attached, before layout) counts as shown at the server's choice, so attaching never pauses a video
|
|
46
|
+
* for the moment before the browser has sized the element.
|
|
47
|
+
*/
|
|
48
|
+
export declare function wantFor(views: readonly {
|
|
49
|
+
box: ViewBox;
|
|
50
|
+
visible: boolean;
|
|
51
|
+
measured?: boolean;
|
|
52
|
+
}[], layers: readonly VideoLayer[], pixelRatio: number, pageVisible: boolean): AdaptiveWant;
|
|
53
|
+
/** Pause a hidden video only after this long, so scrolling past or a quick tab switch does not stop it. */
|
|
54
|
+
export declare const PAUSE_AFTER_MS = 1500;
|
|
55
|
+
/** Go down a layer only after this long (going up is immediate): a tile shrinking for a moment keeps its picture. */
|
|
56
|
+
export declare const LOWER_AFTER_MS = 1200;
|
|
57
|
+
export interface AdaptiveHost {
|
|
58
|
+
/** The layers this producer's publisher declared (smallest first or any order); [] when unknown. */
|
|
59
|
+
layers(producerId: string): readonly VideoLayer[];
|
|
60
|
+
/** Apply a decision for one video. */
|
|
61
|
+
apply(producerId: string, want: AdaptiveWant): void;
|
|
62
|
+
/** For tests: the environment. */
|
|
63
|
+
pixelRatio?(): number;
|
|
64
|
+
pageVisible?(): boolean;
|
|
65
|
+
now?(): number;
|
|
66
|
+
setTimer?(fn: () => void, ms: number): unknown;
|
|
67
|
+
clearTimer?(handle: unknown): void;
|
|
68
|
+
/** For tests: stand-ins for ResizeObserver / IntersectionObserver / document visibility. */
|
|
69
|
+
observe?(element: Element, onChange: (box: ViewBox | null, visible: boolean | null) => void): () => void;
|
|
70
|
+
onPageVisibility?(listener: () => void): () => void;
|
|
71
|
+
}
|
|
72
|
+
/**
|
|
73
|
+
* The videos being shown, by producer id. Decisions are applied through the
|
|
74
|
+
* host; going up (or becoming visible) is applied at once, going down (or
|
|
75
|
+
* hiding) after a short delay.
|
|
76
|
+
*/
|
|
77
|
+
export declare class AdaptiveVideos {
|
|
78
|
+
private readonly host;
|
|
79
|
+
private readonly entries;
|
|
80
|
+
private stopPage;
|
|
81
|
+
constructor(host: AdaptiveHost);
|
|
82
|
+
/** Show `producerId` in `element`; returns the function that stops. */
|
|
83
|
+
attach(producerId: string, element: Element): () => void;
|
|
84
|
+
detach(producerId: string, element: Element): void;
|
|
85
|
+
/** Whether the app shows this video through attachVideo (its decisions then apply to new consumers too). */
|
|
86
|
+
has(producerId: string): boolean;
|
|
87
|
+
/** The decision in force, for a consumer made (or made again) after it: null when the app does not attach it. */
|
|
88
|
+
current(producerId: string): AdaptiveWant | null;
|
|
89
|
+
/**
|
|
90
|
+
* A new consumer was made with `current()` applied (its layers, and paused when hidden): record that as applied,
|
|
91
|
+
* so nothing is sent twice, and drop a pending change made for the old one.
|
|
92
|
+
*/
|
|
93
|
+
consumed(producerId: string): void;
|
|
94
|
+
/** Stop everything (the session closed). */
|
|
95
|
+
dispose(): void;
|
|
96
|
+
private decide;
|
|
97
|
+
private update;
|
|
98
|
+
private clear;
|
|
99
|
+
private observe;
|
|
100
|
+
private watchPage;
|
|
101
|
+
}
|
|
102
|
+
/** The page is visible (not a background tab). */
|
|
103
|
+
export declare function pageIsVisible(): boolean;
|
|
104
|
+
//# sourceMappingURL=adaptive.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"adaptive.d.ts","sourceRoot":"","sources":["../../src/adaptive.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAE/C,iFAAiF;AACjF,MAAM,WAAW,OAAO;IACtB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,OAAO,GAAG,SAAS,CAAC;CAC1B;AAED,mGAAmG;AACnG,eAAO,MAAM,eAAe,IAAI,CAAC;AACjC,kHAAkH;AAClH,eAAO,MAAM,WAAW,OAAO,CAAC;AAEhC;;;;;;;GAOG;AACH,wBAAgB,WAAW,CAAC,MAAM,EAAE,SAAS,UAAU,EAAE,EAAE,GAAG,EAAE,OAAO,EAAE,UAAU,SAAI,GAAG,MAAM,GAAG,IAAI,CAYtG;AAED,wFAAwF;AACxF,MAAM,WAAW,YAAY;IAC3B,OAAO,EAAE,OAAO,CAAC;IACjB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;CAC7B;AAED;;;;GAIG;AACH,wBAAgB,OAAO,CAAC,KAAK,EAAE,SAAS;IAAE,GAAG,EAAE,OAAO,CAAC;IAAC,OAAO,EAAE,OAAO,CAAC;IAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;CAAE,EAAE,EAAE,MAAM,EAAE,SAAS,UAAU,EAAE,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,GAAG,YAAY,CAWvL;AAED,2GAA2G;AAC3G,eAAO,MAAM,cAAc,OAAO,CAAC;AACnC,qHAAqH;AACrH,eAAO,MAAM,cAAc,OAAO,CAAC;AAEnC,MAAM,WAAW,YAAY;IAC3B,oGAAoG;IACpG,MAAM,CAAC,UAAU,EAAE,MAAM,GAAG,SAAS,UAAU,EAAE,CAAC;IAClD,sCAAsC;IACtC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,IAAI,EAAE,YAAY,GAAG,IAAI,CAAC;IACpD,kCAAkC;IAClC,UAAU,CAAC,IAAI,MAAM,CAAC;IACtB,WAAW,CAAC,IAAI,OAAO,CAAC;IACxB,GAAG,CAAC,IAAI,MAAM,CAAC;IACf,QAAQ,CAAC,CAAC,EAAE,EAAE,MAAM,IAAI,EAAE,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC;IAC/C,UAAU,CAAC,CAAC,MAAM,EAAE,OAAO,GAAG,IAAI,CAAC;IACnC,4FAA4F;IAC5F,OAAO,CAAC,CAAC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC,GAAG,EAAE,OAAO,GAAG,IAAI,EAAE,OAAO,EAAE,OAAO,GAAG,IAAI,KAAK,IAAI,GAAG,MAAM,IAAI,CAAC;IACzG,gBAAgB,CAAC,CAAC,QAAQ,EAAE,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC;CACrD;AAkBD;;;;GAIG;AACH,qBAAa,cAAc;IAIb,OAAO,CAAC,QAAQ,CAAC,IAAI;IAHjC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAA4B;IACpD,OAAO,CAAC,QAAQ,CAA6B;gBAEhB,IAAI,EAAE,YAAY;IAE/C,uEAAuE;IACvE,MAAM,CAAC,UAAU,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,MAAM,IAAI;IAsBxD,MAAM,CAAC,UAAU,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,IAAI;IAoBlD,4GAA4G;IAC5G,GAAG,CAAC,UAAU,EAAE,MAAM,GAAG,OAAO;IAIhC,iHAAiH;IACjH,OAAO,CAAC,UAAU,EAAE,MAAM,GAAG,YAAY,GAAG,IAAI;IAKhD;;;OAGG;IACH,QAAQ,CAAC,UAAU,EAAE,MAAM,GAAG,IAAI;IAOlC,4CAA4C;IAC5C,OAAO,IAAI,IAAI;IAUf,OAAO,CAAC,MAAM;IAId,OAAO,CAAC,MAAM;IAiCd,OAAO,CAAC,KAAK;IAKb,OAAO,CAAC,OAAO;IAKf,OAAO,CAAC,SAAS;CAMlB;AAoCD,kDAAkD;AAClD,wBAAgB,aAAa,IAAI,OAAO,CAEvC"}
|