lakutata 3.1.0 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1364 @@
1
+ # Lakutata WebRTC
2
+
3
+ WebRTC connects two endpoints directly, peer to peer: browsers, Electron applications and Node.js processes exchange
4
+ data, audio and video without a server relaying it, behind NATs and firewalls too. The package `@lakutata/webrtc`
5
+ provides everything around these connections: the signaling that introduces the endpoints to each other, a STUN and TURN
6
+ server for the networks that need one, a media server for the rooms of many members, the peer connections of Node.js,
7
+ and a client for the browsers. It is written in TypeScript without native code, and it is not part of `lakutata`:
8
+
9
+ ```bash
10
+ npm install lakutata @lakutata/webrtc
11
+ ```
12
+
13
+ | Entry point | Runs in | What it gives |
14
+ |---|---|---|
15
+ | `@lakutata/webrtc` | Node.js, the main process of Electron, Bun | `Signaling`, `SignalingController`, `TURNServer`, `SignalingClient`, `PeerConnection`, `PeerThreads`, `MediaTrack`, `SFU`, `WHIP`, `WHIPController`, `WHEP`, `WHEPController`, `Cluster`, `webrtcServiceEntrypoint`, `PeerServiceProxy`, `ChunkedChannel` |
16
+ | `@lakutata/webrtc/client` | Browsers, the renderer of Electron | `SignalingConnection`, `PeerServiceProxy`, `ChunkedChannel`, the types of the protocol |
17
+
18
+ The browser entry uses only the APIs of the browsers; a frontend installs the same package, and the framework is not
19
+ installed with it.
20
+
21
+ ## 1. How a connection is made
22
+
23
+ Two endpoints cannot connect until each knows how to reach the other. Each one describes itself in an **offer** or an
24
+ **answer** (its codecs, its keys, its data channels) and lists its **candidates**, the addresses it may be reached at:
25
+ its local addresses, its public address as a STUN server sees it, and an address on a TURN server that relays its
26
+ traffic when nothing else gets through. These messages travel through a server both endpoints are connected to, the
27
+ **signaling**; once the endpoints have them, they test the candidates in pairs (ICE), keep a working pair, and encrypt
28
+ the connection (DTLS). The data channels then carry messages in both directions, reliable and ordered or not, and the
29
+ tracks carry audio and video.
30
+
31
+ The signaling of this package works with **rooms**: a client joins a room, gets the members already there, and the
32
+ offers, answers and candidates are exchanged between each pair of members. The rest is done for you:
33
+
34
+ - The offers and the answers are exchanged as in the W3C "perfect negotiation": either member may add data channels and
35
+ tracks at any time, both at the same moment too.
36
+ - A connection that fails, or stays disconnected for 5 seconds, restarts ICE by itself.
37
+ - When the signaling connection drops, the peer-to-peer connections stay up, and the client joins its rooms again once
38
+ it reconnects.
39
+ - The members get the ICE servers (with TURN credentials) when they join, and new credentials before those expire.
40
+
41
+ ## 2. The server
42
+
43
+ ```typescript
44
+ import {Application} from 'lakutata'
45
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
46
+ import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
47
+ import {jsonRPCServiceEntrypoint} from 'lakutata/adapter/jsonrpc'
48
+ import {buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
49
+
50
+ Application.run(() => ({
51
+ id: 'meet.app',
52
+ name: 'Meet',
53
+ components: {
54
+ turn: buildTURNServerOptions({publicAddress: '203.0.113.10', relayPortRange: [49160, 49200], secret: process.env.TURN_SECRET}),
55
+ signaling: buildSignalingOptions({
56
+ turn: 'turn',
57
+ //Only the members whose id starts with board- join the room board
58
+ authorize: (peer, room): boolean => room !== 'board' || peer.peerId.startsWith('board-')
59
+ }),
60
+ //The browsers over Socket.IO on port 3000, the other clients over JSON-RPC (WebSocket) on port 3001
61
+ entrypoint: buildEntrypoints({
62
+ controllers: [SignalingController],
63
+ service: [socketIOServiceEntrypoint({port: 3000}), jsonRPCServiceEntrypoint({transport: 'websocket', port: 3001})]
64
+ })
65
+ },
66
+ bootstrap: ['turn', 'entrypoint']
67
+ }))
68
+ ```
69
+
70
+ `SignalingController` serves the requests of the signaling; it finds the component under the name `signaling`, which
71
+ finds the entrypoint under the name `entrypoint` (its `entrypoint` option). Its actions go through the middlewares of
72
+ the entrypoint like any other action. The member's id is the client id of its connection: the `auth.id` of Socket.IO
73
+ (its `id` header otherwise), the `id` header or query parameter of a WebSocket. Authenticate the requests with a
74
+ middleware, and decide who joins which room with `authorize`:
75
+ `(peer, room, context) => boolean | Promise<boolean>`, given the member's id and metadata and the context of its
76
+ request.
77
+
78
+ The options of `buildSignalingOptions`:
79
+
80
+ | Option | Default | Meaning |
81
+ |---|---|---|
82
+ | `entrypoint` | `'entrypoint'` | The entrypoint component pushing the events to the members |
83
+ | `turn` | — | A `TURNServer` component: the members get its ICE servers, with credentials |
84
+ | `iceServers` | `[]` | More ICE servers (another STUN or TURN server), after those of `turn` |
85
+ | `authorize` | — | Who joins which room; everyone joins any room without it |
86
+ | `maxPeers` | `0` | The members of a room at most; 0 for no limit |
87
+ | `maxMessageSize` | `65536` | The size of a signal and of a member's metadata at most, in bytes |
88
+ | `maxMessagesPerSecond` | `50` | The signals of a member per second on average (bursts of twice as many) |
89
+ | `heartbeatInterval` | `10000` | How often a client sends a heartbeat when the server cannot see its disconnection, in milliseconds |
90
+
91
+ The component also works without the controller: `join()`, `signal()`, `leave()` and `disconnect()` change the rooms
92
+ from the application's own code, and `rooms()` lists them. A member added by the application gets its events through
93
+ the `receive` option of `join()`. The ids starting with `sfu:` and `whip:` are kept for the members the server adds
94
+ itself (the media server, the streams of WHIP): a client joining with one is refused.
95
+
96
+ ## 3. The STUN and TURN server
97
+
98
+ Most connections succeed directly or with the public address a STUN server reports. Some networks (symmetric NATs,
99
+ corporate firewalls, mobile carriers) let nothing through but a relay: a TURN server forwards the traffic of the
100
+ connection. `TURNServer` is one, inside the application:
101
+
102
+ | Option | Default | Meaning |
103
+ |---|---|---|
104
+ | `host` | `'0.0.0.0'` | The address it listens on |
105
+ | `port` | `3478` | The port of STUN and TURN over UDP and TCP |
106
+ | `udp`, `tcp` | `true` | Whether to listen over UDP, over TCP |
107
+ | `tls` | — | `{cert, key, port}`: TURN over TLS (`turns:`, port 5349 by default), for the networks letting only TLS through |
108
+ | `publicAddress` | the first IPv4 address of the host that is not internal | The address the clients reach it at |
109
+ | `relayPortRange` | any free port | `[first, last]`: the UDP ports of the relays |
110
+ | `secret` | a random one at each start | The secret of the short-lived credentials |
111
+ | `users` | `{}` | Users with a fixed password, `{username: password}` |
112
+ | `realm` | `'lakutata'` | The realm of the credentials |
113
+ | `credentialTtl` | `86400` | How long the short-lived credentials are valid, in seconds; a relay made with them keeps working once they expire |
114
+ | `maxAllocations` | no limit | The relays at once at most |
115
+ | `workers` | `0` | Worker threads serving the clients, to use more cores (on Linux; see below) |
116
+
117
+ The short-lived credentials follow the REST API of coturn: the username is `<expiry>:<user>` and the password is
118
+ `base64(HMAC-SHA1(secret, username))`, so any server holding the secret checks them without storing them.
119
+ `turn.iceServers(user)` gives the ICE servers of a client with credentials, `turn.credentials(user)` the credentials
120
+ alone, `createTURNCredentials(secret, user, ttl)` makes them for another TURN server (a coturn configured with
121
+ `use-auth-secret`), and `turn.allocations` counts the relays in use.
122
+
123
+ ### Several cores
124
+
125
+ The server relays in the main thread by default. Measured on an Apple M4 with Node.js 24, it relayed a data channel
126
+ whose two ends both relayed through it (so it carried each packet twice) at about 38 MiB/s for messages of 16 KiB,
127
+ taking 27 s of CPU per GiB, and 365,000 messages of 100 bytes a second, at 2.8 µs of CPU each. A packet costs the same
128
+ with 500 idle allocations of other clients.
129
+
130
+ With `workers: 3`, three worker threads serve the clients. Their listeners share the ports of the server
131
+ (`SO_REUSEPORT`), and the system keeps each client on one thread, chosen from its address and its port; that thread
132
+ relays its traffic. The threads count their allocations together against `maxAllocations`, and `relayPortRange` is
133
+ shared out among them. Only Linux lets the threads share the ports: elsewhere, the server relays in the main thread and
134
+ says so in the debug messages (`webrtc/turn`). A thread that stops (only a bug stops one) loses its allocations and is
135
+ started again. Meanwhile the system spreads the clients over UDP among the other threads, then among all of them again,
136
+ so most of them find their allocation gone, as when a TURN server restarts; the connections over TCP and TLS of the
137
+ other threads stay.
138
+
139
+ In an arm64 Linux virtual machine (Docker Desktop, 10 cores) on the same Mac, with three connections relaying through it
140
+ at once (six clients):
141
+
142
+ | Server | Messages of 100 bytes relayed | CPU per message |
143
+ |---|---|---|
144
+ | The main thread | 460,000 to 550,000 a second | 1.9 to 2.3 µs |
145
+ | 3 worker threads | 510,000 to 820,000 a second in five runs of six, 220,000 in the other | 2.4 to 3.8 µs, 9 µs in the other run |
146
+
147
+ The system shares out the clients by their address and port, not by their load: with six clients, a thread often had
148
+ none, or twice its share. Large messages were therefore relayed at 34 to 79 MiB/s depending on the run, against 46 to
149
+ 50 MiB/s in the main thread; the more clients, the more evenly the threads share them. Each thread takes about 15 MiB.
150
+ In that virtual machine, a worker thread relayed a quarter to a third fewer messages per core than the main thread; on
151
+ macOS, the two cost the same.
152
+
153
+ ## 4. Browsers
154
+
155
+ ```typescript
156
+ import {io} from 'socket.io-client'
157
+ import {type Peer, type Room, SignalingConnection} from '@lakutata/webrtc/client'
158
+
159
+ const signaling: SignalingConnection = new SignalingConnection(io('https://meet.example.com', {auth: {id: 'alice'}}))
160
+ const room: Room = await signaling.join('room-1', {metadata: {name: 'Alice'}})
161
+ room.on('peer', (peer: Peer): void => {
162
+ //Unordered and unreliable: a lost position is not sent again
163
+ const positions: RTCDataChannel = peer.createDataChannel('positions', {ordered: false, maxRetransmits: 0})
164
+ positions.addEventListener('open', (): void => positions.send(JSON.stringify({x: 1, y: 2})))
165
+ peer.on('datachannel', (channel: RTCDataChannel): void => {
166
+ channel.addEventListener('message', (event: MessageEvent): void => console.log(peer.id, channel.label, event.data))
167
+ })
168
+ peer.on('connectionstatechange', (state: string): void => console.log(peer.id, state))
169
+ })
170
+ room.on('peer-left', (peer: Peer): void => console.log(`${peer.id} left`))
171
+ ```
172
+
173
+ `SignalingConnection` takes the connection of the application:
174
+
175
+ - a Socket.IO socket (`io()` of `socket.io-client`): the member's id is its `auth.id`, and Socket.IO reconnects by
176
+ itself;
177
+ - a `WebSocket` speaking JSON-RPC, the member's id being the `id` query parameter of its URL;
178
+ - a function creating that `WebSocket`, so the connection reconnects after a disconnection (after 1 second, then twice
179
+ as long each time, 30 seconds at most):
180
+ `new SignalingConnection(() => new WebSocket('wss://meet.example.com/rpc?id=alice'))`.
181
+
182
+ Its options: `iceTransportPolicy` (`'relay'` to use only the TURN relays), `configuration` (more options of the
183
+ `RTCPeerConnection`s), `createPeerConnection` (another implementation of `RTCPeerConnection`), `timeout` (of the
184
+ requests, 15 seconds by default).
185
+
186
+ A room emits `peer` for each other member: the members already in the room, then each newcomer. A listener added later
187
+ is called at once for the peers already there. In a room with a media server, `peer` is emitted once the connection
188
+ to the server has exchanged its first offer and answer, so a channel created in the listener does not collide with
189
+ that offer. `room.publish(track, stream)` sends a track to every member (section 8),
190
+ and `room.sfu` is the id of the media server of the room, when it has one (section 9). A `Peer` has the member's `id`,
191
+ its `metadata`, the `connection` to it (an `RTCPeerConnection`: read its statistics), `createDataChannel()`, `pause()`
192
+ and `resume()` (a track of the member), `restartIce()` and `close()`, and emits `datachannel`, `track`,
193
+ `connectionstatechange`, `metadata` (the member joined again with new metadata), `error` and `close`. A room emits
194
+ `peer-left` when a peer closes: its member left, or its connection was lost for good.
195
+
196
+ Both members may create channels: each side gets the channels of the other one through `datachannel`. For a single
197
+ channel shared by both, create it on both sides with `negotiated: true` and the same `id`.
198
+
199
+ In Electron, the renderer uses `@lakutata/webrtc/client` and connects to the signaling itself; the main process uses
200
+ `@lakutata/webrtc`.
201
+
202
+ ## 5. An application as a member
203
+
204
+ `SignalingClient` joins the rooms from a lakutata application, with the peer connections of Node.js. A device, a
205
+ recorder, a bot or a test joins the rooms as the browsers do:
206
+
207
+ ```typescript
208
+ import {Component} from 'lakutata'
209
+ import {type Peer, type PeerChannel, type Room, SignalingClient} from '@lakutata/webrtc'
210
+
211
+ export class Telemetry extends Component {
212
+ protected async init(): Promise<void> {
213
+ const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'device-7'})
214
+ const room: Room = await signaling.join('device-7')
215
+ room.on('peer', (peer: Peer): void => {
216
+ const channel: PeerChannel = peer.createDataChannel('readings')
217
+ const timer: NodeJS.Timeout = setInterval((): void => {
218
+ if (channel.readyState === 'open') channel.send(JSON.stringify({temperature: 21.5}))
219
+ }, 1000)
220
+ peer.on('close', (): void => clearInterval(timer))
221
+ })
222
+ }
223
+ }
224
+ ```
225
+
226
+ An `http:` or `https:` URL uses Socket.IO (install `socket.io-client`), a `ws:` or `wss:` URL uses JSON-RPC over the
227
+ `WebSocket` of Node.js 22 and Bun. Its options: `url`, `id` (a random UUID by default), `path` (of Socket.IO), `auth`
228
+ (more fields of the Socket.IO handshake, a token), `timeout`, and the options of its connections (`iceTransportPolicy`,
229
+ `icePortRange`, `iceAdditionalHostAddresses`, `codecs`, `threads`). It is `@Transient`: each `getObject()` gives a new
230
+ client, and `close()` leaves its rooms and disconnects.
231
+
232
+ ### Calling the actions of an application
233
+
234
+ `webrtcServiceEntrypoint()` lets the members of the rooms an application joins call its service actions
235
+ (`@ServiceAction`) over their peer-to-peer connection to it: a browser on another network calls the actions of a device
236
+ directly, and the calls do not go through a server. It is an adapter of the entrypoint's `service` channel, like those
237
+ of [Service Calls](ServiceCalls.md): the actions keep their input validation, access control and middlewares, and the
238
+ same actions can be served over Socket.IO or JSON-RPC at the same time.
239
+
240
+ ```typescript
241
+ import {Application} from 'lakutata'
242
+ import {buildEntrypoints, Controller} from 'lakutata/com/entrypoint'
243
+ import {ServiceAction} from 'lakutata/decorator/ctrl'
244
+ import {webrtcServiceEntrypoint} from '@lakutata/webrtc'
245
+
246
+ class DoorController extends Controller {
247
+ @ServiceAction({method: 'door.open'})
248
+ public async open(input: {door: string}): Promise<string> {
249
+ return `${input.door} is open`
250
+ }
251
+ }
252
+
253
+ Application.run({
254
+ id: 'door.device',
255
+ name: 'Door',
256
+ components: {
257
+ entrypoint: buildEntrypoints({
258
+ controllers: [DoorController],
259
+ //The device joins the room door-7 as the member door-7
260
+ service: webrtcServiceEntrypoint({
261
+ signaling: {url: 'https://meet.example.com', id: 'door-7'},
262
+ rooms: ['door-7']
263
+ })
264
+ })
265
+ },
266
+ bootstrap: ['entrypoint']
267
+ })
268
+ ```
269
+
270
+ A member of the room creates a data channel labeled `service` to the device, and calls through it with
271
+ `PeerServiceProxy`, which both entry points of the package export:
272
+
273
+ ```typescript
274
+ import {PeerServiceProxy, type Peer, type Room} from '@lakutata/webrtc/client'
275
+
276
+ declare const room: Room
277
+ room.on('peer', (peer: Peer): void => {
278
+ if (peer.id !== 'door-7') return
279
+ const door: PeerServiceProxy = new PeerServiceProxy(peer.createDataChannel('service'))
280
+ door.on('opened', (by: string): void => console.log(`opened by ${by}`))
281
+ door.invoke({method: 'door.open', door: 'front'}).then((result: string): void => console.log(result))
282
+ })
283
+ ```
284
+
285
+ - `invoke(input, timeout)` sends the input's `method` as the method of a JSON-RPC request and the other fields as its
286
+ parameters (to use another field, give both sides the same `methodField`), and resolves to the result. It fails with a
287
+ `PeerServiceError`: its `errno` is the one of the action's error (`E_ACCESS_DENY`, `E_INVALID_VALUE`…), its `code`
288
+ the JSON-RPC code, its `data` the whole error object; or its `errno` is `E_PEER_SERVICE_TIMEOUT` (10 minutes by
289
+ default) or `E_PEER_SERVICE_CLOSED`. A request over 10 MiB as JSON (the proxy's `maxRequestSize`, the same as the
290
+ adapter's `limit`) is not sent and fails at once with `E_PEER_SERVICE_TOO_LARGE`. The calls made before the channel
291
+ opens wait for it.
292
+ - `this.context.clientId` is the member id of the caller. `webrtcRequestOfContext(this.context)` gives its `peer` (with
293
+ its metadata), the `room` it called through and the `channel`. `entrypoint.notify(memberId, event, ...args)` pushes an
294
+ event to the channels of that member, and `broadcast()` to every channel; `on`, `once` and `off` of the proxy receive
295
+ them.
296
+ - The rooms are joined once the application has launched, so the signaling server may run in the same application. A
297
+ room that cannot be joined is reported to `onError` (the application's log by default) and tried again after 1
298
+ second, then after twice as long each time, up to 10 seconds; a room the signaling refuses is not tried again.
299
+ - When a channel closes (its member left), its requests in progress are aborted: `this.signal` of the action fires.
300
+ When the application stops, the new requests fail with the code `-32001`, the requests in progress are answered, and
301
+ then the channels are closed.
302
+ - The adapter serves every service action of its entrypoint, as the other service adapters do, except those of the
303
+ signaling: give the actions meant for the members an entrypoint of their own, or limit them with rules or
304
+ middlewares. The channels with another label are left to the application.
305
+ - In a room with a media server, the channel goes through the server, which sees the messages; for calls that only the
306
+ two members should see, use a room without one.
307
+
308
+ The options of `webrtcServiceEntrypoint()`:
309
+
310
+ | Option | Default | Meaning |
311
+ |---|---|---|
312
+ | `signaling` | — | The options of a `SignalingClient` the adapter creates, and closes when it stops |
313
+ | `client` | — | Instead of `signaling`, a function returning the client of the application, when the application uses it too (two clients with the same id would replace each other in the rooms); the adapter does not leave its rooms or close it |
314
+ | `rooms` | — | The rooms whose members may call; a room the client has already joined is served as it is |
315
+ | `metadata` | — | The metadata of the application in the rooms it joins |
316
+ | `label` | `'service'` | The label of the channels carrying the calls |
317
+ | `limit` | 10 MiB | The largest request; a larger one is answered with the code `-32600` |
318
+ | `methodField`, `toPattern`, `positionalParams` | `'method'` | How a request maps to an action, as with `jsonRPCServiceEntrypoint()` |
319
+ | `onError` | the application's log | Called with the error when a room cannot be joined |
320
+
321
+ The member id is only as reliable as the signaling server makes it. The server sends the signals of each member under
322
+ the id it knows that member by, so a member cannot pass for another one; but the id is the client id of the member's
323
+ connection to the server, which the client gives, and the metadata is what the member gave. Before an action relies on
324
+ `clientId` to decide who may call it, make the signaling server check its clients: authenticate them with a middleware
325
+ and decide who joins which room with `authorize` (section 2).
326
+
327
+ ## 6. Peer connections without the signaling
328
+
329
+ `PeerConnection` has the interface of `RTCPeerConnection`, so the code of a browser works in Node.js, and any signaling
330
+ works: an HTTP request, a QR code, a message queue. Its events are given in both styles: `addEventListener` and the
331
+ `on…` properties with event objects, or `on()` with the values (`on('datachannel', (channel) => …)`).
332
+
333
+ ```typescript
334
+ import {Component} from 'lakutata'
335
+ import {PeerConnection, type PeerChannel, type SignalingCandidate, type SignalingDescription} from '@lakutata/webrtc'
336
+
337
+ export class Echo extends Component {
338
+ //Answer an offer received through an HTTP request, with all the candidates in the answer
339
+ public async answer(offer: SignalingDescription): Promise<SignalingDescription | null> {
340
+ const connection: PeerConnection = await this.getObject(PeerConnection, {iceServers: [{urls: 'stun:stun.example.com:3478'}]})
341
+ connection.on('datachannel', (channel: PeerChannel): void => {
342
+ channel.on('message', (data: string | Buffer | ArrayBuffer): void => channel.send(data.toString()))
343
+ })
344
+ await connection.setRemoteDescription(offer)
345
+ await connection.setLocalDescription()
346
+ await new Promise<void>((resolve: () => void): void => {
347
+ connection.on('icecandidate', (candidate: SignalingCandidate | null): void => {
348
+ if (!candidate) resolve()
349
+ })
350
+ })
351
+ return connection.localDescription
352
+ }
353
+ }
354
+ ```
355
+
356
+ Its options: `iceServers`, `iceTransportPolicy`, and those of a server: `icePortRange` (the UDP ports of its candidates,
357
+ to open in a firewall) and `iceAdditionalHostAddresses` (more addresses offered as candidates, such as the public
358
+ address of a NAT forwarding these ports with the same numbers); `codecs` chooses the audio and video codecs.
359
+ `headerExtensions: true` adds the RTP header extensions a media server uses: each side then estimates, from the feedback
360
+ of the other one, the bandwidth available for what it sends (without them, a browser goes by the packets lost only,
361
+ starting low and rising slowly). A `PeerChannel` sends strings and binary data; a binary message arrives as a `Buffer`,
362
+ or as an `ArrayBuffer` when `binaryType` is `'arraybuffer'`. `threads` serves it in a worker thread (see below).
363
+
364
+ ### Several cores
365
+
366
+ A connection runs its ICE, DTLS, SCTP and SRTP in the main thread by default. Given the name of a `PeerThreads`
367
+ component (its `threads` option, which `SignalingClient` has too, for the connections to the other members), it runs
368
+ them in one of the worker threads of that component, while its object, its events and the messages of its channels stay
369
+ in the main thread: the code using it is the same. Its methods are carried to the thread and resolve once done there;
370
+ its states are the ones the thread last reported, with its events and with the reply to each method. The packets of the
371
+ tracks it receives come to the main thread while they are read or sent on, and the packets of the tracks it sends go to
372
+ the thread.
373
+
374
+ ```typescript
375
+ import {Application} from 'lakutata'
376
+ import {buildPeerThreadsOptions, type PeerChannel, PeerConnection} from '@lakutata/webrtc'
377
+
378
+ Application
379
+ .run(() => ({
380
+ id: 'gateway.app',
381
+ name: 'Gateway',
382
+ components: {
383
+ //Four threads at most, started when needed
384
+ peerThreads: buildPeerThreadsOptions({threads: 4})
385
+ }
386
+ }))
387
+ .onLaunched(async (app: Application): Promise<void> => {
388
+ //Its engine runs in one of the threads
389
+ const connection: PeerConnection = await app.getObject(PeerConnection, {threads: 'peerThreads'})
390
+ const channel: PeerChannel = connection.createDataChannel('data')
391
+ channel.on('open', (): void => channel.send('hello'))
392
+ })
393
+ ```
394
+
395
+ The threads start when needed: a new connection goes to a thread serving none (a new one is started while there are
396
+ fewer than `threads`, by default the number of cores less one), else to the thread serving the fewest. A connection
397
+ stays in its thread; a thread that stops (only a bug stops one) fails its connections (`failed`, then `closed`), and is
398
+ started again for the next ones. `resourceLimits` limits the heap of each thread, as `workerResourceLimits` does for the
399
+ media server (Bun ignores it). The bytes of the messages received on their way to the main thread stay out of the
400
+ receive window: a main thread that reads slowly slows the other peer down, as it would without threads. The RTP packets
401
+ on their way between the threads are 2 MiB at most per connection and direction; beyond that, they are dropped, as a
402
+ full UDP buffer would drop them.
403
+
404
+ Measured on an Apple M4 with Node.js 24, pairs of connections in one process exchanging messages over a data channel
405
+ (`packages/webrtc/test-env/bench/threads.mjs` in the repository):
406
+
407
+ | Pairs | Engines | Messages of 100 bytes | CPU per message | Messages of 16 KiB |
408
+ |---|---|---|---|---|
409
+ | 1 | in the main thread | 274,000 a second | 4.2 µs | 45 MiB/s |
410
+ | 1 | in 2 worker threads | 539,000 a second | 4.5 µs | 85 MiB/s |
411
+ | 4 | in the main thread | 267,000 a second | 4.3 µs | 46 MiB/s |
412
+ | 4 | in 4 worker threads | 782,000 a second | 6.2 µs | 138 MiB/s |
413
+
414
+ The two connections of a pair work at once in two threads: one pair carries twice as much as in the main thread, for
415
+ 0.3 µs more of CPU per message. Four pairs in four threads carry three times as much; the CPU per message rises once the
416
+ threads outnumber the four performance cores of the M4. In an arm64 Linux virtual machine (Docker Desktop, Alpine) on
417
+ the same Mac, one pair carried 414,000 messages of 100 bytes a second in two threads, against 212,000 in the main
418
+ thread, at 6.8 µs of CPU each against 5.5; four pairs carried 460,000 against 222,000, at 10.7 µs against 5.4, and
419
+ 70 MiB/s of large messages against 47, for 2.5 times the CPU: there, several threads allocating memory at once cost more
420
+ per message. The threads give a process more connections, using more cores, not cheaper messages.
421
+
422
+ In that virtual machine, each thread took about 100 MiB more with small messages, 55 MiB with large ones. On macOS, with
423
+ large messages through four pairs in four threads (138 MiB/s), the memory of the process (its physical footprint) in
424
+ most runs rose to between 1.5 and 3.5 GB in the first minute, then fell back: in a three-minute run, it stayed at about
425
+ 700 MiB after the first minute. That is memory freed and kept for a while by the allocator of macOS, not a leak.
426
+ Node.js takes a 64 KiB buffer for each UDP datagram it receives; when several threads do so at that rate while other
427
+ small buffers are still in use, the allocator keeps much of the freed memory. The same run stays at about 540 MiB on
428
+ Linux, and at about 200 MiB in Bun on macOS, which has an allocator of its own.
429
+
430
+ ## 7. Large messages and files
431
+
432
+ A data channel carries messages of up to 64 KiB with any peer; Chrome and the connections of this package accept up to
433
+ 256 KiB. `ChunkedChannel` removes the limit: wrapped around the channel on both sides, it cuts each message into chunks
434
+ of 16 KiB, sends them as the channel's buffer empties, and puts them back together on the other side. It sends strings,
435
+ bytes and `Blob`s (a file of a browser is read a chunk at a time):
436
+
437
+ ```typescript
438
+ import {ChunkedChannel, type Peer} from '@lakutata/webrtc/client'
439
+
440
+ declare const peer: Peer
441
+ declare const file: File
442
+ const transfer: ChunkedChannel = new ChunkedChannel(peer.createDataChannel('files'))
443
+ transfer.channel.addEventListener('open', async (): Promise<void> => {
444
+ await transfer.send(JSON.stringify({name: file.name, size: file.size}))
445
+ await transfer.send(file)
446
+ })
447
+ ```
448
+
449
+ The channel must be ordered and reliable (the default). A message received over `maxMessageSize` (64 MiB by default)
450
+ is dropped with an `error` event.
451
+
452
+ ## 8. Audio and video
453
+
454
+ A track carries the audio or the video of one source. In a browser, publish the tracks of the camera and the microphone
455
+ in the room: `room.publish()` sends a track to every member, the ones joining later too. The other members get it with
456
+ the `track` event of their peer, `event.streams` grouping the audio and the video of a source:
457
+
458
+ ```typescript
459
+ import {type Peer, type Room} from '@lakutata/webrtc/client'
460
+
461
+ declare const room: Room
462
+ declare const grid: HTMLElement
463
+ const camera: MediaStream = await navigator.mediaDevices.getUserMedia({audio: true, video: true})
464
+ for (const track of camera.getTracks()) await room.publish(track, camera)
465
+ room.on('peer', (peer: Peer): void => {
466
+ const video: HTMLVideoElement = document.createElement('video')
467
+ video.autoplay = true
468
+ grid.append(video)
469
+ peer.on('track', (event: RTCTrackEvent): void => {
470
+ video.srcObject = event.streams[0]
471
+ })
472
+ peer.on('close', (): void => video.remove())
473
+ })
474
+ ```
475
+
476
+ `room.unpublish(track)` stops sending a track; its transceivers are kept for the next track published.
477
+ `peer.pause(track)` and `peer.resume(track)` stop and start receiving a track of a member, a video not on the screen
478
+ for example. Without a media server, a track added to `peer.connection` goes to that member only.
479
+
480
+ Without a media server, each member sends its video once per other member: this suits small rooms. With the media
481
+ server of section 9, each member sends it once.
482
+
483
+ Node.js captures, encodes and decodes nothing: the tracks of an application carry RTP packets, the encoded audio and
484
+ video, from an encoder or from another connection.
485
+
486
+ - `new MediaTrack(kind, {codec})` creates a track to send, and `writeRtp()` sends a packet: an `RtpPacket`, or the
487
+ bytes of one, such as the datagrams ffmpeg and GStreamer send to a UDP port. Each connection gives the packets its own
488
+ synchronization source and payload type, and the sequence numbers and the timestamps go on from the ones written.
489
+ - A received track (the `track` event of a `Peer` or of a `PeerConnection`) gives the packets of the other member with
490
+ its `rtp` event; `requestKeyFrame()` asks the other member for a key frame.
491
+ - A received track added to another connection forwards its packets there, and the key frame requests of that
492
+ connection go back to the sender of the track (one every 500 ms at most).
493
+ - The `keyframerequest` event of a track fires when a peer the track is sent to wants a key frame.
494
+
495
+ A camera sending what ffmpeg encodes:
496
+
497
+ ```typescript
498
+ import {createSocket, type Socket} from 'node:dgram'
499
+ import {Component} from 'lakutata'
500
+ import {MediaTrack, type Room, SignalingClient} from '@lakutata/webrtc'
501
+
502
+ //ffmpeg -f v4l2 -i /dev/video0 -c:v libx264 -profile:v baseline -pix_fmt yuv420p -tune zerolatency -g 30
503
+ // -bsf:v dump_extra=freq=keyframe -f rtp rtp://127.0.0.1:5004
504
+ export class Camera extends Component {
505
+ protected async init(): Promise<void> {
506
+ const video: MediaTrack = new MediaTrack('video', {codec: 'video/H264'})
507
+ const socket: Socket = createSocket('udp4')
508
+ socket.on('message', (datagram: Buffer): void => video.writeRtp(datagram))
509
+ socket.bind(5004)
510
+ const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'camera-3'})
511
+ const room: Room = await signaling.join('camera-3')
512
+ await room.publish(video, 'camera-3')
513
+ }
514
+ }
515
+ ```
516
+
517
+ A viewer starts decoding at a key frame: give the encoder a key frame every second or two (`-g` of ffmpeg), or have the
518
+ encoder make one when the track emits `keyframerequest`, if it can. The bitstream filter `dump_extra` puts the
519
+ parameters of H.264 before each key frame, where the browsers need them.
520
+
521
+ An application sending the camera of a member on to the viewers who join, each viewer getting the video from the
522
+ application instead of the camera (the media server of section 9 does it for every member of a room):
523
+
524
+ ```typescript
525
+ import {Component} from 'lakutata'
526
+ import {type MediaTrack, type Peer, type PeerTrackEvent, type Room, SignalingClient} from '@lakutata/webrtc'
527
+
528
+ //The page of the camera adds its tracks to the peer of the relay only
529
+ export class Relay extends Component {
530
+ protected async init(): Promise<void> {
531
+ const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'relay'})
532
+ const room: Room = await signaling.join('stage')
533
+ const tracks: MediaTrack[] = []
534
+ const viewers: Set<Peer> = new Set()
535
+ room.on('peer', (peer: Peer): void => {
536
+ if (peer.id === 'camera') {
537
+ peer.on('track', (event: PeerTrackEvent): void => {
538
+ tracks.push(event.track)
539
+ for (const viewer of viewers) viewer.connection.addTrack(event.track, 'stage')
540
+ })
541
+ return
542
+ }
543
+ viewers.add(peer)
544
+ for (const track of tracks) peer.connection.addTrack(track, 'stage')
545
+ peer.on('close', (): void => void viewers.delete(peer))
546
+ })
547
+ }
548
+ }
549
+ ```
550
+
551
+ The methods of `PeerConnection` are those of `RTCPeerConnection`: `addTrack(track, ...streamIds)` (the other member
552
+ groups the tracks of a stream), `addTransceiver(trackOrKind, {direction, streams})` (`recvonly` to receive without
553
+ sending), `removeTrack(sender)`, `getTransceivers()`, `getSenders()` and `getReceivers()`; `sender.replaceTrack()` sends
554
+ another track, of any codec negotiated, without negotiating again. The codecs are Opus and PCMU for the audio, VP8,
555
+ H.264 and VP9 for the video; the `codecs` option of `PeerConnection` and of `SignalingClient` chooses them and their
556
+ order, and a track created with a `codec` makes its connections negotiate that one.
557
+
558
+ ## 9. The media server
559
+
560
+ Without a media server, each member of a room sends its tracks to each other member: in a meeting of eight, each one
561
+ uploads its video seven times and keeps seven connections. The `SFU` component is a media server for the rooms of the
562
+ signaling (a selective forwarding unit): each member connects to it only, sends each track once, and receives the
563
+ tracks of the others from it. The server forwards their packets without decoding them, and a member uploads its video
564
+ once, however large the room.
565
+
566
+ ```typescript
567
+ import {Application} from 'lakutata'
568
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
569
+ import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
570
+ import {buildSFUOptions, buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
571
+
572
+ Application.run(() => ({
573
+ id: 'meet.app',
574
+ name: 'Meet',
575
+ components: {
576
+ turn: buildTURNServerOptions({publicAddress: '203.0.113.10', relayPortRange: [49160, 49200], secret: process.env.TURN_SECRET}),
577
+ signaling: buildSignalingOptions({turn: 'turn'}),
578
+ //The meetings go through the media server; the other rooms stay peer to peer
579
+ sfu: buildSFUOptions({
580
+ serves: (room: string): boolean => room.startsWith('meeting-'),
581
+ iceAdditionalHostAddresses: ['203.0.113.10'],
582
+ icePortRange: [40000, 40999]
583
+ }),
584
+ entrypoint: buildEntrypoints({
585
+ controllers: [SignalingController],
586
+ service: socketIOServiceEntrypoint({port: 3000})
587
+ })
588
+ },
589
+ bootstrap: ['turn', 'sfu', 'entrypoint']
590
+ }))
591
+ ```
592
+
593
+ The clients need no change. The server joins each room it serves as a member whose id starts with `sfu:`, and the
594
+ clients of this package (`SignalingConnection` in the browsers, `SignalingClient` in Node.js) connect to it instead of
595
+ to each other. The rooms and the peers work as without it:
596
+
597
+ - `room.publish()` sends a track once, to the server, which forwards it to every member, the ones joining later too.
598
+ - Each peer emits the tracks of its member. Their `event.streams` are the streams of the connection to the server;
599
+ `room.streamOf(id)` gives the member a stream comes from, and the id of the stream on the connection of that member.
600
+ - `peer.createDataChannel()` opens a channel through the server, which relays its messages both ways. Unlike between
601
+ two members, the server sees them. Such a channel cannot be given a `protocol`, and cannot be `negotiated`.
602
+ - `peer.pause(track)` asks the server to stop sending a track, `peer.resume(track)` to send it again.
603
+ - `room.sfu` is the id of the server, and `peer.connection` is the connection to it, the same for all the peers:
604
+ publish the tracks with `room.publish()` rather than add them to it.
605
+
606
+ The options of `buildSFUOptions`:
607
+
608
+ | Option | Default | Meaning |
609
+ |---|---|---|
610
+ | `signaling` | `'signaling'` | The `Signaling` component whose rooms the server serves |
611
+ | `serves` | all the rooms | `(room) => boolean`: the rooms served; the others stay peer to peer |
612
+ | `select` | every member receives every track | `(subscriber, track) => boolean`: whether a member receives a track |
613
+ | `maxMembers` | `0` | The members of all the rooms at most, 0 for no limit; a member joining once it is reached is refused (`SignalingLimitException`) |
614
+ | `codecs` | Opus, PCMU, VP8, H.264, VP9 | The codecs the members may publish |
615
+ | `icePortRange`, `iceAdditionalHostAddresses` | | The UDP ports of the connections, and the public address of the server. Each connection (a member, a stream of WHIP, a viewer of WHEP) takes a port on each address of the machine: make the range large enough for all of them |
616
+ | `workers` | `0` | Worker threads forwarding the rooms, to use more cores |
617
+ | `workerResourceLimits` | the limits of V8 | The limits of the heap and the stack of each worker thread (`resourceLimits` of `node:worker_threads`), to use less memory for a little more CPU (see Several cores below); Bun ignores them |
618
+ | `cluster`, `selectSFU` | | The media servers of several nodes (section 13) |
619
+
620
+ `select` is asked for each member and each track of the others, and asked again when a member joins the room again
621
+ with other metadata: the tracks the member receives follow the new answers. A `select` that throws counts as `false`,
622
+ and its error goes to the debug messages (section 15). In a webinar, for example, everyone receives the audio of all
623
+ and the video of the speakers only:
624
+
625
+ ```typescript
626
+ import {buildSFUOptions, type SFUTrack, type SignalingPeer} from '@lakutata/webrtc'
627
+
628
+ const sfu = buildSFUOptions({
629
+ select: (subscriber: SignalingPeer, track: SFUTrack): boolean => track.kind === 'audio' || track.peerId.startsWith('speaker-')
630
+ })
631
+ ```
632
+
633
+ How the server forwards:
634
+
635
+ - It does not transcode: a member receives the tracks of the codecs it supports, and the server emits `unsupported`
636
+ for the others. For the browsers, publish the video in VP8 or H.264.
637
+ - The key frame requests of the members go to the publisher, one every 500 ms at most, and the server asks the
638
+ publisher for a key frame when a member starts receiving a video. The packets lost on the way to a member are sent
639
+ again by the server.
640
+ - Each connection estimates its bandwidth from the feedback of the other side: the browsers adapt what they send to
641
+ the server, and the server emits `congestion` when the bandwidth to a member stays too low for what it receives, and
642
+ again once it is enough. The server sends each video as it is published (there is no simulcast yet): pause some of
643
+ the videos the member receives to lower the load.
644
+ - A member reading the messages of a channel slower than they are sent makes their sender wait; the server keeps a few
645
+ megabytes of them at most.
646
+
647
+ Its events: `publish` and `unpublish` (an `SFUTrack`: its `room`, `peerId`, `id`, `kind`, `codec` and `stream`) when a
648
+ member starts and stops publishing a track; `unsupported` (the room, the id of the member, the track) when a member
649
+ does not support the codec of a track it should receive; `congestion` (the room, the id of the member, whether the
650
+ bandwidth is too low, the bit rate estimated in bits per second). `rooms()` lists the rooms with their members, their
651
+ tracks and their viewers of WHEP. `track(room, peerId, trackId)` gives a track published as a `MediaTrack`, whose `rtp`
652
+ event gives its packets, to record it or to send it on another connection:
653
+
654
+ ```typescript
655
+ import {Component} from 'lakutata'
656
+ import type {RtpPacket, SFU, SFUTrack} from '@lakutata/webrtc'
657
+
658
+ export class Recorder extends Component {
659
+ protected async init(): Promise<void> {
660
+ const sfu: SFU = await this.getObject<SFU>('sfu')
661
+ sfu.on('publish', (track: SFUTrack): void => {
662
+ if (track.kind !== 'audio') return
663
+ sfu.track(track.room, track.peerId, track.id)?.on('rtp', (packet: RtpPacket): void => console.log(track.peerId, packet.timestamp))
664
+ })
665
+ }
666
+ }
667
+ ```
668
+
669
+ ### Several cores
670
+
671
+ The server forwards in the main thread by default. With `workers: 4`, four worker threads forward the rooms: each room
672
+ in one of them (the one forwarding the fewest members and viewers when the room opens), with its connections and its
673
+ UDP ports, while the hooks and the events stay in the main thread. `icePortRange` is shared out among the threads. A
674
+ thread that stops closes its rooms; the server opens them again in another thread, and their members connect to it
675
+ again. `rooms()` shows what each thread reports, so a change made in a thread (a viewer leaving) shows a moment later.
676
+
677
+ Measured on an Apple M4 with Node.js 24, each publisher sending a video of about 2 Mbit/s (200 packets of 1250 bytes
678
+ a second), the members in other processes:
679
+
680
+ | Rooms | Packets to forward | Forwarded | CPU |
681
+ |---|---|---|---|
682
+ | 1 room of 4 publishers and 10 subscribers, in the main thread | 7,600 a second | all | 45% of a core |
683
+ | 4 rooms of 12 publishers and 10 subscribers each, in the main thread | about 92,000 a second | 86,000 to 88,000 a second | 145%: the main thread is full, the rest is the garbage collector |
684
+ | The same 4 rooms, in 4 worker threads | about 92,000 a second | all | 174% |
685
+
686
+ At a load as light as the first room's, the share of a core depends on how fast the system runs the cores, and it runs
687
+ them faster when the machine is busy: with two other cores kept busy, the same room took 21%.
688
+
689
+ In an arm64 Linux virtual machine (Docker Desktop, 10 cores) on the same Mac, with the members running in it too, the
690
+ main thread forwarded all the packets of the 4 rooms (93,000 a second) at 160% of a core. Four threads took 380% to
691
+ 390% of a core, and forwarded from 69,000 to all of the 94,000 a second, depending on the run: there, at that load, the
692
+ threads do not pay off.
693
+
694
+ A thread takes about 22 MiB and 90 ms to start. Under the load above, the process used 1.4 GiB with the threads and
695
+ 0.6 GiB without: each thread has a heap of its own, with the large young generation V8 gives it for speed. With
696
+ `workerResourceLimits: {maxYoungGenerationSizeMb: 32}`, the process used 1.0 to 1.2 GiB, for 4% more CPU (181% of a
697
+ core instead of 174%). A track forwarded from one node to another (section 13) costs the two nodes about 11 µs of CPU
698
+ per packet: 0.2% of a core for such a video.
699
+
700
+ ## 10. Publishing with WHIP
701
+
702
+ WHIP (RFC 9725) publishes a stream with one HTTP request: the encoder (OBS, GStreamer, ffmpeg) or the browser posts its
703
+ offer to the URL of the stream, and gets the answer and the URL of its session. The `WHIP` component answers the offers
704
+ and emits `publish` with the session (its `id`, `stream`, `connection` and `tracks`) and `unpublish` when it ends: the
705
+ publisher deleted it, its connection failed or closed, or the application stopped. `WHIPController` is its HTTP
706
+ interface; the HTTP adapter must read `application/sdp` as text (its `body.text` option).
707
+
708
+ ```typescript
709
+ import {Application, Component} from 'lakutata'
710
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
711
+ import {expressHTTPEntrypoint} from 'lakutata/adapter/express'
712
+ import {buildWHIPOptions, type MediaTrack, type WHIP, type WHIPSession, WHIPController} from '@lakutata/webrtc'
713
+
714
+ export class Studio extends Component {
715
+ protected async init(): Promise<void> {
716
+ const whip: WHIP = await this.getObject<WHIP>('whip')
717
+ whip.on('publish', (session: WHIPSession): void => {
718
+ session.tracks?.forEach((track: MediaTrack): void => console.log(`${session.stream}: ${track.kind} ${track.codec}`))
719
+ })
720
+ whip.on('unpublish', (session: WHIPSession): void => console.log(`${session.stream} ended`))
721
+ }
722
+ }
723
+
724
+ Application.run(() => ({
725
+ id: 'studio.app',
726
+ name: 'Studio',
727
+ components: {
728
+ whip: buildWHIPOptions({
729
+ authorize: ({stream, token}): boolean => stream === 'live' && token === process.env.WHIP_TOKEN,
730
+ iceAdditionalHostAddresses: ['203.0.113.10'],
731
+ icePortRange: [40000, 40999]
732
+ }),
733
+ studio: {class: Studio},
734
+ entrypoint: buildEntrypoints({
735
+ controllers: [WHIPController],
736
+ http: expressHTTPEntrypoint({port: 8080, body: {text: ['application/sdp', 'application/trickle-ice-sdpfrag']}})
737
+ })
738
+ },
739
+ bootstrap: ['studio', 'entrypoint']
740
+ }))
741
+ ```
742
+
743
+ In OBS, choose the service WHIP, the server `https://studio.example.com/whip/live` and the bearer token; ffmpeg
744
+ publishes with `-f whip -authorization <token> https://studio.example.com/whip/live` (stereo audio: `-ac 2`). The tracks
745
+ of the session give their RTP packets, to record them or to send them on to other connections (section 8).
746
+
747
+ | Request | Answer |
748
+ |---|---|
749
+ | `POST /whip/:stream` with the offer (`application/sdp`) | `201 Created`: the answer, `Location` (the session), `Link` (the ICE servers), `ETag` (its ICE session) |
750
+ | `DELETE /whip/:stream/:session` | `200`: the session ends |
751
+ | `OPTIONS` | `204`, with the CORS headers and `Accept-Post: application/sdp` |
752
+ | `GET` | `204` |
753
+ | `PATCH /whip/:stream/:session` with a fragment of SDP (`application/trickle-ice-sdpfrag`) | `204`: the candidates of the publisher (trickle ICE); `200`: an ICE restart, with the fragment of the answer and a new `ETag`; `412` for an `If-Match` other than `*` and the `ETag` |
754
+
755
+ The refusals: 401 with `WWW-Authenticate: Bearer` when `authorize` refuses, 415 for a body that is not
756
+ `application/sdp` (`application/trickle-ice-sdpfrag` for a PATCH), 400 for an offer without audio or video, 503 once
757
+ there are `maxSessions` sessions, 404 for an unknown session. The URL of a session is the capability to end it: its id
758
+ is random, and its DELETE and PATCH are not authorized again. A subclass with `@Prefix({route: '/live'})` serves the
759
+ same interface under another path.
760
+
761
+ | Option | Default | Meaning |
762
+ |---|---|---|
763
+ | `authorize` | none | `({stream, token}, context) => boolean`: who may publish which stream |
764
+ | `turn` | none | The name of a `TURNServer` component: the publishers get its ICE servers in the `Link` headers |
765
+ | `iceServers` | `[]` | More ICE servers for the publishers |
766
+ | `maxSessions` | `100` | The sessions at once at most, `0` for no limit |
767
+ | `allowOrigin` | `'*'` | The origins the browsers may publish from |
768
+ | `icePortRange`, `iceAdditionalHostAddresses` | | The candidates of the server: its UDP ports (each session takes a port on each address of the machine; with `sfu`, the ports of the media server are used), its public address |
769
+ | `codecs` | Opus, PCMU, VP8, H.264, VP9 | The codecs accepted |
770
+ | `sfu` | none | The name of an `SFU` component: the streams go to its rooms |
771
+ | `roomOf` | the stream | `({stream, token}, context) => string`: the room of the media server a stream goes to |
772
+
773
+ The answer carries all the candidates of the server, which gathers them before answering: behind a NAT or in a
774
+ container, give it its public address and open its ports.
775
+
776
+ With `sfu`, a stream goes to a room of the media server (section 9) instead of the application: the room named after
777
+ the stream, or the one `roomOf` gives. It joins the room as a member whose id starts with `whip:`, the members of the
778
+ room receive its tracks as those of any member, and the viewers of WHEP play it (section 11). Its connection then
779
+ belongs to the media server, so the session has the `room` and the `peerId` of the stream instead of `connection` and
780
+ `tracks`; `sfu.track()` gives its packets.
781
+
782
+ ## 11. Playing with WHEP
783
+
784
+ WHEP (the WebRTC-HTTP Egress Protocol) plays a stream with one HTTP request: the player posts a receive-only offer, and
785
+ gets the answer and the URL of its session. The `WHEP` component plays the rooms of the media server:
786
+ each viewer receives the audio and the video of one member of a room, by default the stream published into it with
787
+ WHIP. The viewers are not members of the room, so the members do not see them, and a stream may have many of them.
788
+ `WHEPController` is its HTTP interface, under `/whep`.
789
+
790
+ ```typescript
791
+ import {Application} from 'lakutata'
792
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
793
+ import {expressHTTPEntrypoint} from 'lakutata/adapter/express'
794
+ import {buildSFUOptions, buildSignalingOptions, buildWHEPOptions, buildWHIPOptions, WHEPController, WHIPController} from '@lakutata/webrtc'
795
+
796
+ //OBS publishes to https://live.example.com/whip/concert, the browsers play https://live.example.com/whep/concert
797
+ Application.run(() => ({
798
+ id: 'live.app',
799
+ name: 'Live',
800
+ components: {
801
+ signaling: buildSignalingOptions(),
802
+ sfu: buildSFUOptions({iceAdditionalHostAddresses: ['203.0.113.10'], icePortRange: [40000, 40999]}),
803
+ whip: buildWHIPOptions({sfu: 'sfu', authorize: ({token}): boolean => token === process.env.WHIP_TOKEN}),
804
+ whep: buildWHEPOptions({maxSessions: 500}),
805
+ entrypoint: buildEntrypoints({
806
+ controllers: [WHIPController, WHEPController],
807
+ http: expressHTTPEntrypoint({port: 8080, body: {text: ['application/sdp', 'application/trickle-ice-sdpfrag']}})
808
+ })
809
+ },
810
+ bootstrap: ['sfu', 'entrypoint']
811
+ }))
812
+ ```
813
+
814
+ A browser plays it with its own `RTCPeerConnection`, its offer carrying all its candidates:
815
+
816
+ ```typescript
817
+ declare const player: HTMLVideoElement
818
+ const connection: RTCPeerConnection = new RTCPeerConnection()
819
+ const stream: MediaStream = new MediaStream()
820
+ player.srcObject = stream
821
+ connection.addEventListener('track', (event: RTCTrackEvent): void => stream.addTrack(event.track))
822
+ connection.addTransceiver('audio', {direction: 'recvonly'})
823
+ connection.addTransceiver('video', {direction: 'recvonly'})
824
+ await connection.setLocalDescription()
825
+ await new Promise<void>((resolve: () => void): void => {
826
+ if (connection.iceGatheringState === 'complete') return resolve()
827
+ connection.addEventListener('icegatheringstatechange', (): void => {
828
+ if (connection.iceGatheringState === 'complete') resolve()
829
+ })
830
+ })
831
+ const response: Response = await fetch('https://live.example.com/whep/concert', {method: 'POST', headers: {'Content-Type': 'application/sdp'}, body: connection.localDescription!.sdp})
832
+ await connection.setRemoteDescription({type: 'answer', sdp: await response.text()})
833
+ ```
834
+
835
+ | Request | Answer |
836
+ |---|---|
837
+ | `POST /whep/:stream` with the offer (`application/sdp`) | `201 Created`: the answer, `Location` (the session), `Link` (the ICE servers), `ETag` (its ICE session) |
838
+ | `DELETE /whep/:stream/:session` | `200`: the session ends |
839
+ | `PATCH /whep/:stream/:session` with a fragment of SDP (`application/trickle-ice-sdpfrag`) | As for WHIP: `204` for the candidates of the viewer, `200` with the fragment of the answer for an ICE restart, `412` for an `If-Match` other than `*` and the `ETag` |
840
+ | `OPTIONS` | `204`, with the CORS headers and `Accept-Post: application/sdp` |
841
+ | `GET` | `204` |
842
+
843
+ The refusals: 401 with `WWW-Authenticate: Bearer` when `authorize` refuses, 415 for a body of another type, 400 for an
844
+ offer without audio or video or a room the media server does not serve, 503 once there are `maxSessions` sessions, 404
845
+ for an unknown session.
846
+
847
+ | Option | Default | Meaning |
848
+ |---|---|---|
849
+ | `sfu` | `'sfu'` | The name of the `SFU` component whose rooms are played |
850
+ | `roomOf` | the stream | `({stream, token}, context) => string`: the room a stream comes from |
851
+ | `select` | the newest stream of WHIP, else the first member publishing | `({stream, token}, tracks) => string \| undefined`: the id of the member a viewer receives |
852
+ | `authorize` | none | `({stream, token}, context) => boolean`: who may play which stream |
853
+ | `maxSessions` | `100` | The sessions at once on this node, `0` for no limit |
854
+ | `allowOrigin` | `'*'` | The origins the browsers may play from |
855
+ | `turn`, `iceServers` | | The ICE servers given to the viewers in the `Link` headers |
856
+ | `cluster`, `turnSecret`, `selectTURN` | | Several nodes (section 13) |
857
+
858
+ A viewer receives one member: by default the newest stream published into the room with WHIP, else the first member
859
+ publishing audio or video. `select` chooses another one; it is asked again each time the tracks of the room change,
860
+ and when it throws, the viewer receives the default one. When the member stops publishing, the viewer goes on with the
861
+ next one on the same connection: its picture stops for a moment and starts again at a key frame. The offer of a viewer
862
+ fixes what it receives, one audio and one video, and its connection negotiates every codec of the offer that the
863
+ server has: a stream published later in any of them is sent too.
864
+
865
+ The component emits `play` (the session: its `id`, `stream`, `room`, `viewerId` and `source`, the member it receives)
866
+ once it answers, and `stop` when the session ends: the viewer deleted it, its connection failed or closed, or the
867
+ application stopped. `whep.sessions` lists the sessions of this node, and `sfu.rooms()` counts the viewers of each
868
+ room.
869
+
870
+ ## 12. Deployment
871
+
872
+ - **Ports.** Open port 3478 over UDP and TCP, the relay ports (`relayPortRange`) over UDP, and the TLS port when TURN
873
+ over TLS is on. In a container, publish them all; the relay range keeps the list short.
874
+ - **The public address.** `publicAddress` is the address the clients reach the TURN server at: the public address of
875
+ the host, or that of its NAT when the NAT forwards the ports. On a local network the default is enough.
876
+ - **The applications as members.** A server behind a NAT that forwards `icePortRange` with the same port numbers gives
877
+ its public address in `iceAdditionalHostAddresses`; otherwise it relies on the TURN server like any client.
878
+ - **The media server, WHIP and WHEP.** The clients connect to them directly: open their `icePortRange` over UDP, and
879
+ give them in `iceAdditionalHostAddresses` the public address of a NAT forwarding these ports with the same numbers.
880
+ A client that cannot reach them goes through the TURN server.
881
+ - **Several instances.** Give them a `Cluster` (section 13), and their TURN servers the same `secret`, so each accepts
882
+ the credentials made by the others.
883
+ - **Another TURN server.** coturn (`use-auth-secret`, `static-auth-secret` set to the same secret) accepts the
884
+ credentials of `createTURNCredentials()`; list it in the `iceServers` option of `Signaling`.
885
+ - **HTTPS.** Browsers allow WebRTC on any page, but the camera and the microphone only on secure pages (HTTPS or
886
+ `localhost`).
887
+ - **The UDP buffers.** The connections and the TURN server ask for a receive buffer of 4 MiB for their UDP sockets:
888
+ a smaller one loses packets in a burst, or when a large transfer arrives faster than the process reads it. Linux
889
+ caps the size at `net.core.rmem_max`, which is 208 KiB on most systems, so raise it on a Linux server
890
+ (`sysctl -w net.core.rmem_max=4194304`, on the nodes for the pods on the network of their node).
891
+
892
+ ## 13. Several instances
893
+
894
+ Several instances of an application (the pods of a deployment, the processes behind a load balancer) work together
895
+ through a `Cluster`: each instance is a node of the cluster, and the nodes talk through NATS or Redis. The components
896
+ given its name work across the nodes:
897
+
898
+ - the signaling puts the members connected to all the nodes in the same rooms;
899
+ - the TURN servers tell the other nodes their address and their load, and each member gets the least loaded ones;
900
+ - the media servers share the rooms: each member, stream of WHIP and viewer of WHEP connects to the media server of one
901
+ node, and the media servers of a room forward its tracks to each other;
902
+ - a session of WHIP or WHEP is ended or patched whichever node its DELETE or its PATCH reaches.
903
+
904
+ The bus carries only these messages. The audio, the video and the data go directly between the clients and the nodes,
905
+ between the media servers of the nodes, or through the TURN servers.
906
+
907
+ ```typescript
908
+ import {Application} from 'lakutata'
909
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
910
+ import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
911
+ import {buildClusterOptions, buildSignalingOptions, SignalingController} from '@lakutata/webrtc'
912
+
913
+ Application.run(() => ({
914
+ id: 'meet.app',
915
+ name: 'Meet',
916
+ components: {
917
+ cluster: buildClusterOptions({transport: {type: 'nats', servers: process.env.NATS_URL}}),
918
+ signaling: buildSignalingOptions({cluster: 'cluster', turnSecret: process.env.TURN_SECRET}),
919
+ entrypoint: buildEntrypoints({
920
+ controllers: [SignalingController],
921
+ service: socketIOServiceEntrypoint({port: 3000})
922
+ })
923
+ },
924
+ bootstrap: ['signaling', 'entrypoint']
925
+ }))
926
+ ```
927
+
928
+ | Option of `buildClusterOptions` | Default | |
929
+ |---|---|---|
930
+ | `transport` | (required) | `{type: 'nats', servers, user, pass, token}`; `{type: 'redis', url}` or `{type: 'redis', host, port, username, password, database, tls}`; `{type: 'memory'}` for the applications of one process (tests) |
931
+ | `prefix` | `'lakutata.webrtc'` | The prefix of the subjects or channels: the clusters sharing a server stay separate with it |
932
+ | `nodeId` | the host name and a random suffix | The id of the node; in Kubernetes the host name is the name of the pod |
933
+ | `labels` | `{}` | What the other nodes see of this one (a zone, a region), to choose the TURN servers |
934
+ | `heartbeatInterval` | `2000` | How often the node tells the others it is there, in milliseconds |
935
+ | `requestTimeout` | `5000` | How long `request()` and `requestAll()` wait, in milliseconds |
936
+
937
+ NATS needs `npm install nats`, and its core subjects are enough (no JetStream); Redis needs `npm install ioredis`, and
938
+ only its channels are used (`PUBLISH`, `SUBSCRIBE`). The component connects when the application starts, and fails
939
+ to start when the server cannot be reached then (in Kubernetes, the pod restarts until it can); afterwards it
940
+ reconnects by itself. The other nodes take a node as gone after three heartbeat intervals without news of it, or at
941
+ once when it stops.
942
+
943
+ ### The rooms across the nodes
944
+
945
+ Each node tells the others who joins and leaves through it, and gets all the members of a node when that node comes up
946
+ or when some of its news was lost (the bus restarting, for example). A signal to a member of another node goes through
947
+ that node.
948
+
949
+ - When a node stops, or the others cannot reach it any more, its members leave their rooms as `disconnected`: their
950
+ connections stay up while their clients reconnect to another node and join again.
951
+ - A client joining again through another node before the first one saw it go is moved there: the others see it join
952
+ again, not leave.
953
+ - `maxPeers` counts the members of all the nodes. Each room falls to one node (by hashing over the nodes), which admits
954
+ its members one at a time, so two members joining at once through different nodes cannot both take the last place.
955
+ When a node joins or leaves the cluster, some rooms fall to another node, and a member joining at that very moment
956
+ may be admitted over the limit.
957
+ - Socket.IO behind a load balancer: its long polling sends the requests of a client one by one, and they must all reach
958
+ the same instance. Connect the clients with `transports: ['websocket']` (a single connection, so no sticky sessions),
959
+ or turn on sticky sessions in the load balancer.
960
+
961
+ ### The TURN servers of the cluster
962
+
963
+ A TURN server joins the cluster with its `cluster` option and the `secret` shared by all of them (it refuses to start
964
+ without one); it tells the others its address, its ports and its allocations every second. The `Signaling` and `WHIP`
965
+ components given `turnSecret` give each client the two least loaded TURN servers that are not full, after the one of
966
+ their own node (`turn`). `selectTURN` chooses otherwise, for example by the labels of the nodes:
967
+
968
+ ```typescript
969
+ import {buildSignalingOptions, type ClusterTURNServer} from '@lakutata/webrtc'
970
+
971
+ //The members whose id starts with eu- get the TURN servers of the nodes labeled {region: 'eu'}, and so on
972
+ const signaling = buildSignalingOptions({
973
+ cluster: 'cluster',
974
+ turnSecret: process.env.TURN_SECRET,
975
+ selectTURN: (servers: ClusterTURNServer[], user: string): ClusterTURNServer[] => {
976
+ return servers.filter((server: ClusterTURNServer): boolean => user.startsWith(`${server.labels.region}-`))
977
+ }
978
+ })
979
+ ```
980
+
981
+ ### The media servers of the cluster
982
+
983
+ Given `cluster`, the media servers of the nodes share the rooms: the id of each one is `sfu:` followed by the id of its
984
+ node, and each member is connected to the media server of one node.
985
+
986
+ - A room stays on as few nodes as it can. Its first member goes to the node it joins through, or to the least loaded
987
+ node when that one is full; the next members go to the least loaded node the room already has, and to the least
988
+ loaded node of all once those are full. A load balancer spreads the clients over the nodes at random: without this
989
+ rule, a meeting of three could take three nodes.
990
+ - `selectSFU` chooses the node instead, the closest to the member for example; when the node it chooses is full or
991
+ does not answer, the default choice is made.
992
+ - A node with the signaling but no media server asks the least loaded node that has one, which makes the choice
993
+ (`selectSFU` runs there).
994
+ - `maxMembers` is the limit of each node: a member joining when no node takes it is refused
995
+ (`SignalingLimitException`). The streams of WHIP and the viewers of WHEP are given a node the same way (the viewers
996
+ are not counted by `maxMembers`: `maxSessions` of WHEP limits them).
997
+ - The events of each media server and its `rooms()` cover the members of its own node.
998
+
999
+ ```typescript
1000
+ import {buildSFUOptions, type SFUNode, type SFUPlacement} from '@lakutata/webrtc'
1001
+
1002
+ //The members whose id starts with eu- go to the least loaded node labeled {region: 'eu'}, and so on
1003
+ const sfu = buildSFUOptions({
1004
+ cluster: 'cluster',
1005
+ selectSFU: (placement: SFUPlacement, nodes: SFUNode[]): string | undefined => {
1006
+ const region: string = placement.peerId.split('-')[0]
1007
+ return nodes
1008
+ .filter((node: SFUNode): boolean => node.labels.region === region)
1009
+ .sort((a: SFUNode, b: SFUNode): number => a.members - b.members)[0]?.id
1010
+ }
1011
+ })
1012
+ ```
1013
+
1014
+ The media servers of a room forward its tracks to each other. A track goes once to each node where someone receives
1015
+ it (a member selected for it, a viewer, `sfu.track()`), however many receive it there, straight from the node of its
1016
+ publisher. The nodes connect to each other as the clients do, with WebRTC connections (DTLS-SRTP) over their
1017
+ `icePortRange`. A data channel between two members of different nodes goes through both media servers.
1018
+
1019
+ When a node goes down (it stops, or the others miss three of its heartbeats, about 6 seconds), its media server leaves
1020
+ its rooms. Its members join their rooms again, through the signaling, are given the media server of another node and
1021
+ publish their tracks again; the others see their tracks stop for a few seconds. The peers of a member that moved close
1022
+ and open again on every side, as when a member reloads its page: the listeners of `peer` create their channels again.
1023
+ When every node is full at that moment, the client does not give up as a first `join()` does: it tries again after 1
1024
+ second, then twice as long each time (10 seconds at most), its signaling emitting `error` after each failure, until a
1025
+ node takes the member or the room is left.
1026
+
1027
+ A session of WHIP or WHEP stays on the node that answered its POST, and its connection goes to the node chosen for it.
1028
+ Its PATCH and its DELETE may reach any node; a PATCH reaching a node without the session waits for the answers of all
1029
+ the nodes, 12 seconds at most. A viewer receives by default the stream of WHIP its node learned of last.
1030
+
1031
+ ### WHIP and WHEP behind a load balancer
1032
+
1033
+ A session of WHIP or WHEP lives on the node that answered its POST. Given `cluster`, the `WHIP` or `WHEP` component of
1034
+ another node receiving its DELETE or its PATCH asks the other nodes, and the one holding the session applies it.
1035
+ `maxSessions` counts the sessions of each node.
1036
+
1037
+ ### Kubernetes and k3s
1038
+
1039
+ The manifests below deploy NATS, the signaling (two pods), and a TURN server and a media server on each node. The
1040
+ application is one image for the three roles, chosen by the variable `ROLE`:
1041
+
1042
+ ```typescript
1043
+ import {Application, type ApplicationOptions} from 'lakutata'
1044
+ import {buildEntrypoints} from 'lakutata/com/entrypoint'
1045
+ import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
1046
+ import {buildClusterOptions, buildSFUOptions, buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
1047
+
1048
+ const cluster = buildClusterOptions({transport: {type: 'nats', servers: process.env.NATS_URL}})
1049
+ const signaling = buildSignalingOptions({cluster: 'cluster', turnSecret: process.env.TURN_SECRET})
1050
+ const roles: Record<string, ApplicationOptions['components']> = {
1051
+ signaling: {
1052
+ cluster: cluster,
1053
+ signaling: signaling,
1054
+ entrypoint: buildEntrypoints({
1055
+ controllers: [SignalingController],
1056
+ service: socketIOServiceEntrypoint({port: 3000})
1057
+ })
1058
+ },
1059
+ //On the network of the node: its address, and its ports 3478 and 49160-49200
1060
+ turn: {
1061
+ cluster: cluster,
1062
+ turn: buildTURNServerOptions({
1063
+ cluster: 'cluster',
1064
+ secret: process.env.TURN_SECRET,
1065
+ publicAddress: process.env.PUBLIC_ADDRESS,
1066
+ relayPortRange: [49160, 49200]
1067
+ })
1068
+ },
1069
+ //On the network of the node too, its ports 40000-40999; its signaling has no entrypoint: the members join through
1070
+ //the pods of the signaling, which give each of them the media server of a node
1071
+ sfu: {
1072
+ cluster: cluster,
1073
+ signaling: signaling,
1074
+ sfu: buildSFUOptions({cluster: 'cluster', icePortRange: [40000, 40999], workers: 2})
1075
+ }
1076
+ }
1077
+ const components: ApplicationOptions['components'] = roles[process.env.ROLE ?? 'signaling']
1078
+
1079
+ Application.run(() => ({
1080
+ id: 'meet.app',
1081
+ name: 'Meet',
1082
+ components: components,
1083
+ bootstrap: Object.keys(components ?? {})
1084
+ }))
1085
+ ```
1086
+
1087
+ ```yaml
1088
+ apiVersion: v1
1089
+ kind: Secret
1090
+ metadata:
1091
+ name: turn
1092
+ stringData:
1093
+ secret: change-me
1094
+ ---
1095
+ apiVersion: apps/v1
1096
+ kind: Deployment
1097
+ metadata:
1098
+ name: nats
1099
+ spec:
1100
+ replicas: 1
1101
+ selector:
1102
+ matchLabels:
1103
+ app: nats
1104
+ template:
1105
+ metadata:
1106
+ labels:
1107
+ app: nats
1108
+ spec:
1109
+ containers:
1110
+ - name: nats
1111
+ image: nats:latest
1112
+ ports:
1113
+ - containerPort: 4222
1114
+ ---
1115
+ apiVersion: v1
1116
+ kind: Service
1117
+ metadata:
1118
+ name: nats
1119
+ spec:
1120
+ selector:
1121
+ app: nats
1122
+ ports:
1123
+ - port: 4222
1124
+ ---
1125
+ apiVersion: apps/v1
1126
+ kind: Deployment
1127
+ metadata:
1128
+ name: signaling
1129
+ spec:
1130
+ replicas: 2
1131
+ selector:
1132
+ matchLabels:
1133
+ app: signaling
1134
+ template:
1135
+ metadata:
1136
+ labels:
1137
+ app: signaling
1138
+ spec:
1139
+ containers:
1140
+ - name: signaling
1141
+ image: registry.example.com/meet:1.0.0
1142
+ env:
1143
+ - name: ROLE
1144
+ value: signaling
1145
+ - name: NATS_URL
1146
+ value: nats://nats:4222
1147
+ - name: TURN_SECRET
1148
+ valueFrom:
1149
+ secretKeyRef:
1150
+ name: turn
1151
+ key: secret
1152
+ ports:
1153
+ - containerPort: 3000
1154
+ ---
1155
+ apiVersion: v1
1156
+ kind: Service
1157
+ metadata:
1158
+ name: signaling
1159
+ spec:
1160
+ selector:
1161
+ app: signaling
1162
+ ports:
1163
+ - port: 3000
1164
+ ---
1165
+ apiVersion: apps/v1
1166
+ kind: DaemonSet
1167
+ metadata:
1168
+ name: turn
1169
+ spec:
1170
+ selector:
1171
+ matchLabels:
1172
+ app: turn
1173
+ template:
1174
+ metadata:
1175
+ labels:
1176
+ app: turn
1177
+ spec:
1178
+ hostNetwork: true
1179
+ dnsPolicy: ClusterFirstWithHostNet
1180
+ containers:
1181
+ - name: turn
1182
+ image: registry.example.com/meet:1.0.0
1183
+ env:
1184
+ - name: ROLE
1185
+ value: turn
1186
+ - name: NATS_URL
1187
+ value: nats://nats:4222
1188
+ - name: PUBLIC_ADDRESS
1189
+ valueFrom:
1190
+ fieldRef:
1191
+ fieldPath: status.hostIP
1192
+ - name: TURN_SECRET
1193
+ valueFrom:
1194
+ secretKeyRef:
1195
+ name: turn
1196
+ key: secret
1197
+ ---
1198
+ apiVersion: apps/v1
1199
+ kind: DaemonSet
1200
+ metadata:
1201
+ name: sfu
1202
+ spec:
1203
+ selector:
1204
+ matchLabels:
1205
+ app: sfu
1206
+ template:
1207
+ metadata:
1208
+ labels:
1209
+ app: sfu
1210
+ spec:
1211
+ hostNetwork: true
1212
+ dnsPolicy: ClusterFirstWithHostNet
1213
+ containers:
1214
+ - name: sfu
1215
+ image: registry.example.com/meet:1.0.0
1216
+ env:
1217
+ - name: ROLE
1218
+ value: sfu
1219
+ - name: NATS_URL
1220
+ value: nats://nats:4222
1221
+ - name: TURN_SECRET
1222
+ valueFrom:
1223
+ secretKeyRef:
1224
+ name: turn
1225
+ key: secret
1226
+ ```
1227
+
1228
+ - The TURN servers and the media servers run on the network of their node (`hostNetwork`): the clients reach them at
1229
+ the address of the node, on its ports. Open UDP and TCP 3478, UDP 49160–49200 and UDP 40000–40999 in the firewall of
1230
+ the nodes; the media servers of the nodes also reach each other on 40000–40999.
1231
+ - `status.hostIP` is the address of the node inside the cluster; it is the right one when the clients reach the nodes
1232
+ at it (a local network, nodes with public addresses). Behind the NAT of a cloud, give `PUBLIC_ADDRESS` the public
1233
+ address of each node instead, the NAT forwarding the same ports, and give the media server that address in
1234
+ `iceAdditionalHostAddresses`.
1235
+ - A pod on the network of its node resolves the names of the services with `dnsPolicy: ClusterFirstWithHostNet`.
1236
+ - The browsers reach the signaling through an Ingress or a load balancer in front of the Service `signaling`, connecting
1237
+ with `transports: ['websocket']`.
1238
+ - A single NATS pod is enough for a test; in production, run a NATS cluster (its Helm chart) or a Redis with a replica.
1239
+ - When the whole cluster starts, a pod may start before the DNS of the cluster resolves `nats` (its log then shows
1240
+ `getaddrinfo EAI_AGAIN nats`): the application stops when it cannot reach NATS as it starts, and Kubernetes starts it
1241
+ again until it can. A few restarts of each pod at that moment are expected.
1242
+
1243
+ In the repository, `packages/webrtc/test-env/k3s/run.sh` deploys these manifests in k3s in Docker (two nodes, the code
1244
+ mounted instead of an image). It connects two members through two pods and the TURN servers only, then two members
1245
+ through the media servers of the two nodes, and deletes the pod of one of them: that member moves to the pod started
1246
+ in its place.
1247
+
1248
+ ### The cluster for the application
1249
+
1250
+ The application talks to its other instances through the same component: messages to all the other nodes, and requests
1251
+ to one node or to all of them.
1252
+
1253
+ ```typescript
1254
+ import {Component} from 'lakutata'
1255
+ import type {Cluster, ClusterReply} from '@lakutata/webrtc'
1256
+
1257
+ export class Viewers extends Component {
1258
+ //The viewers of each stream on this node, counted by the application
1259
+ protected readonly counts: Map<string, number> = new Map()
1260
+
1261
+ protected async init(): Promise<void> {
1262
+ const cluster: Cluster = await this.getObject<Cluster>('cluster')
1263
+ //Each node answers with its own count
1264
+ cluster.handle<{stream: string}, number>('viewers', ({stream}: {stream: string}): number => this.counts.get(stream) ?? 0)
1265
+ cluster.subscribe<{text: string}>('announcements', (message: {text: string}): void => console.log(message.text))
1266
+ }
1267
+
1268
+ //The viewers of a stream on all the nodes
1269
+ public async total(stream: string): Promise<number> {
1270
+ const cluster: Cluster = await this.getObject<Cluster>('cluster')
1271
+ const replies: ClusterReply<number>[] = await cluster.requestAll<number>('viewers', {stream: stream})
1272
+ return replies.reduce((sum: number, reply: ClusterReply<number>): number => sum + reply.result, this.counts.get(stream) ?? 0)
1273
+ }
1274
+ }
1275
+ ```
1276
+
1277
+ `cluster.nodes()` lists the nodes with their labels and what they announce (`announce()`), and the component emits
1278
+ `node-up`, `node-down` and `node-update`. The messages are sent as JSON; a message larger than the server carries (1 MiB
1279
+ by default on NATS) is sent in parts.
1280
+
1281
+ ## 14. The signaling protocol
1282
+
1283
+ Clients in other languages and native applications speak the protocol directly. The requests are service calls whose
1284
+ pattern is `{method}`: over Socket.IO, the event `message` with `{method: 'webrtc.join', room: 'room-1'}` and an
1285
+ acknowledgement (`{success, error, payload}`); over JSON-RPC 2.0, a request whose method is `webrtc.join`. The server
1286
+ pushes the events to the members: Socket.IO events, or JSON-RPC notifications whose `params` hold the payload.
1287
+
1288
+ | Request | Parameters | Answer |
1289
+ |---|---|---|
1290
+ | `webrtc.join` | `room`, `metadata`, `session` | `{v, peerId, peers, iceServers, credentialTtl, heartbeatInterval, sfu}` |
1291
+ | `webrtc.signal` | `room`, `to`, and `description` (`{type, sdp}`) or `candidate` (`{candidate, sdpMid, sdpMLineIndex, usernameFragment}`) | `{delivered}` |
1292
+ | `webrtc.leave` | `room` | `{}` |
1293
+ | `webrtc.iceServers` | — | `{iceServers, credentialTtl}`: new credentials |
1294
+ | `webrtc.heartbeat` | — | `{}` |
1295
+
1296
+ | Event | Payload |
1297
+ |---|---|
1298
+ | `webrtc:peer-joined` | `{room, peerId, metadata, session, role, sfu}`: a member joined, or joined again |
1299
+ | `webrtc:peer-left` | `{room, peerId, reason}`: `left` when it asked, `disconnected` when its client went away |
1300
+ | `webrtc:signal` | `{room, from, session, description, streams}` or `{room, from, session, candidate}` |
1301
+
1302
+ - `session` is a random string a client chooses once. A member joining again with the same session (after a
1303
+ reconnection of its signaling) keeps the connections of the others; a new session (a reloaded page) replaces them.
1304
+ - `heartbeatInterval` is 0 over Socket.IO and JSON-RPC over WebSocket: the server sees their connections close. Over
1305
+ another transport, the client sends `webrtc.heartbeat` that often; after three intervals without a request, it leaves
1306
+ its rooms.
1307
+ - The member whose id comes first (by string comparison) is the polite one of a pair: when both send an offer at the
1308
+ same time, it gives up its own and answers the other one.
1309
+ - The fields of the media server, all set by the server: `role` is `sfu` for a media server and `whip` for a stream
1310
+ of WHIP (none for the clients); `sfu` is the id of the media server a member is connected to, and in the answer to
1311
+ `webrtc.join`, the one the client connects to. A client in a room with a media server connects to it only, and is
1312
+ the polite one with it. The descriptions of the media server carry `streams`: for each stream it sends, its `id` on
1313
+ that connection, the `peerId` of the member it comes from, and its id on the connection of that member (`stream`).
1314
+ - On the connection to the media server, the data channel 1023 (label `webrtc.control`, created by both sides with
1315
+ `negotiated: true` and this `id`) carries `{type: 'pause' | 'resume', mid}` in JSON: the server stops or starts
1316
+ sending the track of that transceiver. A data channel to a member through the server has the id of that member as
1317
+ its `protocol`; the server opens the channel on the side of that member with the id of the sender. The media servers
1318
+ of the nodes use the channel 1023 between them too, and channels whose `protocol` is `["sender", "member"]` in JSON.
1319
+ - The calls of `webrtcServiceEntrypoint()` (section 5) are JSON-RPC 2.0 messages on a data channel labeled `service`,
1320
+ ordered and reliable, which the caller creates. Each message is cut into frames as `ChunkedChannel` does: a frame is
1321
+ a binary message whose first byte holds flags (1 for the last frame of a message, 2 for a text message) and whose
1322
+ other bytes are the next part of the message (16 KiB at most from this package). A call is a text message: its
1323
+ frames carry the flags 2, and the last one 3. The responses and the events (notifications whose `params` are the
1324
+ arguments) come back the same way.
1325
+ - The errors carry the `errno` of the server's exception: `E_SIGNALING_DENIED`, `E_SIGNALING_LIMIT` (a full room, or
1326
+ a member no media server takes), `E_SIGNALING_NOT_A_MEMBER`, `E_SIGNALING_CLIENT_REQUIRED`, `E_INVALID_VALUE`.
1327
+ JSON-RPC over HTTP cannot carry the signaling: the events need a connection to be pushed through.
1328
+
1329
+ ## 15. When a connection fails
1330
+
1331
+ `setDebugHandler()` gives the debug messages of ICE, DTLS, SCTP and TURN:
1332
+
1333
+ ```typescript
1334
+ import {format} from 'node:util'
1335
+ import {setDebugHandler} from '@lakutata/webrtc'
1336
+
1337
+ setDebugHandler((namespace: string, args: unknown[]): void => console.debug(namespace, format(...args)))
1338
+ ```
1339
+
1340
+ The worker threads of a media server send theirs to the same handler, each message starting with the number of its
1341
+ thread (`[thread 0] …`). A `select` of the media server or of WHEP that throws is reported under `webrtc/sfu` and
1342
+ `webrtc/whep`.
1343
+
1344
+ To check the TURN server, join a room from two clients with `iceTransportPolicy: 'relay'`: they connect only through
1345
+ the relays, and `turn.allocations` counts them. In Chrome, `chrome://webrtc-internals` shows the candidates and the
1346
+ pairs tried.
1347
+
1348
+ ## 16. Limits
1349
+
1350
+ - Node.js does not capture, encode or decode audio and video: its tracks carry RTP packets, from an encoder or from
1351
+ another connection.
1352
+ - The throughput of one data channel between two Node.js processes on an Apple M4 is about 80 MiB/s for large
1353
+ messages, and 626,000 messages of 100 bytes per second; in an arm64 Linux virtual machine (Docker Desktop) on the
1354
+ same Mac, about 43 MiB/s, and 548,000 messages a second. `packages/webrtc/test-env/bench/engine.mjs` in the
1355
+ repository measures it on another machine.
1356
+ - The media server does not transcode, and sends each video as it is published: there is no simulcast yet, so a
1357
+ member with little bandwidth gets `congestion` events rather than a lower quality. Its capacity on one machine is in
1358
+ section 9; `packages/webrtc/test-env/bench/forward.mjs` in the repository measures it on another.
1359
+ - The media servers of a cluster send a track straight from the node of its publisher to each node wanting it, which
1360
+ suits rooms spread over tens of nodes; beyond that, the nodes would have to forward through each other.
1361
+ - Verified in October 2026 with Chrome 154, Firefox 153 and WebKit 26.5 (the builds of Playwright 1.62), Safari
1362
+ 27.0.1 on macOS 27, Pion 4.2.25 (Go) and aiortc 1.15.0 (Python), on Node.js 24 (macOS, and Linux on Alpine), Bun
1363
+ 1.4.3 (macOS, and Linux on Debian) and the main process of Electron 44. The scripts are in `packages/webrtc/test-env`
1364
+ of the repository.