lakutata 3.1.0 → 3.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/README.zh-CN.md +1 -1
- package/doc/en/Command.md +3 -0
- package/doc/en/FFI.md +458 -0
- package/doc/en/HTTP.md +8 -4
- package/doc/en/NativeAddons.md +2 -1
- package/doc/en/Overview.md +5 -2
- package/doc/en/ServiceCalls.md +8 -4
- package/doc/en/UpgradingFrom2.md +4 -2
- package/doc/en/WebRTC.md +1364 -0
- package/doc/en/examples/Meeting.md +573 -0
- package/doc/en/examples/README.md +1 -0
- package/doc/zh/Command.md +2 -0
- package/doc/zh/FFI.md +337 -0
- package/doc/zh/HTTP.md +3 -2
- package/doc/zh/NativeAddons.md +1 -1
- package/doc/zh/Overview.md +3 -1
- package/doc/zh/ServiceCalls.md +5 -2
- package/doc/zh/UpgradingFrom2.md +1 -1
- package/doc/zh/WebRTC.md +1049 -0
- package/doc/zh/examples/Meeting.md +524 -0
- package/doc/zh/examples/README.md +1 -0
- package/package.json +14 -14
package/doc/en/WebRTC.md
ADDED
|
@@ -0,0 +1,1364 @@
|
|
|
1
|
+
# Lakutata WebRTC
|
|
2
|
+
|
|
3
|
+
WebRTC connects two endpoints directly, peer to peer: browsers, Electron applications and Node.js processes exchange
|
|
4
|
+
data, audio and video without a server relaying it, behind NATs and firewalls too. The package `@lakutata/webrtc`
|
|
5
|
+
provides everything around these connections: the signaling that introduces the endpoints to each other, a STUN and TURN
|
|
6
|
+
server for the networks that need one, a media server for the rooms of many members, the peer connections of Node.js,
|
|
7
|
+
and a client for the browsers. It is written in TypeScript without native code, and it is not part of `lakutata`:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm install lakutata @lakutata/webrtc
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
| Entry point | Runs in | What it gives |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| `@lakutata/webrtc` | Node.js, the main process of Electron, Bun | `Signaling`, `SignalingController`, `TURNServer`, `SignalingClient`, `PeerConnection`, `PeerThreads`, `MediaTrack`, `SFU`, `WHIP`, `WHIPController`, `WHEP`, `WHEPController`, `Cluster`, `webrtcServiceEntrypoint`, `PeerServiceProxy`, `ChunkedChannel` |
|
|
16
|
+
| `@lakutata/webrtc/client` | Browsers, the renderer of Electron | `SignalingConnection`, `PeerServiceProxy`, `ChunkedChannel`, the types of the protocol |
|
|
17
|
+
|
|
18
|
+
The browser entry uses only the APIs of the browsers; a frontend installs the same package, and the framework is not
|
|
19
|
+
installed with it.
|
|
20
|
+
|
|
21
|
+
## 1. How a connection is made
|
|
22
|
+
|
|
23
|
+
Two endpoints cannot connect until each knows how to reach the other. Each one describes itself in an **offer** or an
|
|
24
|
+
**answer** (its codecs, its keys, its data channels) and lists its **candidates**, the addresses it may be reached at:
|
|
25
|
+
its local addresses, its public address as a STUN server sees it, and an address on a TURN server that relays its
|
|
26
|
+
traffic when nothing else gets through. These messages travel through a server both endpoints are connected to, the
|
|
27
|
+
**signaling**; once the endpoints have them, they test the candidates in pairs (ICE), keep a working pair, and encrypt
|
|
28
|
+
the connection (DTLS). The data channels then carry messages in both directions, reliable and ordered or not, and the
|
|
29
|
+
tracks carry audio and video.
|
|
30
|
+
|
|
31
|
+
The signaling of this package works with **rooms**: a client joins a room, gets the members already there, and the
|
|
32
|
+
offers, answers and candidates are exchanged between each pair of members. The rest is done for you:
|
|
33
|
+
|
|
34
|
+
- The offers and the answers are exchanged as in the W3C "perfect negotiation": either member may add data channels and
|
|
35
|
+
tracks at any time, both at the same moment too.
|
|
36
|
+
- A connection that fails, or stays disconnected for 5 seconds, restarts ICE by itself.
|
|
37
|
+
- When the signaling connection drops, the peer-to-peer connections stay up, and the client joins its rooms again once
|
|
38
|
+
it reconnects.
|
|
39
|
+
- The members get the ICE servers (with TURN credentials) when they join, and new credentials before those expire.
|
|
40
|
+
|
|
41
|
+
## 2. The server
|
|
42
|
+
|
|
43
|
+
```typescript
|
|
44
|
+
import {Application} from 'lakutata'
|
|
45
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
46
|
+
import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
|
|
47
|
+
import {jsonRPCServiceEntrypoint} from 'lakutata/adapter/jsonrpc'
|
|
48
|
+
import {buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
|
|
49
|
+
|
|
50
|
+
Application.run(() => ({
|
|
51
|
+
id: 'meet.app',
|
|
52
|
+
name: 'Meet',
|
|
53
|
+
components: {
|
|
54
|
+
turn: buildTURNServerOptions({publicAddress: '203.0.113.10', relayPortRange: [49160, 49200], secret: process.env.TURN_SECRET}),
|
|
55
|
+
signaling: buildSignalingOptions({
|
|
56
|
+
turn: 'turn',
|
|
57
|
+
//Only the members whose id starts with board- join the room board
|
|
58
|
+
authorize: (peer, room): boolean => room !== 'board' || peer.peerId.startsWith('board-')
|
|
59
|
+
}),
|
|
60
|
+
//The browsers over Socket.IO on port 3000, the other clients over JSON-RPC (WebSocket) on port 3001
|
|
61
|
+
entrypoint: buildEntrypoints({
|
|
62
|
+
controllers: [SignalingController],
|
|
63
|
+
service: [socketIOServiceEntrypoint({port: 3000}), jsonRPCServiceEntrypoint({transport: 'websocket', port: 3001})]
|
|
64
|
+
})
|
|
65
|
+
},
|
|
66
|
+
bootstrap: ['turn', 'entrypoint']
|
|
67
|
+
}))
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`SignalingController` serves the requests of the signaling; it finds the component under the name `signaling`, which
|
|
71
|
+
finds the entrypoint under the name `entrypoint` (its `entrypoint` option). Its actions go through the middlewares of
|
|
72
|
+
the entrypoint like any other action. The member's id is the client id of its connection: the `auth.id` of Socket.IO
|
|
73
|
+
(its `id` header otherwise), the `id` header or query parameter of a WebSocket. Authenticate the requests with a
|
|
74
|
+
middleware, and decide who joins which room with `authorize`:
|
|
75
|
+
`(peer, room, context) => boolean | Promise<boolean>`, given the member's id and metadata and the context of its
|
|
76
|
+
request.
|
|
77
|
+
|
|
78
|
+
The options of `buildSignalingOptions`:
|
|
79
|
+
|
|
80
|
+
| Option | Default | Meaning |
|
|
81
|
+
|---|---|---|
|
|
82
|
+
| `entrypoint` | `'entrypoint'` | The entrypoint component pushing the events to the members |
|
|
83
|
+
| `turn` | — | A `TURNServer` component: the members get its ICE servers, with credentials |
|
|
84
|
+
| `iceServers` | `[]` | More ICE servers (another STUN or TURN server), after those of `turn` |
|
|
85
|
+
| `authorize` | — | Who joins which room; everyone joins any room without it |
|
|
86
|
+
| `maxPeers` | `0` | The members of a room at most; 0 for no limit |
|
|
87
|
+
| `maxMessageSize` | `65536` | The size of a signal and of a member's metadata at most, in bytes |
|
|
88
|
+
| `maxMessagesPerSecond` | `50` | The signals of a member per second on average (bursts of twice as many) |
|
|
89
|
+
| `heartbeatInterval` | `10000` | How often a client sends a heartbeat when the server cannot see its disconnection, in milliseconds |
|
|
90
|
+
|
|
91
|
+
The component also works without the controller: `join()`, `signal()`, `leave()` and `disconnect()` change the rooms
|
|
92
|
+
from the application's own code, and `rooms()` lists them. A member added by the application gets its events through
|
|
93
|
+
the `receive` option of `join()`. The ids starting with `sfu:` and `whip:` are kept for the members the server adds
|
|
94
|
+
itself (the media server, the streams of WHIP): a client joining with one is refused.
|
|
95
|
+
|
|
96
|
+
## 3. The STUN and TURN server
|
|
97
|
+
|
|
98
|
+
Most connections succeed directly or with the public address a STUN server reports. Some networks (symmetric NATs,
|
|
99
|
+
corporate firewalls, mobile carriers) let nothing through but a relay: a TURN server forwards the traffic of the
|
|
100
|
+
connection. `TURNServer` is one, inside the application:
|
|
101
|
+
|
|
102
|
+
| Option | Default | Meaning |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| `host` | `'0.0.0.0'` | The address it listens on |
|
|
105
|
+
| `port` | `3478` | The port of STUN and TURN over UDP and TCP |
|
|
106
|
+
| `udp`, `tcp` | `true` | Whether to listen over UDP, over TCP |
|
|
107
|
+
| `tls` | — | `{cert, key, port}`: TURN over TLS (`turns:`, port 5349 by default), for the networks letting only TLS through |
|
|
108
|
+
| `publicAddress` | the first IPv4 address of the host that is not internal | The address the clients reach it at |
|
|
109
|
+
| `relayPortRange` | any free port | `[first, last]`: the UDP ports of the relays |
|
|
110
|
+
| `secret` | a random one at each start | The secret of the short-lived credentials |
|
|
111
|
+
| `users` | `{}` | Users with a fixed password, `{username: password}` |
|
|
112
|
+
| `realm` | `'lakutata'` | The realm of the credentials |
|
|
113
|
+
| `credentialTtl` | `86400` | How long the short-lived credentials are valid, in seconds; a relay made with them keeps working once they expire |
|
|
114
|
+
| `maxAllocations` | no limit | The relays at once at most |
|
|
115
|
+
| `workers` | `0` | Worker threads serving the clients, to use more cores (on Linux; see below) |
|
|
116
|
+
|
|
117
|
+
The short-lived credentials follow the REST API of coturn: the username is `<expiry>:<user>` and the password is
|
|
118
|
+
`base64(HMAC-SHA1(secret, username))`, so any server holding the secret checks them without storing them.
|
|
119
|
+
`turn.iceServers(user)` gives the ICE servers of a client with credentials, `turn.credentials(user)` the credentials
|
|
120
|
+
alone, `createTURNCredentials(secret, user, ttl)` makes them for another TURN server (a coturn configured with
|
|
121
|
+
`use-auth-secret`), and `turn.allocations` counts the relays in use.
|
|
122
|
+
|
|
123
|
+
### Several cores
|
|
124
|
+
|
|
125
|
+
The server relays in the main thread by default. Measured on an Apple M4 with Node.js 24, it relayed a data channel
|
|
126
|
+
whose two ends both relayed through it (so it carried each packet twice) at about 38 MiB/s for messages of 16 KiB,
|
|
127
|
+
taking 27 s of CPU per GiB, and 365,000 messages of 100 bytes a second, at 2.8 µs of CPU each. A packet costs the same
|
|
128
|
+
with 500 idle allocations of other clients.
|
|
129
|
+
|
|
130
|
+
With `workers: 3`, three worker threads serve the clients. Their listeners share the ports of the server
|
|
131
|
+
(`SO_REUSEPORT`), and the system keeps each client on one thread, chosen from its address and its port; that thread
|
|
132
|
+
relays its traffic. The threads count their allocations together against `maxAllocations`, and `relayPortRange` is
|
|
133
|
+
shared out among them. Only Linux lets the threads share the ports: elsewhere, the server relays in the main thread and
|
|
134
|
+
says so in the debug messages (`webrtc/turn`). A thread that stops (only a bug stops one) loses its allocations and is
|
|
135
|
+
started again. Meanwhile the system spreads the clients over UDP among the other threads, then among all of them again,
|
|
136
|
+
so most of them find their allocation gone, as when a TURN server restarts; the connections over TCP and TLS of the
|
|
137
|
+
other threads stay.
|
|
138
|
+
|
|
139
|
+
In an arm64 Linux virtual machine (Docker Desktop, 10 cores) on the same Mac, with three connections relaying through it
|
|
140
|
+
at once (six clients):
|
|
141
|
+
|
|
142
|
+
| Server | Messages of 100 bytes relayed | CPU per message |
|
|
143
|
+
|---|---|---|
|
|
144
|
+
| The main thread | 460,000 to 550,000 a second | 1.9 to 2.3 µs |
|
|
145
|
+
| 3 worker threads | 510,000 to 820,000 a second in five runs of six, 220,000 in the other | 2.4 to 3.8 µs, 9 µs in the other run |
|
|
146
|
+
|
|
147
|
+
The system shares out the clients by their address and port, not by their load: with six clients, a thread often had
|
|
148
|
+
none, or twice its share. Large messages were therefore relayed at 34 to 79 MiB/s depending on the run, against 46 to
|
|
149
|
+
50 MiB/s in the main thread; the more clients, the more evenly the threads share them. Each thread takes about 15 MiB.
|
|
150
|
+
In that virtual machine, a worker thread relayed a quarter to a third fewer messages per core than the main thread; on
|
|
151
|
+
macOS, the two cost the same.
|
|
152
|
+
|
|
153
|
+
## 4. Browsers
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
import {io} from 'socket.io-client'
|
|
157
|
+
import {type Peer, type Room, SignalingConnection} from '@lakutata/webrtc/client'
|
|
158
|
+
|
|
159
|
+
const signaling: SignalingConnection = new SignalingConnection(io('https://meet.example.com', {auth: {id: 'alice'}}))
|
|
160
|
+
const room: Room = await signaling.join('room-1', {metadata: {name: 'Alice'}})
|
|
161
|
+
room.on('peer', (peer: Peer): void => {
|
|
162
|
+
//Unordered and unreliable: a lost position is not sent again
|
|
163
|
+
const positions: RTCDataChannel = peer.createDataChannel('positions', {ordered: false, maxRetransmits: 0})
|
|
164
|
+
positions.addEventListener('open', (): void => positions.send(JSON.stringify({x: 1, y: 2})))
|
|
165
|
+
peer.on('datachannel', (channel: RTCDataChannel): void => {
|
|
166
|
+
channel.addEventListener('message', (event: MessageEvent): void => console.log(peer.id, channel.label, event.data))
|
|
167
|
+
})
|
|
168
|
+
peer.on('connectionstatechange', (state: string): void => console.log(peer.id, state))
|
|
169
|
+
})
|
|
170
|
+
room.on('peer-left', (peer: Peer): void => console.log(`${peer.id} left`))
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
`SignalingConnection` takes the connection of the application:
|
|
174
|
+
|
|
175
|
+
- a Socket.IO socket (`io()` of `socket.io-client`): the member's id is its `auth.id`, and Socket.IO reconnects by
|
|
176
|
+
itself;
|
|
177
|
+
- a `WebSocket` speaking JSON-RPC, the member's id being the `id` query parameter of its URL;
|
|
178
|
+
- a function creating that `WebSocket`, so the connection reconnects after a disconnection (after 1 second, then twice
|
|
179
|
+
as long each time, 30 seconds at most):
|
|
180
|
+
`new SignalingConnection(() => new WebSocket('wss://meet.example.com/rpc?id=alice'))`.
|
|
181
|
+
|
|
182
|
+
Its options: `iceTransportPolicy` (`'relay'` to use only the TURN relays), `configuration` (more options of the
|
|
183
|
+
`RTCPeerConnection`s), `createPeerConnection` (another implementation of `RTCPeerConnection`), `timeout` (of the
|
|
184
|
+
requests, 15 seconds by default).
|
|
185
|
+
|
|
186
|
+
A room emits `peer` for each other member: the members already in the room, then each newcomer. A listener added later
|
|
187
|
+
is called at once for the peers already there. In a room with a media server, `peer` is emitted once the connection
|
|
188
|
+
to the server has exchanged its first offer and answer, so a channel created in the listener does not collide with
|
|
189
|
+
that offer. `room.publish(track, stream)` sends a track to every member (section 8),
|
|
190
|
+
and `room.sfu` is the id of the media server of the room, when it has one (section 9). A `Peer` has the member's `id`,
|
|
191
|
+
its `metadata`, the `connection` to it (an `RTCPeerConnection`: read its statistics), `createDataChannel()`, `pause()`
|
|
192
|
+
and `resume()` (a track of the member), `restartIce()` and `close()`, and emits `datachannel`, `track`,
|
|
193
|
+
`connectionstatechange`, `metadata` (the member joined again with new metadata), `error` and `close`. A room emits
|
|
194
|
+
`peer-left` when a peer closes: its member left, or its connection was lost for good.
|
|
195
|
+
|
|
196
|
+
Both members may create channels: each side gets the channels of the other one through `datachannel`. For a single
|
|
197
|
+
channel shared by both, create it on both sides with `negotiated: true` and the same `id`.
|
|
198
|
+
|
|
199
|
+
In Electron, the renderer uses `@lakutata/webrtc/client` and connects to the signaling itself; the main process uses
|
|
200
|
+
`@lakutata/webrtc`.
|
|
201
|
+
|
|
202
|
+
## 5. An application as a member
|
|
203
|
+
|
|
204
|
+
`SignalingClient` joins the rooms from a lakutata application, with the peer connections of Node.js. A device, a
|
|
205
|
+
recorder, a bot or a test joins the rooms as the browsers do:
|
|
206
|
+
|
|
207
|
+
```typescript
|
|
208
|
+
import {Component} from 'lakutata'
|
|
209
|
+
import {type Peer, type PeerChannel, type Room, SignalingClient} from '@lakutata/webrtc'
|
|
210
|
+
|
|
211
|
+
export class Telemetry extends Component {
|
|
212
|
+
protected async init(): Promise<void> {
|
|
213
|
+
const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'device-7'})
|
|
214
|
+
const room: Room = await signaling.join('device-7')
|
|
215
|
+
room.on('peer', (peer: Peer): void => {
|
|
216
|
+
const channel: PeerChannel = peer.createDataChannel('readings')
|
|
217
|
+
const timer: NodeJS.Timeout = setInterval((): void => {
|
|
218
|
+
if (channel.readyState === 'open') channel.send(JSON.stringify({temperature: 21.5}))
|
|
219
|
+
}, 1000)
|
|
220
|
+
peer.on('close', (): void => clearInterval(timer))
|
|
221
|
+
})
|
|
222
|
+
}
|
|
223
|
+
}
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
An `http:` or `https:` URL uses Socket.IO (install `socket.io-client`), a `ws:` or `wss:` URL uses JSON-RPC over the
|
|
227
|
+
`WebSocket` of Node.js 22 and Bun. Its options: `url`, `id` (a random UUID by default), `path` (of Socket.IO), `auth`
|
|
228
|
+
(more fields of the Socket.IO handshake, a token), `timeout`, and the options of its connections (`iceTransportPolicy`,
|
|
229
|
+
`icePortRange`, `iceAdditionalHostAddresses`, `codecs`, `threads`). It is `@Transient`: each `getObject()` gives a new
|
|
230
|
+
client, and `close()` leaves its rooms and disconnects.
|
|
231
|
+
|
|
232
|
+
### Calling the actions of an application
|
|
233
|
+
|
|
234
|
+
`webrtcServiceEntrypoint()` lets the members of the rooms an application joins call its service actions
|
|
235
|
+
(`@ServiceAction`) over their peer-to-peer connection to it: a browser on another network calls the actions of a device
|
|
236
|
+
directly, and the calls do not go through a server. It is an adapter of the entrypoint's `service` channel, like those
|
|
237
|
+
of [Service Calls](ServiceCalls.md): the actions keep their input validation, access control and middlewares, and the
|
|
238
|
+
same actions can be served over Socket.IO or JSON-RPC at the same time.
|
|
239
|
+
|
|
240
|
+
```typescript
|
|
241
|
+
import {Application} from 'lakutata'
|
|
242
|
+
import {buildEntrypoints, Controller} from 'lakutata/com/entrypoint'
|
|
243
|
+
import {ServiceAction} from 'lakutata/decorator/ctrl'
|
|
244
|
+
import {webrtcServiceEntrypoint} from '@lakutata/webrtc'
|
|
245
|
+
|
|
246
|
+
class DoorController extends Controller {
|
|
247
|
+
@ServiceAction({method: 'door.open'})
|
|
248
|
+
public async open(input: {door: string}): Promise<string> {
|
|
249
|
+
return `${input.door} is open`
|
|
250
|
+
}
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
Application.run({
|
|
254
|
+
id: 'door.device',
|
|
255
|
+
name: 'Door',
|
|
256
|
+
components: {
|
|
257
|
+
entrypoint: buildEntrypoints({
|
|
258
|
+
controllers: [DoorController],
|
|
259
|
+
//The device joins the room door-7 as the member door-7
|
|
260
|
+
service: webrtcServiceEntrypoint({
|
|
261
|
+
signaling: {url: 'https://meet.example.com', id: 'door-7'},
|
|
262
|
+
rooms: ['door-7']
|
|
263
|
+
})
|
|
264
|
+
})
|
|
265
|
+
},
|
|
266
|
+
bootstrap: ['entrypoint']
|
|
267
|
+
})
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
A member of the room creates a data channel labeled `service` to the device, and calls through it with
|
|
271
|
+
`PeerServiceProxy`, which both entry points of the package export:
|
|
272
|
+
|
|
273
|
+
```typescript
|
|
274
|
+
import {PeerServiceProxy, type Peer, type Room} from '@lakutata/webrtc/client'
|
|
275
|
+
|
|
276
|
+
declare const room: Room
|
|
277
|
+
room.on('peer', (peer: Peer): void => {
|
|
278
|
+
if (peer.id !== 'door-7') return
|
|
279
|
+
const door: PeerServiceProxy = new PeerServiceProxy(peer.createDataChannel('service'))
|
|
280
|
+
door.on('opened', (by: string): void => console.log(`opened by ${by}`))
|
|
281
|
+
door.invoke({method: 'door.open', door: 'front'}).then((result: string): void => console.log(result))
|
|
282
|
+
})
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
- `invoke(input, timeout)` sends the input's `method` as the method of a JSON-RPC request and the other fields as its
|
|
286
|
+
parameters (to use another field, give both sides the same `methodField`), and resolves to the result. It fails with a
|
|
287
|
+
`PeerServiceError`: its `errno` is the one of the action's error (`E_ACCESS_DENY`, `E_INVALID_VALUE`…), its `code`
|
|
288
|
+
the JSON-RPC code, its `data` the whole error object; or its `errno` is `E_PEER_SERVICE_TIMEOUT` (10 minutes by
|
|
289
|
+
default) or `E_PEER_SERVICE_CLOSED`. A request over 10 MiB as JSON (the proxy's `maxRequestSize`, the same as the
|
|
290
|
+
adapter's `limit`) is not sent and fails at once with `E_PEER_SERVICE_TOO_LARGE`. The calls made before the channel
|
|
291
|
+
opens wait for it.
|
|
292
|
+
- `this.context.clientId` is the member id of the caller. `webrtcRequestOfContext(this.context)` gives its `peer` (with
|
|
293
|
+
its metadata), the `room` it called through and the `channel`. `entrypoint.notify(memberId, event, ...args)` pushes an
|
|
294
|
+
event to the channels of that member, and `broadcast()` to every channel; `on`, `once` and `off` of the proxy receive
|
|
295
|
+
them.
|
|
296
|
+
- The rooms are joined once the application has launched, so the signaling server may run in the same application. A
|
|
297
|
+
room that cannot be joined is reported to `onError` (the application's log by default) and tried again after 1
|
|
298
|
+
second, then after twice as long each time, up to 10 seconds; a room the signaling refuses is not tried again.
|
|
299
|
+
- When a channel closes (its member left), its requests in progress are aborted: `this.signal` of the action fires.
|
|
300
|
+
When the application stops, the new requests fail with the code `-32001`, the requests in progress are answered, and
|
|
301
|
+
then the channels are closed.
|
|
302
|
+
- The adapter serves every service action of its entrypoint, as the other service adapters do, except those of the
|
|
303
|
+
signaling: give the actions meant for the members an entrypoint of their own, or limit them with rules or
|
|
304
|
+
middlewares. The channels with another label are left to the application.
|
|
305
|
+
- In a room with a media server, the channel goes through the server, which sees the messages; for calls that only the
|
|
306
|
+
two members should see, use a room without one.
|
|
307
|
+
|
|
308
|
+
The options of `webrtcServiceEntrypoint()`:
|
|
309
|
+
|
|
310
|
+
| Option | Default | Meaning |
|
|
311
|
+
|---|---|---|
|
|
312
|
+
| `signaling` | — | The options of a `SignalingClient` the adapter creates, and closes when it stops |
|
|
313
|
+
| `client` | — | Instead of `signaling`, a function returning the client of the application, when the application uses it too (two clients with the same id would replace each other in the rooms); the adapter does not leave its rooms or close it |
|
|
314
|
+
| `rooms` | — | The rooms whose members may call; a room the client has already joined is served as it is |
|
|
315
|
+
| `metadata` | — | The metadata of the application in the rooms it joins |
|
|
316
|
+
| `label` | `'service'` | The label of the channels carrying the calls |
|
|
317
|
+
| `limit` | 10 MiB | The largest request; a larger one is answered with the code `-32600` |
|
|
318
|
+
| `methodField`, `toPattern`, `positionalParams` | `'method'` | How a request maps to an action, as with `jsonRPCServiceEntrypoint()` |
|
|
319
|
+
| `onError` | the application's log | Called with the error when a room cannot be joined |
|
|
320
|
+
|
|
321
|
+
The member id is only as reliable as the signaling server makes it. The server sends the signals of each member under
|
|
322
|
+
the id it knows that member by, so a member cannot pass for another one; but the id is the client id of the member's
|
|
323
|
+
connection to the server, which the client gives, and the metadata is what the member gave. Before an action relies on
|
|
324
|
+
`clientId` to decide who may call it, make the signaling server check its clients: authenticate them with a middleware
|
|
325
|
+
and decide who joins which room with `authorize` (section 2).
|
|
326
|
+
|
|
327
|
+
## 6. Peer connections without the signaling
|
|
328
|
+
|
|
329
|
+
`PeerConnection` has the interface of `RTCPeerConnection`, so the code of a browser works in Node.js, and any signaling
|
|
330
|
+
works: an HTTP request, a QR code, a message queue. Its events are given in both styles: `addEventListener` and the
|
|
331
|
+
`on…` properties with event objects, or `on()` with the values (`on('datachannel', (channel) => …)`).
|
|
332
|
+
|
|
333
|
+
```typescript
|
|
334
|
+
import {Component} from 'lakutata'
|
|
335
|
+
import {PeerConnection, type PeerChannel, type SignalingCandidate, type SignalingDescription} from '@lakutata/webrtc'
|
|
336
|
+
|
|
337
|
+
export class Echo extends Component {
|
|
338
|
+
//Answer an offer received through an HTTP request, with all the candidates in the answer
|
|
339
|
+
public async answer(offer: SignalingDescription): Promise<SignalingDescription | null> {
|
|
340
|
+
const connection: PeerConnection = await this.getObject(PeerConnection, {iceServers: [{urls: 'stun:stun.example.com:3478'}]})
|
|
341
|
+
connection.on('datachannel', (channel: PeerChannel): void => {
|
|
342
|
+
channel.on('message', (data: string | Buffer | ArrayBuffer): void => channel.send(data.toString()))
|
|
343
|
+
})
|
|
344
|
+
await connection.setRemoteDescription(offer)
|
|
345
|
+
await connection.setLocalDescription()
|
|
346
|
+
await new Promise<void>((resolve: () => void): void => {
|
|
347
|
+
connection.on('icecandidate', (candidate: SignalingCandidate | null): void => {
|
|
348
|
+
if (!candidate) resolve()
|
|
349
|
+
})
|
|
350
|
+
})
|
|
351
|
+
return connection.localDescription
|
|
352
|
+
}
|
|
353
|
+
}
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Its options: `iceServers`, `iceTransportPolicy`, and those of a server: `icePortRange` (the UDP ports of its candidates,
|
|
357
|
+
to open in a firewall) and `iceAdditionalHostAddresses` (more addresses offered as candidates, such as the public
|
|
358
|
+
address of a NAT forwarding these ports with the same numbers); `codecs` chooses the audio and video codecs.
|
|
359
|
+
`headerExtensions: true` adds the RTP header extensions a media server uses: each side then estimates, from the feedback
|
|
360
|
+
of the other one, the bandwidth available for what it sends (without them, a browser goes by the packets lost only,
|
|
361
|
+
starting low and rising slowly). A `PeerChannel` sends strings and binary data; a binary message arrives as a `Buffer`,
|
|
362
|
+
or as an `ArrayBuffer` when `binaryType` is `'arraybuffer'`. `threads` serves it in a worker thread (see below).
|
|
363
|
+
|
|
364
|
+
### Several cores
|
|
365
|
+
|
|
366
|
+
A connection runs its ICE, DTLS, SCTP and SRTP in the main thread by default. Given the name of a `PeerThreads`
|
|
367
|
+
component (its `threads` option, which `SignalingClient` has too, for the connections to the other members), it runs
|
|
368
|
+
them in one of the worker threads of that component, while its object, its events and the messages of its channels stay
|
|
369
|
+
in the main thread: the code using it is the same. Its methods are carried to the thread and resolve once done there;
|
|
370
|
+
its states are the ones the thread last reported, with its events and with the reply to each method. The packets of the
|
|
371
|
+
tracks it receives come to the main thread while they are read or sent on, and the packets of the tracks it sends go to
|
|
372
|
+
the thread.
|
|
373
|
+
|
|
374
|
+
```typescript
|
|
375
|
+
import {Application} from 'lakutata'
|
|
376
|
+
import {buildPeerThreadsOptions, type PeerChannel, PeerConnection} from '@lakutata/webrtc'
|
|
377
|
+
|
|
378
|
+
Application
|
|
379
|
+
.run(() => ({
|
|
380
|
+
id: 'gateway.app',
|
|
381
|
+
name: 'Gateway',
|
|
382
|
+
components: {
|
|
383
|
+
//Four threads at most, started when needed
|
|
384
|
+
peerThreads: buildPeerThreadsOptions({threads: 4})
|
|
385
|
+
}
|
|
386
|
+
}))
|
|
387
|
+
.onLaunched(async (app: Application): Promise<void> => {
|
|
388
|
+
//Its engine runs in one of the threads
|
|
389
|
+
const connection: PeerConnection = await app.getObject(PeerConnection, {threads: 'peerThreads'})
|
|
390
|
+
const channel: PeerChannel = connection.createDataChannel('data')
|
|
391
|
+
channel.on('open', (): void => channel.send('hello'))
|
|
392
|
+
})
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
The threads start when needed: a new connection goes to a thread serving none (a new one is started while there are
|
|
396
|
+
fewer than `threads`, by default the number of cores less one), else to the thread serving the fewest. A connection
|
|
397
|
+
stays in its thread; a thread that stops (only a bug stops one) fails its connections (`failed`, then `closed`), and is
|
|
398
|
+
started again for the next ones. `resourceLimits` limits the heap of each thread, as `workerResourceLimits` does for the
|
|
399
|
+
media server (Bun ignores it). The bytes of the messages received on their way to the main thread stay out of the
|
|
400
|
+
receive window: a main thread that reads slowly slows the other peer down, as it would without threads. The RTP packets
|
|
401
|
+
on their way between the threads are 2 MiB at most per connection and direction; beyond that, they are dropped, as a
|
|
402
|
+
full UDP buffer would drop them.
|
|
403
|
+
|
|
404
|
+
Measured on an Apple M4 with Node.js 24, pairs of connections in one process exchanging messages over a data channel
|
|
405
|
+
(`packages/webrtc/test-env/bench/threads.mjs` in the repository):
|
|
406
|
+
|
|
407
|
+
| Pairs | Engines | Messages of 100 bytes | CPU per message | Messages of 16 KiB |
|
|
408
|
+
|---|---|---|---|---|
|
|
409
|
+
| 1 | in the main thread | 274,000 a second | 4.2 µs | 45 MiB/s |
|
|
410
|
+
| 1 | in 2 worker threads | 539,000 a second | 4.5 µs | 85 MiB/s |
|
|
411
|
+
| 4 | in the main thread | 267,000 a second | 4.3 µs | 46 MiB/s |
|
|
412
|
+
| 4 | in 4 worker threads | 782,000 a second | 6.2 µs | 138 MiB/s |
|
|
413
|
+
|
|
414
|
+
The two connections of a pair work at once in two threads: one pair carries twice as much as in the main thread, for
|
|
415
|
+
0.3 µs more of CPU per message. Four pairs in four threads carry three times as much; the CPU per message rises once the
|
|
416
|
+
threads outnumber the four performance cores of the M4. In an arm64 Linux virtual machine (Docker Desktop, Alpine) on
|
|
417
|
+
the same Mac, one pair carried 414,000 messages of 100 bytes a second in two threads, against 212,000 in the main
|
|
418
|
+
thread, at 6.8 µs of CPU each against 5.5; four pairs carried 460,000 against 222,000, at 10.7 µs against 5.4, and
|
|
419
|
+
70 MiB/s of large messages against 47, for 2.5 times the CPU: there, several threads allocating memory at once cost more
|
|
420
|
+
per message. The threads give a process more connections, using more cores, not cheaper messages.
|
|
421
|
+
|
|
422
|
+
In that virtual machine, each thread took about 100 MiB more with small messages, 55 MiB with large ones. On macOS, with
|
|
423
|
+
large messages through four pairs in four threads (138 MiB/s), the memory of the process (its physical footprint) in
|
|
424
|
+
most runs rose to between 1.5 and 3.5 GB in the first minute, then fell back: in a three-minute run, it stayed at about
|
|
425
|
+
700 MiB after the first minute. That is memory freed and kept for a while by the allocator of macOS, not a leak.
|
|
426
|
+
Node.js takes a 64 KiB buffer for each UDP datagram it receives; when several threads do so at that rate while other
|
|
427
|
+
small buffers are still in use, the allocator keeps much of the freed memory. The same run stays at about 540 MiB on
|
|
428
|
+
Linux, and at about 200 MiB in Bun on macOS, which has an allocator of its own.
|
|
429
|
+
|
|
430
|
+
## 7. Large messages and files
|
|
431
|
+
|
|
432
|
+
A data channel carries messages of up to 64 KiB with any peer; Chrome and the connections of this package accept up to
|
|
433
|
+
256 KiB. `ChunkedChannel` removes the limit: wrapped around the channel on both sides, it cuts each message into chunks
|
|
434
|
+
of 16 KiB, sends them as the channel's buffer empties, and puts them back together on the other side. It sends strings,
|
|
435
|
+
bytes and `Blob`s (a file of a browser is read a chunk at a time):
|
|
436
|
+
|
|
437
|
+
```typescript
|
|
438
|
+
import {ChunkedChannel, type Peer} from '@lakutata/webrtc/client'
|
|
439
|
+
|
|
440
|
+
declare const peer: Peer
|
|
441
|
+
declare const file: File
|
|
442
|
+
const transfer: ChunkedChannel = new ChunkedChannel(peer.createDataChannel('files'))
|
|
443
|
+
transfer.channel.addEventListener('open', async (): Promise<void> => {
|
|
444
|
+
await transfer.send(JSON.stringify({name: file.name, size: file.size}))
|
|
445
|
+
await transfer.send(file)
|
|
446
|
+
})
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
The channel must be ordered and reliable (the default). A message received over `maxMessageSize` (64 MiB by default)
|
|
450
|
+
is dropped with an `error` event.
|
|
451
|
+
|
|
452
|
+
## 8. Audio and video
|
|
453
|
+
|
|
454
|
+
A track carries the audio or the video of one source. In a browser, publish the tracks of the camera and the microphone
|
|
455
|
+
in the room: `room.publish()` sends a track to every member, the ones joining later too. The other members get it with
|
|
456
|
+
the `track` event of their peer, `event.streams` grouping the audio and the video of a source:
|
|
457
|
+
|
|
458
|
+
```typescript
|
|
459
|
+
import {type Peer, type Room} from '@lakutata/webrtc/client'
|
|
460
|
+
|
|
461
|
+
declare const room: Room
|
|
462
|
+
declare const grid: HTMLElement
|
|
463
|
+
const camera: MediaStream = await navigator.mediaDevices.getUserMedia({audio: true, video: true})
|
|
464
|
+
for (const track of camera.getTracks()) await room.publish(track, camera)
|
|
465
|
+
room.on('peer', (peer: Peer): void => {
|
|
466
|
+
const video: HTMLVideoElement = document.createElement('video')
|
|
467
|
+
video.autoplay = true
|
|
468
|
+
grid.append(video)
|
|
469
|
+
peer.on('track', (event: RTCTrackEvent): void => {
|
|
470
|
+
video.srcObject = event.streams[0]
|
|
471
|
+
})
|
|
472
|
+
peer.on('close', (): void => video.remove())
|
|
473
|
+
})
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
`room.unpublish(track)` stops sending a track; its transceivers are kept for the next track published.
|
|
477
|
+
`peer.pause(track)` and `peer.resume(track)` stop and start receiving a track of a member, a video not on the screen
|
|
478
|
+
for example. Without a media server, a track added to `peer.connection` goes to that member only.
|
|
479
|
+
|
|
480
|
+
Without a media server, each member sends its video once per other member: this suits small rooms. With the media
|
|
481
|
+
server of section 9, each member sends it once.
|
|
482
|
+
|
|
483
|
+
Node.js captures, encodes and decodes nothing: the tracks of an application carry RTP packets, the encoded audio and
|
|
484
|
+
video, from an encoder or from another connection.
|
|
485
|
+
|
|
486
|
+
- `new MediaTrack(kind, {codec})` creates a track to send, and `writeRtp()` sends a packet: an `RtpPacket`, or the
|
|
487
|
+
bytes of one, such as the datagrams ffmpeg and GStreamer send to a UDP port. Each connection gives the packets its own
|
|
488
|
+
synchronization source and payload type, and the sequence numbers and the timestamps go on from the ones written.
|
|
489
|
+
- A received track (the `track` event of a `Peer` or of a `PeerConnection`) gives the packets of the other member with
|
|
490
|
+
its `rtp` event; `requestKeyFrame()` asks the other member for a key frame.
|
|
491
|
+
- A received track added to another connection forwards its packets there, and the key frame requests of that
|
|
492
|
+
connection go back to the sender of the track (one every 500 ms at most).
|
|
493
|
+
- The `keyframerequest` event of a track fires when a peer the track is sent to wants a key frame.
|
|
494
|
+
|
|
495
|
+
A camera sending what ffmpeg encodes:
|
|
496
|
+
|
|
497
|
+
```typescript
|
|
498
|
+
import {createSocket, type Socket} from 'node:dgram'
|
|
499
|
+
import {Component} from 'lakutata'
|
|
500
|
+
import {MediaTrack, type Room, SignalingClient} from '@lakutata/webrtc'
|
|
501
|
+
|
|
502
|
+
//ffmpeg -f v4l2 -i /dev/video0 -c:v libx264 -profile:v baseline -pix_fmt yuv420p -tune zerolatency -g 30
|
|
503
|
+
// -bsf:v dump_extra=freq=keyframe -f rtp rtp://127.0.0.1:5004
|
|
504
|
+
export class Camera extends Component {
|
|
505
|
+
protected async init(): Promise<void> {
|
|
506
|
+
const video: MediaTrack = new MediaTrack('video', {codec: 'video/H264'})
|
|
507
|
+
const socket: Socket = createSocket('udp4')
|
|
508
|
+
socket.on('message', (datagram: Buffer): void => video.writeRtp(datagram))
|
|
509
|
+
socket.bind(5004)
|
|
510
|
+
const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'camera-3'})
|
|
511
|
+
const room: Room = await signaling.join('camera-3')
|
|
512
|
+
await room.publish(video, 'camera-3')
|
|
513
|
+
}
|
|
514
|
+
}
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
A viewer starts decoding at a key frame: give the encoder a key frame every second or two (`-g` of ffmpeg), or have the
|
|
518
|
+
encoder make one when the track emits `keyframerequest`, if it can. The bitstream filter `dump_extra` puts the
|
|
519
|
+
parameters of H.264 before each key frame, where the browsers need them.
|
|
520
|
+
|
|
521
|
+
An application sending the camera of a member on to the viewers who join, each viewer getting the video from the
|
|
522
|
+
application instead of the camera (the media server of section 9 does it for every member of a room):
|
|
523
|
+
|
|
524
|
+
```typescript
|
|
525
|
+
import {Component} from 'lakutata'
|
|
526
|
+
import {type MediaTrack, type Peer, type PeerTrackEvent, type Room, SignalingClient} from '@lakutata/webrtc'
|
|
527
|
+
|
|
528
|
+
//The page of the camera adds its tracks to the peer of the relay only
|
|
529
|
+
export class Relay extends Component {
|
|
530
|
+
protected async init(): Promise<void> {
|
|
531
|
+
const signaling: SignalingClient = await this.getObject(SignalingClient, {url: 'https://meet.example.com', id: 'relay'})
|
|
532
|
+
const room: Room = await signaling.join('stage')
|
|
533
|
+
const tracks: MediaTrack[] = []
|
|
534
|
+
const viewers: Set<Peer> = new Set()
|
|
535
|
+
room.on('peer', (peer: Peer): void => {
|
|
536
|
+
if (peer.id === 'camera') {
|
|
537
|
+
peer.on('track', (event: PeerTrackEvent): void => {
|
|
538
|
+
tracks.push(event.track)
|
|
539
|
+
for (const viewer of viewers) viewer.connection.addTrack(event.track, 'stage')
|
|
540
|
+
})
|
|
541
|
+
return
|
|
542
|
+
}
|
|
543
|
+
viewers.add(peer)
|
|
544
|
+
for (const track of tracks) peer.connection.addTrack(track, 'stage')
|
|
545
|
+
peer.on('close', (): void => void viewers.delete(peer))
|
|
546
|
+
})
|
|
547
|
+
}
|
|
548
|
+
}
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
The methods of `PeerConnection` are those of `RTCPeerConnection`: `addTrack(track, ...streamIds)` (the other member
|
|
552
|
+
groups the tracks of a stream), `addTransceiver(trackOrKind, {direction, streams})` (`recvonly` to receive without
|
|
553
|
+
sending), `removeTrack(sender)`, `getTransceivers()`, `getSenders()` and `getReceivers()`; `sender.replaceTrack()` sends
|
|
554
|
+
another track, of any codec negotiated, without negotiating again. The codecs are Opus and PCMU for the audio, VP8,
|
|
555
|
+
H.264 and VP9 for the video; the `codecs` option of `PeerConnection` and of `SignalingClient` chooses them and their
|
|
556
|
+
order, and a track created with a `codec` makes its connections negotiate that one.
|
|
557
|
+
|
|
558
|
+
## 9. The media server
|
|
559
|
+
|
|
560
|
+
Without a media server, each member of a room sends its tracks to each other member: in a meeting of eight, each one
|
|
561
|
+
uploads its video seven times and keeps seven connections. The `SFU` component is a media server for the rooms of the
|
|
562
|
+
signaling (a selective forwarding unit): each member connects to it only, sends each track once, and receives the
|
|
563
|
+
tracks of the others from it. The server forwards their packets without decoding them, and a member uploads its video
|
|
564
|
+
once, however large the room.
|
|
565
|
+
|
|
566
|
+
```typescript
|
|
567
|
+
import {Application} from 'lakutata'
|
|
568
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
569
|
+
import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
|
|
570
|
+
import {buildSFUOptions, buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
|
|
571
|
+
|
|
572
|
+
Application.run(() => ({
|
|
573
|
+
id: 'meet.app',
|
|
574
|
+
name: 'Meet',
|
|
575
|
+
components: {
|
|
576
|
+
turn: buildTURNServerOptions({publicAddress: '203.0.113.10', relayPortRange: [49160, 49200], secret: process.env.TURN_SECRET}),
|
|
577
|
+
signaling: buildSignalingOptions({turn: 'turn'}),
|
|
578
|
+
//The meetings go through the media server; the other rooms stay peer to peer
|
|
579
|
+
sfu: buildSFUOptions({
|
|
580
|
+
serves: (room: string): boolean => room.startsWith('meeting-'),
|
|
581
|
+
iceAdditionalHostAddresses: ['203.0.113.10'],
|
|
582
|
+
icePortRange: [40000, 40999]
|
|
583
|
+
}),
|
|
584
|
+
entrypoint: buildEntrypoints({
|
|
585
|
+
controllers: [SignalingController],
|
|
586
|
+
service: socketIOServiceEntrypoint({port: 3000})
|
|
587
|
+
})
|
|
588
|
+
},
|
|
589
|
+
bootstrap: ['turn', 'sfu', 'entrypoint']
|
|
590
|
+
}))
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
The clients need no change. The server joins each room it serves as a member whose id starts with `sfu:`, and the
|
|
594
|
+
clients of this package (`SignalingConnection` in the browsers, `SignalingClient` in Node.js) connect to it instead of
|
|
595
|
+
to each other. The rooms and the peers work as without it:
|
|
596
|
+
|
|
597
|
+
- `room.publish()` sends a track once, to the server, which forwards it to every member, the ones joining later too.
|
|
598
|
+
- Each peer emits the tracks of its member. Their `event.streams` are the streams of the connection to the server;
|
|
599
|
+
`room.streamOf(id)` gives the member a stream comes from, and the id of the stream on the connection of that member.
|
|
600
|
+
- `peer.createDataChannel()` opens a channel through the server, which relays its messages both ways. Unlike between
|
|
601
|
+
two members, the server sees them. Such a channel cannot be given a `protocol`, and cannot be `negotiated`.
|
|
602
|
+
- `peer.pause(track)` asks the server to stop sending a track, `peer.resume(track)` to send it again.
|
|
603
|
+
- `room.sfu` is the id of the server, and `peer.connection` is the connection to it, the same for all the peers:
|
|
604
|
+
publish the tracks with `room.publish()` rather than add them to it.
|
|
605
|
+
|
|
606
|
+
The options of `buildSFUOptions`:
|
|
607
|
+
|
|
608
|
+
| Option | Default | Meaning |
|
|
609
|
+
|---|---|---|
|
|
610
|
+
| `signaling` | `'signaling'` | The `Signaling` component whose rooms the server serves |
|
|
611
|
+
| `serves` | all the rooms | `(room) => boolean`: the rooms served; the others stay peer to peer |
|
|
612
|
+
| `select` | every member receives every track | `(subscriber, track) => boolean`: whether a member receives a track |
|
|
613
|
+
| `maxMembers` | `0` | The members of all the rooms at most, 0 for no limit; a member joining once it is reached is refused (`SignalingLimitException`) |
|
|
614
|
+
| `codecs` | Opus, PCMU, VP8, H.264, VP9 | The codecs the members may publish |
|
|
615
|
+
| `icePortRange`, `iceAdditionalHostAddresses` | | The UDP ports of the connections, and the public address of the server. Each connection (a member, a stream of WHIP, a viewer of WHEP) takes a port on each address of the machine: make the range large enough for all of them |
|
|
616
|
+
| `workers` | `0` | Worker threads forwarding the rooms, to use more cores |
|
|
617
|
+
| `workerResourceLimits` | the limits of V8 | The limits of the heap and the stack of each worker thread (`resourceLimits` of `node:worker_threads`), to use less memory for a little more CPU (see Several cores below); Bun ignores them |
|
|
618
|
+
| `cluster`, `selectSFU` | | The media servers of several nodes (section 13) |
|
|
619
|
+
|
|
620
|
+
`select` is asked for each member and each track of the others, and asked again when a member joins the room again
|
|
621
|
+
with other metadata: the tracks the member receives follow the new answers. A `select` that throws counts as `false`,
|
|
622
|
+
and its error goes to the debug messages (section 15). In a webinar, for example, everyone receives the audio of all
|
|
623
|
+
and the video of the speakers only:
|
|
624
|
+
|
|
625
|
+
```typescript
|
|
626
|
+
import {buildSFUOptions, type SFUTrack, type SignalingPeer} from '@lakutata/webrtc'
|
|
627
|
+
|
|
628
|
+
const sfu = buildSFUOptions({
|
|
629
|
+
select: (subscriber: SignalingPeer, track: SFUTrack): boolean => track.kind === 'audio' || track.peerId.startsWith('speaker-')
|
|
630
|
+
})
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
How the server forwards:
|
|
634
|
+
|
|
635
|
+
- It does not transcode: a member receives the tracks of the codecs it supports, and the server emits `unsupported`
|
|
636
|
+
for the others. For the browsers, publish the video in VP8 or H.264.
|
|
637
|
+
- The key frame requests of the members go to the publisher, one every 500 ms at most, and the server asks the
|
|
638
|
+
publisher for a key frame when a member starts receiving a video. The packets lost on the way to a member are sent
|
|
639
|
+
again by the server.
|
|
640
|
+
- Each connection estimates its bandwidth from the feedback of the other side: the browsers adapt what they send to
|
|
641
|
+
the server, and the server emits `congestion` when the bandwidth to a member stays too low for what it receives, and
|
|
642
|
+
again once it is enough. The server sends each video as it is published (there is no simulcast yet): pause some of
|
|
643
|
+
the videos the member receives to lower the load.
|
|
644
|
+
- A member reading the messages of a channel slower than they are sent makes their sender wait; the server keeps a few
|
|
645
|
+
megabytes of them at most.
|
|
646
|
+
|
|
647
|
+
Its events: `publish` and `unpublish` (an `SFUTrack`: its `room`, `peerId`, `id`, `kind`, `codec` and `stream`) when a
|
|
648
|
+
member starts and stops publishing a track; `unsupported` (the room, the id of the member, the track) when a member
|
|
649
|
+
does not support the codec of a track it should receive; `congestion` (the room, the id of the member, whether the
|
|
650
|
+
bandwidth is too low, the bit rate estimated in bits per second). `rooms()` lists the rooms with their members, their
|
|
651
|
+
tracks and their viewers of WHEP. `track(room, peerId, trackId)` gives a track published as a `MediaTrack`, whose `rtp`
|
|
652
|
+
event gives its packets, to record it or to send it on another connection:
|
|
653
|
+
|
|
654
|
+
```typescript
|
|
655
|
+
import {Component} from 'lakutata'
|
|
656
|
+
import type {RtpPacket, SFU, SFUTrack} from '@lakutata/webrtc'
|
|
657
|
+
|
|
658
|
+
export class Recorder extends Component {
|
|
659
|
+
protected async init(): Promise<void> {
|
|
660
|
+
const sfu: SFU = await this.getObject<SFU>('sfu')
|
|
661
|
+
sfu.on('publish', (track: SFUTrack): void => {
|
|
662
|
+
if (track.kind !== 'audio') return
|
|
663
|
+
sfu.track(track.room, track.peerId, track.id)?.on('rtp', (packet: RtpPacket): void => console.log(track.peerId, packet.timestamp))
|
|
664
|
+
})
|
|
665
|
+
}
|
|
666
|
+
}
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
### Several cores
|
|
670
|
+
|
|
671
|
+
The server forwards in the main thread by default. With `workers: 4`, four worker threads forward the rooms: each room
|
|
672
|
+
in one of them (the one forwarding the fewest members and viewers when the room opens), with its connections and its
|
|
673
|
+
UDP ports, while the hooks and the events stay in the main thread. `icePortRange` is shared out among the threads. A
|
|
674
|
+
thread that stops closes its rooms; the server opens them again in another thread, and their members connect to it
|
|
675
|
+
again. `rooms()` shows what each thread reports, so a change made in a thread (a viewer leaving) shows a moment later.
|
|
676
|
+
|
|
677
|
+
Measured on an Apple M4 with Node.js 24, each publisher sending a video of about 2 Mbit/s (200 packets of 1250 bytes
|
|
678
|
+
a second), the members in other processes:
|
|
679
|
+
|
|
680
|
+
| Rooms | Packets to forward | Forwarded | CPU |
|
|
681
|
+
|---|---|---|---|
|
|
682
|
+
| 1 room of 4 publishers and 10 subscribers, in the main thread | 7,600 a second | all | 45% of a core |
|
|
683
|
+
| 4 rooms of 12 publishers and 10 subscribers each, in the main thread | about 92,000 a second | 86,000 to 88,000 a second | 145%: the main thread is full, the rest is the garbage collector |
|
|
684
|
+
| The same 4 rooms, in 4 worker threads | about 92,000 a second | all | 174% |
|
|
685
|
+
|
|
686
|
+
At a load as light as the first room's, the share of a core depends on how fast the system runs the cores, and it runs
|
|
687
|
+
them faster when the machine is busy: with two other cores kept busy, the same room took 21%.
|
|
688
|
+
|
|
689
|
+
In an arm64 Linux virtual machine (Docker Desktop, 10 cores) on the same Mac, with the members running in it too, the
|
|
690
|
+
main thread forwarded all the packets of the 4 rooms (93,000 a second) at 160% of a core. Four threads took 380% to
|
|
691
|
+
390% of a core, and forwarded from 69,000 to all of the 94,000 a second, depending on the run: there, at that load, the
|
|
692
|
+
threads do not pay off.
|
|
693
|
+
|
|
694
|
+
A thread takes about 22 MiB and 90 ms to start. Under the load above, the process used 1.4 GiB with the threads and
|
|
695
|
+
0.6 GiB without: each thread has a heap of its own, with the large young generation V8 gives it for speed. With
|
|
696
|
+
`workerResourceLimits: {maxYoungGenerationSizeMb: 32}`, the process used 1.0 to 1.2 GiB, for 4% more CPU (181% of a
|
|
697
|
+
core instead of 174%). A track forwarded from one node to another (section 13) costs the two nodes about 11 µs of CPU
|
|
698
|
+
per packet: 0.2% of a core for such a video.
|
|
699
|
+
|
|
700
|
+
## 10. Publishing with WHIP
|
|
701
|
+
|
|
702
|
+
WHIP (RFC 9725) publishes a stream with one HTTP request: the encoder (OBS, GStreamer, ffmpeg) or the browser posts its
|
|
703
|
+
offer to the URL of the stream, and gets the answer and the URL of its session. The `WHIP` component answers the offers
|
|
704
|
+
and emits `publish` with the session (its `id`, `stream`, `connection` and `tracks`) and `unpublish` when it ends: the
|
|
705
|
+
publisher deleted it, its connection failed or closed, or the application stopped. `WHIPController` is its HTTP
|
|
706
|
+
interface; the HTTP adapter must read `application/sdp` as text (its `body.text` option).
|
|
707
|
+
|
|
708
|
+
```typescript
|
|
709
|
+
import {Application, Component} from 'lakutata'
|
|
710
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
711
|
+
import {expressHTTPEntrypoint} from 'lakutata/adapter/express'
|
|
712
|
+
import {buildWHIPOptions, type MediaTrack, type WHIP, type WHIPSession, WHIPController} from '@lakutata/webrtc'
|
|
713
|
+
|
|
714
|
+
export class Studio extends Component {
|
|
715
|
+
protected async init(): Promise<void> {
|
|
716
|
+
const whip: WHIP = await this.getObject<WHIP>('whip')
|
|
717
|
+
whip.on('publish', (session: WHIPSession): void => {
|
|
718
|
+
session.tracks?.forEach((track: MediaTrack): void => console.log(`${session.stream}: ${track.kind} ${track.codec}`))
|
|
719
|
+
})
|
|
720
|
+
whip.on('unpublish', (session: WHIPSession): void => console.log(`${session.stream} ended`))
|
|
721
|
+
}
|
|
722
|
+
}
|
|
723
|
+
|
|
724
|
+
Application.run(() => ({
|
|
725
|
+
id: 'studio.app',
|
|
726
|
+
name: 'Studio',
|
|
727
|
+
components: {
|
|
728
|
+
whip: buildWHIPOptions({
|
|
729
|
+
authorize: ({stream, token}): boolean => stream === 'live' && token === process.env.WHIP_TOKEN,
|
|
730
|
+
iceAdditionalHostAddresses: ['203.0.113.10'],
|
|
731
|
+
icePortRange: [40000, 40999]
|
|
732
|
+
}),
|
|
733
|
+
studio: {class: Studio},
|
|
734
|
+
entrypoint: buildEntrypoints({
|
|
735
|
+
controllers: [WHIPController],
|
|
736
|
+
http: expressHTTPEntrypoint({port: 8080, body: {text: ['application/sdp', 'application/trickle-ice-sdpfrag']}})
|
|
737
|
+
})
|
|
738
|
+
},
|
|
739
|
+
bootstrap: ['studio', 'entrypoint']
|
|
740
|
+
}))
|
|
741
|
+
```
|
|
742
|
+
|
|
743
|
+
In OBS, choose the service WHIP, the server `https://studio.example.com/whip/live` and the bearer token; ffmpeg
|
|
744
|
+
publishes with `-f whip -authorization <token> https://studio.example.com/whip/live` (stereo audio: `-ac 2`). The tracks
|
|
745
|
+
of the session give their RTP packets, to record them or to send them on to other connections (section 8).
|
|
746
|
+
|
|
747
|
+
| Request | Answer |
|
|
748
|
+
|---|---|
|
|
749
|
+
| `POST /whip/:stream` with the offer (`application/sdp`) | `201 Created`: the answer, `Location` (the session), `Link` (the ICE servers), `ETag` (its ICE session) |
|
|
750
|
+
| `DELETE /whip/:stream/:session` | `200`: the session ends |
|
|
751
|
+
| `OPTIONS` | `204`, with the CORS headers and `Accept-Post: application/sdp` |
|
|
752
|
+
| `GET` | `204` |
|
|
753
|
+
| `PATCH /whip/:stream/:session` with a fragment of SDP (`application/trickle-ice-sdpfrag`) | `204`: the candidates of the publisher (trickle ICE); `200`: an ICE restart, with the fragment of the answer and a new `ETag`; `412` for an `If-Match` other than `*` and the `ETag` |
|
|
754
|
+
|
|
755
|
+
The refusals: 401 with `WWW-Authenticate: Bearer` when `authorize` refuses, 415 for a body that is not
|
|
756
|
+
`application/sdp` (`application/trickle-ice-sdpfrag` for a PATCH), 400 for an offer without audio or video, 503 once
|
|
757
|
+
there are `maxSessions` sessions, 404 for an unknown session. The URL of a session is the capability to end it: its id
|
|
758
|
+
is random, and its DELETE and PATCH are not authorized again. A subclass with `@Prefix({route: '/live'})` serves the
|
|
759
|
+
same interface under another path.
|
|
760
|
+
|
|
761
|
+
| Option | Default | Meaning |
|
|
762
|
+
|---|---|---|
|
|
763
|
+
| `authorize` | none | `({stream, token}, context) => boolean`: who may publish which stream |
|
|
764
|
+
| `turn` | none | The name of a `TURNServer` component: the publishers get its ICE servers in the `Link` headers |
|
|
765
|
+
| `iceServers` | `[]` | More ICE servers for the publishers |
|
|
766
|
+
| `maxSessions` | `100` | The sessions at once at most, `0` for no limit |
|
|
767
|
+
| `allowOrigin` | `'*'` | The origins the browsers may publish from |
|
|
768
|
+
| `icePortRange`, `iceAdditionalHostAddresses` | | The candidates of the server: its UDP ports (each session takes a port on each address of the machine; with `sfu`, the ports of the media server are used), its public address |
|
|
769
|
+
| `codecs` | Opus, PCMU, VP8, H.264, VP9 | The codecs accepted |
|
|
770
|
+
| `sfu` | none | The name of an `SFU` component: the streams go to its rooms |
|
|
771
|
+
| `roomOf` | the stream | `({stream, token}, context) => string`: the room of the media server a stream goes to |
|
|
772
|
+
|
|
773
|
+
The answer carries all the candidates of the server, which gathers them before answering: behind a NAT or in a
|
|
774
|
+
container, give it its public address and open its ports.
|
|
775
|
+
|
|
776
|
+
With `sfu`, a stream goes to a room of the media server (section 9) instead of the application: the room named after
|
|
777
|
+
the stream, or the one `roomOf` gives. It joins the room as a member whose id starts with `whip:`, the members of the
|
|
778
|
+
room receive its tracks as those of any member, and the viewers of WHEP play it (section 11). Its connection then
|
|
779
|
+
belongs to the media server, so the session has the `room` and the `peerId` of the stream instead of `connection` and
|
|
780
|
+
`tracks`; `sfu.track()` gives its packets.
|
|
781
|
+
|
|
782
|
+
## 11. Playing with WHEP
|
|
783
|
+
|
|
784
|
+
WHEP (the WebRTC-HTTP Egress Protocol) plays a stream with one HTTP request: the player posts a receive-only offer, and
|
|
785
|
+
gets the answer and the URL of its session. The `WHEP` component plays the rooms of the media server:
|
|
786
|
+
each viewer receives the audio and the video of one member of a room, by default the stream published into it with
|
|
787
|
+
WHIP. The viewers are not members of the room, so the members do not see them, and a stream may have many of them.
|
|
788
|
+
`WHEPController` is its HTTP interface, under `/whep`.
|
|
789
|
+
|
|
790
|
+
```typescript
|
|
791
|
+
import {Application} from 'lakutata'
|
|
792
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
793
|
+
import {expressHTTPEntrypoint} from 'lakutata/adapter/express'
|
|
794
|
+
import {buildSFUOptions, buildSignalingOptions, buildWHEPOptions, buildWHIPOptions, WHEPController, WHIPController} from '@lakutata/webrtc'
|
|
795
|
+
|
|
796
|
+
//OBS publishes to https://live.example.com/whip/concert, the browsers play https://live.example.com/whep/concert
|
|
797
|
+
Application.run(() => ({
|
|
798
|
+
id: 'live.app',
|
|
799
|
+
name: 'Live',
|
|
800
|
+
components: {
|
|
801
|
+
signaling: buildSignalingOptions(),
|
|
802
|
+
sfu: buildSFUOptions({iceAdditionalHostAddresses: ['203.0.113.10'], icePortRange: [40000, 40999]}),
|
|
803
|
+
whip: buildWHIPOptions({sfu: 'sfu', authorize: ({token}): boolean => token === process.env.WHIP_TOKEN}),
|
|
804
|
+
whep: buildWHEPOptions({maxSessions: 500}),
|
|
805
|
+
entrypoint: buildEntrypoints({
|
|
806
|
+
controllers: [WHIPController, WHEPController],
|
|
807
|
+
http: expressHTTPEntrypoint({port: 8080, body: {text: ['application/sdp', 'application/trickle-ice-sdpfrag']}})
|
|
808
|
+
})
|
|
809
|
+
},
|
|
810
|
+
bootstrap: ['sfu', 'entrypoint']
|
|
811
|
+
}))
|
|
812
|
+
```
|
|
813
|
+
|
|
814
|
+
A browser plays it with its own `RTCPeerConnection`, its offer carrying all its candidates:
|
|
815
|
+
|
|
816
|
+
```typescript
|
|
817
|
+
declare const player: HTMLVideoElement
|
|
818
|
+
const connection: RTCPeerConnection = new RTCPeerConnection()
|
|
819
|
+
const stream: MediaStream = new MediaStream()
|
|
820
|
+
player.srcObject = stream
|
|
821
|
+
connection.addEventListener('track', (event: RTCTrackEvent): void => stream.addTrack(event.track))
|
|
822
|
+
connection.addTransceiver('audio', {direction: 'recvonly'})
|
|
823
|
+
connection.addTransceiver('video', {direction: 'recvonly'})
|
|
824
|
+
await connection.setLocalDescription()
|
|
825
|
+
await new Promise<void>((resolve: () => void): void => {
|
|
826
|
+
if (connection.iceGatheringState === 'complete') return resolve()
|
|
827
|
+
connection.addEventListener('icegatheringstatechange', (): void => {
|
|
828
|
+
if (connection.iceGatheringState === 'complete') resolve()
|
|
829
|
+
})
|
|
830
|
+
})
|
|
831
|
+
const response: Response = await fetch('https://live.example.com/whep/concert', {method: 'POST', headers: {'Content-Type': 'application/sdp'}, body: connection.localDescription!.sdp})
|
|
832
|
+
await connection.setRemoteDescription({type: 'answer', sdp: await response.text()})
|
|
833
|
+
```
|
|
834
|
+
|
|
835
|
+
| Request | Answer |
|
|
836
|
+
|---|---|
|
|
837
|
+
| `POST /whep/:stream` with the offer (`application/sdp`) | `201 Created`: the answer, `Location` (the session), `Link` (the ICE servers), `ETag` (its ICE session) |
|
|
838
|
+
| `DELETE /whep/:stream/:session` | `200`: the session ends |
|
|
839
|
+
| `PATCH /whep/:stream/:session` with a fragment of SDP (`application/trickle-ice-sdpfrag`) | As for WHIP: `204` for the candidates of the viewer, `200` with the fragment of the answer for an ICE restart, `412` for an `If-Match` other than `*` and the `ETag` |
|
|
840
|
+
| `OPTIONS` | `204`, with the CORS headers and `Accept-Post: application/sdp` |
|
|
841
|
+
| `GET` | `204` |
|
|
842
|
+
|
|
843
|
+
The refusals: 401 with `WWW-Authenticate: Bearer` when `authorize` refuses, 415 for a body of another type, 400 for an
|
|
844
|
+
offer without audio or video or a room the media server does not serve, 503 once there are `maxSessions` sessions, 404
|
|
845
|
+
for an unknown session.
|
|
846
|
+
|
|
847
|
+
| Option | Default | Meaning |
|
|
848
|
+
|---|---|---|
|
|
849
|
+
| `sfu` | `'sfu'` | The name of the `SFU` component whose rooms are played |
|
|
850
|
+
| `roomOf` | the stream | `({stream, token}, context) => string`: the room a stream comes from |
|
|
851
|
+
| `select` | the newest stream of WHIP, else the first member publishing | `({stream, token}, tracks) => string \| undefined`: the id of the member a viewer receives |
|
|
852
|
+
| `authorize` | none | `({stream, token}, context) => boolean`: who may play which stream |
|
|
853
|
+
| `maxSessions` | `100` | The sessions at once on this node, `0` for no limit |
|
|
854
|
+
| `allowOrigin` | `'*'` | The origins the browsers may play from |
|
|
855
|
+
| `turn`, `iceServers` | | The ICE servers given to the viewers in the `Link` headers |
|
|
856
|
+
| `cluster`, `turnSecret`, `selectTURN` | | Several nodes (section 13) |
|
|
857
|
+
|
|
858
|
+
A viewer receives one member: by default the newest stream published into the room with WHIP, else the first member
|
|
859
|
+
publishing audio or video. `select` chooses another one; it is asked again each time the tracks of the room change,
|
|
860
|
+
and when it throws, the viewer receives the default one. When the member stops publishing, the viewer goes on with the
|
|
861
|
+
next one on the same connection: its picture stops for a moment and starts again at a key frame. The offer of a viewer
|
|
862
|
+
fixes what it receives, one audio and one video, and its connection negotiates every codec of the offer that the
|
|
863
|
+
server has: a stream published later in any of them is sent too.
|
|
864
|
+
|
|
865
|
+
The component emits `play` (the session: its `id`, `stream`, `room`, `viewerId` and `source`, the member it receives)
|
|
866
|
+
once it answers, and `stop` when the session ends: the viewer deleted it, its connection failed or closed, or the
|
|
867
|
+
application stopped. `whep.sessions` lists the sessions of this node, and `sfu.rooms()` counts the viewers of each
|
|
868
|
+
room.
|
|
869
|
+
|
|
870
|
+
## 12. Deployment
|
|
871
|
+
|
|
872
|
+
- **Ports.** Open port 3478 over UDP and TCP, the relay ports (`relayPortRange`) over UDP, and the TLS port when TURN
|
|
873
|
+
over TLS is on. In a container, publish them all; the relay range keeps the list short.
|
|
874
|
+
- **The public address.** `publicAddress` is the address the clients reach the TURN server at: the public address of
|
|
875
|
+
the host, or that of its NAT when the NAT forwards the ports. On a local network the default is enough.
|
|
876
|
+
- **The applications as members.** A server behind a NAT that forwards `icePortRange` with the same port numbers gives
|
|
877
|
+
its public address in `iceAdditionalHostAddresses`; otherwise it relies on the TURN server like any client.
|
|
878
|
+
- **The media server, WHIP and WHEP.** The clients connect to them directly: open their `icePortRange` over UDP, and
|
|
879
|
+
give them in `iceAdditionalHostAddresses` the public address of a NAT forwarding these ports with the same numbers.
|
|
880
|
+
A client that cannot reach them goes through the TURN server.
|
|
881
|
+
- **Several instances.** Give them a `Cluster` (section 13), and their TURN servers the same `secret`, so each accepts
|
|
882
|
+
the credentials made by the others.
|
|
883
|
+
- **Another TURN server.** coturn (`use-auth-secret`, `static-auth-secret` set to the same secret) accepts the
|
|
884
|
+
credentials of `createTURNCredentials()`; list it in the `iceServers` option of `Signaling`.
|
|
885
|
+
- **HTTPS.** Browsers allow WebRTC on any page, but the camera and the microphone only on secure pages (HTTPS or
|
|
886
|
+
`localhost`).
|
|
887
|
+
- **The UDP buffers.** The connections and the TURN server ask for a receive buffer of 4 MiB for their UDP sockets:
|
|
888
|
+
a smaller one loses packets in a burst, or when a large transfer arrives faster than the process reads it. Linux
|
|
889
|
+
caps the size at `net.core.rmem_max`, which is 208 KiB on most systems, so raise it on a Linux server
|
|
890
|
+
(`sysctl -w net.core.rmem_max=4194304`, on the nodes for the pods on the network of their node).
|
|
891
|
+
|
|
892
|
+
## 13. Several instances
|
|
893
|
+
|
|
894
|
+
Several instances of an application (the pods of a deployment, the processes behind a load balancer) work together
|
|
895
|
+
through a `Cluster`: each instance is a node of the cluster, and the nodes talk through NATS or Redis. The components
|
|
896
|
+
given its name work across the nodes:
|
|
897
|
+
|
|
898
|
+
- the signaling puts the members connected to all the nodes in the same rooms;
|
|
899
|
+
- the TURN servers tell the other nodes their address and their load, and each member gets the least loaded ones;
|
|
900
|
+
- the media servers share the rooms: each member, stream of WHIP and viewer of WHEP connects to the media server of one
|
|
901
|
+
node, and the media servers of a room forward its tracks to each other;
|
|
902
|
+
- a session of WHIP or WHEP is ended or patched whichever node its DELETE or its PATCH reaches.
|
|
903
|
+
|
|
904
|
+
The bus carries only these messages. The audio, the video and the data go directly between the clients and the nodes,
|
|
905
|
+
between the media servers of the nodes, or through the TURN servers.
|
|
906
|
+
|
|
907
|
+
```typescript
|
|
908
|
+
import {Application} from 'lakutata'
|
|
909
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
910
|
+
import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
|
|
911
|
+
import {buildClusterOptions, buildSignalingOptions, SignalingController} from '@lakutata/webrtc'
|
|
912
|
+
|
|
913
|
+
Application.run(() => ({
|
|
914
|
+
id: 'meet.app',
|
|
915
|
+
name: 'Meet',
|
|
916
|
+
components: {
|
|
917
|
+
cluster: buildClusterOptions({transport: {type: 'nats', servers: process.env.NATS_URL}}),
|
|
918
|
+
signaling: buildSignalingOptions({cluster: 'cluster', turnSecret: process.env.TURN_SECRET}),
|
|
919
|
+
entrypoint: buildEntrypoints({
|
|
920
|
+
controllers: [SignalingController],
|
|
921
|
+
service: socketIOServiceEntrypoint({port: 3000})
|
|
922
|
+
})
|
|
923
|
+
},
|
|
924
|
+
bootstrap: ['signaling', 'entrypoint']
|
|
925
|
+
}))
|
|
926
|
+
```
|
|
927
|
+
|
|
928
|
+
| Option of `buildClusterOptions` | Default | |
|
|
929
|
+
|---|---|---|
|
|
930
|
+
| `transport` | (required) | `{type: 'nats', servers, user, pass, token}`; `{type: 'redis', url}` or `{type: 'redis', host, port, username, password, database, tls}`; `{type: 'memory'}` for the applications of one process (tests) |
|
|
931
|
+
| `prefix` | `'lakutata.webrtc'` | The prefix of the subjects or channels: the clusters sharing a server stay separate with it |
|
|
932
|
+
| `nodeId` | the host name and a random suffix | The id of the node; in Kubernetes the host name is the name of the pod |
|
|
933
|
+
| `labels` | `{}` | What the other nodes see of this one (a zone, a region), to choose the TURN servers |
|
|
934
|
+
| `heartbeatInterval` | `2000` | How often the node tells the others it is there, in milliseconds |
|
|
935
|
+
| `requestTimeout` | `5000` | How long `request()` and `requestAll()` wait, in milliseconds |
|
|
936
|
+
|
|
937
|
+
NATS needs `npm install nats`, and its core subjects are enough (no JetStream); Redis needs `npm install ioredis`, and
|
|
938
|
+
only its channels are used (`PUBLISH`, `SUBSCRIBE`). The component connects when the application starts, and fails
|
|
939
|
+
to start when the server cannot be reached then (in Kubernetes, the pod restarts until it can); afterwards it
|
|
940
|
+
reconnects by itself. The other nodes take a node as gone after three heartbeat intervals without news of it, or at
|
|
941
|
+
once when it stops.
|
|
942
|
+
|
|
943
|
+
### The rooms across the nodes
|
|
944
|
+
|
|
945
|
+
Each node tells the others who joins and leaves through it, and gets all the members of a node when that node comes up
|
|
946
|
+
or when some of its news was lost (the bus restarting, for example). A signal to a member of another node goes through
|
|
947
|
+
that node.
|
|
948
|
+
|
|
949
|
+
- When a node stops, or the others cannot reach it any more, its members leave their rooms as `disconnected`: their
|
|
950
|
+
connections stay up while their clients reconnect to another node and join again.
|
|
951
|
+
- A client joining again through another node before the first one saw it go is moved there: the others see it join
|
|
952
|
+
again, not leave.
|
|
953
|
+
- `maxPeers` counts the members of all the nodes. Each room falls to one node (by hashing over the nodes), which admits
|
|
954
|
+
its members one at a time, so two members joining at once through different nodes cannot both take the last place.
|
|
955
|
+
When a node joins or leaves the cluster, some rooms fall to another node, and a member joining at that very moment
|
|
956
|
+
may be admitted over the limit.
|
|
957
|
+
- Socket.IO behind a load balancer: its long polling sends the requests of a client one by one, and they must all reach
|
|
958
|
+
the same instance. Connect the clients with `transports: ['websocket']` (a single connection, so no sticky sessions),
|
|
959
|
+
or turn on sticky sessions in the load balancer.
|
|
960
|
+
|
|
961
|
+
### The TURN servers of the cluster
|
|
962
|
+
|
|
963
|
+
A TURN server joins the cluster with its `cluster` option and the `secret` shared by all of them (it refuses to start
|
|
964
|
+
without one); it tells the others its address, its ports and its allocations every second. The `Signaling` and `WHIP`
|
|
965
|
+
components given `turnSecret` give each client the two least loaded TURN servers that are not full, after the one of
|
|
966
|
+
their own node (`turn`). `selectTURN` chooses otherwise, for example by the labels of the nodes:
|
|
967
|
+
|
|
968
|
+
```typescript
|
|
969
|
+
import {buildSignalingOptions, type ClusterTURNServer} from '@lakutata/webrtc'
|
|
970
|
+
|
|
971
|
+
//The members whose id starts with eu- get the TURN servers of the nodes labeled {region: 'eu'}, and so on
|
|
972
|
+
const signaling = buildSignalingOptions({
|
|
973
|
+
cluster: 'cluster',
|
|
974
|
+
turnSecret: process.env.TURN_SECRET,
|
|
975
|
+
selectTURN: (servers: ClusterTURNServer[], user: string): ClusterTURNServer[] => {
|
|
976
|
+
return servers.filter((server: ClusterTURNServer): boolean => user.startsWith(`${server.labels.region}-`))
|
|
977
|
+
}
|
|
978
|
+
})
|
|
979
|
+
```
|
|
980
|
+
|
|
981
|
+
### The media servers of the cluster
|
|
982
|
+
|
|
983
|
+
Given `cluster`, the media servers of the nodes share the rooms: the id of each one is `sfu:` followed by the id of its
|
|
984
|
+
node, and each member is connected to the media server of one node.
|
|
985
|
+
|
|
986
|
+
- A room stays on as few nodes as it can. Its first member goes to the node it joins through, or to the least loaded
|
|
987
|
+
node when that one is full; the next members go to the least loaded node the room already has, and to the least
|
|
988
|
+
loaded node of all once those are full. A load balancer spreads the clients over the nodes at random: without this
|
|
989
|
+
rule, a meeting of three could take three nodes.
|
|
990
|
+
- `selectSFU` chooses the node instead, the closest to the member for example; when the node it chooses is full or
|
|
991
|
+
does not answer, the default choice is made.
|
|
992
|
+
- A node with the signaling but no media server asks the least loaded node that has one, which makes the choice
|
|
993
|
+
(`selectSFU` runs there).
|
|
994
|
+
- `maxMembers` is the limit of each node: a member joining when no node takes it is refused
|
|
995
|
+
(`SignalingLimitException`). The streams of WHIP and the viewers of WHEP are given a node the same way (the viewers
|
|
996
|
+
are not counted by `maxMembers`: `maxSessions` of WHEP limits them).
|
|
997
|
+
- The events of each media server and its `rooms()` cover the members of its own node.
|
|
998
|
+
|
|
999
|
+
```typescript
|
|
1000
|
+
import {buildSFUOptions, type SFUNode, type SFUPlacement} from '@lakutata/webrtc'
|
|
1001
|
+
|
|
1002
|
+
//The members whose id starts with eu- go to the least loaded node labeled {region: 'eu'}, and so on
|
|
1003
|
+
const sfu = buildSFUOptions({
|
|
1004
|
+
cluster: 'cluster',
|
|
1005
|
+
selectSFU: (placement: SFUPlacement, nodes: SFUNode[]): string | undefined => {
|
|
1006
|
+
const region: string = placement.peerId.split('-')[0]
|
|
1007
|
+
return nodes
|
|
1008
|
+
.filter((node: SFUNode): boolean => node.labels.region === region)
|
|
1009
|
+
.sort((a: SFUNode, b: SFUNode): number => a.members - b.members)[0]?.id
|
|
1010
|
+
}
|
|
1011
|
+
})
|
|
1012
|
+
```
|
|
1013
|
+
|
|
1014
|
+
The media servers of a room forward its tracks to each other. A track goes once to each node where someone receives
|
|
1015
|
+
it (a member selected for it, a viewer, `sfu.track()`), however many receive it there, straight from the node of its
|
|
1016
|
+
publisher. The nodes connect to each other as the clients do, with WebRTC connections (DTLS-SRTP) over their
|
|
1017
|
+
`icePortRange`. A data channel between two members of different nodes goes through both media servers.
|
|
1018
|
+
|
|
1019
|
+
When a node goes down (it stops, or the others miss three of its heartbeats, about 6 seconds), its media server leaves
|
|
1020
|
+
its rooms. Its members join their rooms again, through the signaling, are given the media server of another node and
|
|
1021
|
+
publish their tracks again; the others see their tracks stop for a few seconds. The peers of a member that moved close
|
|
1022
|
+
and open again on every side, as when a member reloads its page: the listeners of `peer` create their channels again.
|
|
1023
|
+
When every node is full at that moment, the client does not give up as a first `join()` does: it tries again after 1
|
|
1024
|
+
second, then twice as long each time (10 seconds at most), its signaling emitting `error` after each failure, until a
|
|
1025
|
+
node takes the member or the room is left.
|
|
1026
|
+
|
|
1027
|
+
A session of WHIP or WHEP stays on the node that answered its POST, and its connection goes to the node chosen for it.
|
|
1028
|
+
Its PATCH and its DELETE may reach any node; a PATCH reaching a node without the session waits for the answers of all
|
|
1029
|
+
the nodes, 12 seconds at most. A viewer receives by default the stream of WHIP its node learned of last.
|
|
1030
|
+
|
|
1031
|
+
### WHIP and WHEP behind a load balancer
|
|
1032
|
+
|
|
1033
|
+
A session of WHIP or WHEP lives on the node that answered its POST. Given `cluster`, the `WHIP` or `WHEP` component of
|
|
1034
|
+
another node receiving its DELETE or its PATCH asks the other nodes, and the one holding the session applies it.
|
|
1035
|
+
`maxSessions` counts the sessions of each node.
|
|
1036
|
+
|
|
1037
|
+
### Kubernetes and k3s
|
|
1038
|
+
|
|
1039
|
+
The manifests below deploy NATS, the signaling (two pods), and a TURN server and a media server on each node. The
|
|
1040
|
+
application is one image for the three roles, chosen by the variable `ROLE`:
|
|
1041
|
+
|
|
1042
|
+
```typescript
|
|
1043
|
+
import {Application, type ApplicationOptions} from 'lakutata'
|
|
1044
|
+
import {buildEntrypoints} from 'lakutata/com/entrypoint'
|
|
1045
|
+
import {socketIOServiceEntrypoint} from 'lakutata/adapter/socket.io'
|
|
1046
|
+
import {buildClusterOptions, buildSFUOptions, buildSignalingOptions, buildTURNServerOptions, SignalingController} from '@lakutata/webrtc'
|
|
1047
|
+
|
|
1048
|
+
const cluster = buildClusterOptions({transport: {type: 'nats', servers: process.env.NATS_URL}})
|
|
1049
|
+
const signaling = buildSignalingOptions({cluster: 'cluster', turnSecret: process.env.TURN_SECRET})
|
|
1050
|
+
const roles: Record<string, ApplicationOptions['components']> = {
|
|
1051
|
+
signaling: {
|
|
1052
|
+
cluster: cluster,
|
|
1053
|
+
signaling: signaling,
|
|
1054
|
+
entrypoint: buildEntrypoints({
|
|
1055
|
+
controllers: [SignalingController],
|
|
1056
|
+
service: socketIOServiceEntrypoint({port: 3000})
|
|
1057
|
+
})
|
|
1058
|
+
},
|
|
1059
|
+
//On the network of the node: its address, and its ports 3478 and 49160-49200
|
|
1060
|
+
turn: {
|
|
1061
|
+
cluster: cluster,
|
|
1062
|
+
turn: buildTURNServerOptions({
|
|
1063
|
+
cluster: 'cluster',
|
|
1064
|
+
secret: process.env.TURN_SECRET,
|
|
1065
|
+
publicAddress: process.env.PUBLIC_ADDRESS,
|
|
1066
|
+
relayPortRange: [49160, 49200]
|
|
1067
|
+
})
|
|
1068
|
+
},
|
|
1069
|
+
//On the network of the node too, its ports 40000-40999; its signaling has no entrypoint: the members join through
|
|
1070
|
+
//the pods of the signaling, which give each of them the media server of a node
|
|
1071
|
+
sfu: {
|
|
1072
|
+
cluster: cluster,
|
|
1073
|
+
signaling: signaling,
|
|
1074
|
+
sfu: buildSFUOptions({cluster: 'cluster', icePortRange: [40000, 40999], workers: 2})
|
|
1075
|
+
}
|
|
1076
|
+
}
|
|
1077
|
+
const components: ApplicationOptions['components'] = roles[process.env.ROLE ?? 'signaling']
|
|
1078
|
+
|
|
1079
|
+
Application.run(() => ({
|
|
1080
|
+
id: 'meet.app',
|
|
1081
|
+
name: 'Meet',
|
|
1082
|
+
components: components,
|
|
1083
|
+
bootstrap: Object.keys(components ?? {})
|
|
1084
|
+
}))
|
|
1085
|
+
```
|
|
1086
|
+
|
|
1087
|
+
```yaml
|
|
1088
|
+
apiVersion: v1
|
|
1089
|
+
kind: Secret
|
|
1090
|
+
metadata:
|
|
1091
|
+
name: turn
|
|
1092
|
+
stringData:
|
|
1093
|
+
secret: change-me
|
|
1094
|
+
---
|
|
1095
|
+
apiVersion: apps/v1
|
|
1096
|
+
kind: Deployment
|
|
1097
|
+
metadata:
|
|
1098
|
+
name: nats
|
|
1099
|
+
spec:
|
|
1100
|
+
replicas: 1
|
|
1101
|
+
selector:
|
|
1102
|
+
matchLabels:
|
|
1103
|
+
app: nats
|
|
1104
|
+
template:
|
|
1105
|
+
metadata:
|
|
1106
|
+
labels:
|
|
1107
|
+
app: nats
|
|
1108
|
+
spec:
|
|
1109
|
+
containers:
|
|
1110
|
+
- name: nats
|
|
1111
|
+
image: nats:latest
|
|
1112
|
+
ports:
|
|
1113
|
+
- containerPort: 4222
|
|
1114
|
+
---
|
|
1115
|
+
apiVersion: v1
|
|
1116
|
+
kind: Service
|
|
1117
|
+
metadata:
|
|
1118
|
+
name: nats
|
|
1119
|
+
spec:
|
|
1120
|
+
selector:
|
|
1121
|
+
app: nats
|
|
1122
|
+
ports:
|
|
1123
|
+
- port: 4222
|
|
1124
|
+
---
|
|
1125
|
+
apiVersion: apps/v1
|
|
1126
|
+
kind: Deployment
|
|
1127
|
+
metadata:
|
|
1128
|
+
name: signaling
|
|
1129
|
+
spec:
|
|
1130
|
+
replicas: 2
|
|
1131
|
+
selector:
|
|
1132
|
+
matchLabels:
|
|
1133
|
+
app: signaling
|
|
1134
|
+
template:
|
|
1135
|
+
metadata:
|
|
1136
|
+
labels:
|
|
1137
|
+
app: signaling
|
|
1138
|
+
spec:
|
|
1139
|
+
containers:
|
|
1140
|
+
- name: signaling
|
|
1141
|
+
image: registry.example.com/meet:1.0.0
|
|
1142
|
+
env:
|
|
1143
|
+
- name: ROLE
|
|
1144
|
+
value: signaling
|
|
1145
|
+
- name: NATS_URL
|
|
1146
|
+
value: nats://nats:4222
|
|
1147
|
+
- name: TURN_SECRET
|
|
1148
|
+
valueFrom:
|
|
1149
|
+
secretKeyRef:
|
|
1150
|
+
name: turn
|
|
1151
|
+
key: secret
|
|
1152
|
+
ports:
|
|
1153
|
+
- containerPort: 3000
|
|
1154
|
+
---
|
|
1155
|
+
apiVersion: v1
|
|
1156
|
+
kind: Service
|
|
1157
|
+
metadata:
|
|
1158
|
+
name: signaling
|
|
1159
|
+
spec:
|
|
1160
|
+
selector:
|
|
1161
|
+
app: signaling
|
|
1162
|
+
ports:
|
|
1163
|
+
- port: 3000
|
|
1164
|
+
---
|
|
1165
|
+
apiVersion: apps/v1
|
|
1166
|
+
kind: DaemonSet
|
|
1167
|
+
metadata:
|
|
1168
|
+
name: turn
|
|
1169
|
+
spec:
|
|
1170
|
+
selector:
|
|
1171
|
+
matchLabels:
|
|
1172
|
+
app: turn
|
|
1173
|
+
template:
|
|
1174
|
+
metadata:
|
|
1175
|
+
labels:
|
|
1176
|
+
app: turn
|
|
1177
|
+
spec:
|
|
1178
|
+
hostNetwork: true
|
|
1179
|
+
dnsPolicy: ClusterFirstWithHostNet
|
|
1180
|
+
containers:
|
|
1181
|
+
- name: turn
|
|
1182
|
+
image: registry.example.com/meet:1.0.0
|
|
1183
|
+
env:
|
|
1184
|
+
- name: ROLE
|
|
1185
|
+
value: turn
|
|
1186
|
+
- name: NATS_URL
|
|
1187
|
+
value: nats://nats:4222
|
|
1188
|
+
- name: PUBLIC_ADDRESS
|
|
1189
|
+
valueFrom:
|
|
1190
|
+
fieldRef:
|
|
1191
|
+
fieldPath: status.hostIP
|
|
1192
|
+
- name: TURN_SECRET
|
|
1193
|
+
valueFrom:
|
|
1194
|
+
secretKeyRef:
|
|
1195
|
+
name: turn
|
|
1196
|
+
key: secret
|
|
1197
|
+
---
|
|
1198
|
+
apiVersion: apps/v1
|
|
1199
|
+
kind: DaemonSet
|
|
1200
|
+
metadata:
|
|
1201
|
+
name: sfu
|
|
1202
|
+
spec:
|
|
1203
|
+
selector:
|
|
1204
|
+
matchLabels:
|
|
1205
|
+
app: sfu
|
|
1206
|
+
template:
|
|
1207
|
+
metadata:
|
|
1208
|
+
labels:
|
|
1209
|
+
app: sfu
|
|
1210
|
+
spec:
|
|
1211
|
+
hostNetwork: true
|
|
1212
|
+
dnsPolicy: ClusterFirstWithHostNet
|
|
1213
|
+
containers:
|
|
1214
|
+
- name: sfu
|
|
1215
|
+
image: registry.example.com/meet:1.0.0
|
|
1216
|
+
env:
|
|
1217
|
+
- name: ROLE
|
|
1218
|
+
value: sfu
|
|
1219
|
+
- name: NATS_URL
|
|
1220
|
+
value: nats://nats:4222
|
|
1221
|
+
- name: TURN_SECRET
|
|
1222
|
+
valueFrom:
|
|
1223
|
+
secretKeyRef:
|
|
1224
|
+
name: turn
|
|
1225
|
+
key: secret
|
|
1226
|
+
```
|
|
1227
|
+
|
|
1228
|
+
- The TURN servers and the media servers run on the network of their node (`hostNetwork`): the clients reach them at
|
|
1229
|
+
the address of the node, on its ports. Open UDP and TCP 3478, UDP 49160–49200 and UDP 40000–40999 in the firewall of
|
|
1230
|
+
the nodes; the media servers of the nodes also reach each other on 40000–40999.
|
|
1231
|
+
- `status.hostIP` is the address of the node inside the cluster; it is the right one when the clients reach the nodes
|
|
1232
|
+
at it (a local network, nodes with public addresses). Behind the NAT of a cloud, give `PUBLIC_ADDRESS` the public
|
|
1233
|
+
address of each node instead, the NAT forwarding the same ports, and give the media server that address in
|
|
1234
|
+
`iceAdditionalHostAddresses`.
|
|
1235
|
+
- A pod on the network of its node resolves the names of the services with `dnsPolicy: ClusterFirstWithHostNet`.
|
|
1236
|
+
- The browsers reach the signaling through an Ingress or a load balancer in front of the Service `signaling`, connecting
|
|
1237
|
+
with `transports: ['websocket']`.
|
|
1238
|
+
- A single NATS pod is enough for a test; in production, run a NATS cluster (its Helm chart) or a Redis with a replica.
|
|
1239
|
+
- When the whole cluster starts, a pod may start before the DNS of the cluster resolves `nats` (its log then shows
|
|
1240
|
+
`getaddrinfo EAI_AGAIN nats`): the application stops when it cannot reach NATS as it starts, and Kubernetes starts it
|
|
1241
|
+
again until it can. A few restarts of each pod at that moment are expected.
|
|
1242
|
+
|
|
1243
|
+
In the repository, `packages/webrtc/test-env/k3s/run.sh` deploys these manifests in k3s in Docker (two nodes, the code
|
|
1244
|
+
mounted instead of an image). It connects two members through two pods and the TURN servers only, then two members
|
|
1245
|
+
through the media servers of the two nodes, and deletes the pod of one of them: that member moves to the pod started
|
|
1246
|
+
in its place.
|
|
1247
|
+
|
|
1248
|
+
### The cluster for the application
|
|
1249
|
+
|
|
1250
|
+
The application talks to its other instances through the same component: messages to all the other nodes, and requests
|
|
1251
|
+
to one node or to all of them.
|
|
1252
|
+
|
|
1253
|
+
```typescript
|
|
1254
|
+
import {Component} from 'lakutata'
|
|
1255
|
+
import type {Cluster, ClusterReply} from '@lakutata/webrtc'
|
|
1256
|
+
|
|
1257
|
+
export class Viewers extends Component {
|
|
1258
|
+
//The viewers of each stream on this node, counted by the application
|
|
1259
|
+
protected readonly counts: Map<string, number> = new Map()
|
|
1260
|
+
|
|
1261
|
+
protected async init(): Promise<void> {
|
|
1262
|
+
const cluster: Cluster = await this.getObject<Cluster>('cluster')
|
|
1263
|
+
//Each node answers with its own count
|
|
1264
|
+
cluster.handle<{stream: string}, number>('viewers', ({stream}: {stream: string}): number => this.counts.get(stream) ?? 0)
|
|
1265
|
+
cluster.subscribe<{text: string}>('announcements', (message: {text: string}): void => console.log(message.text))
|
|
1266
|
+
}
|
|
1267
|
+
|
|
1268
|
+
//The viewers of a stream on all the nodes
|
|
1269
|
+
public async total(stream: string): Promise<number> {
|
|
1270
|
+
const cluster: Cluster = await this.getObject<Cluster>('cluster')
|
|
1271
|
+
const replies: ClusterReply<number>[] = await cluster.requestAll<number>('viewers', {stream: stream})
|
|
1272
|
+
return replies.reduce((sum: number, reply: ClusterReply<number>): number => sum + reply.result, this.counts.get(stream) ?? 0)
|
|
1273
|
+
}
|
|
1274
|
+
}
|
|
1275
|
+
```
|
|
1276
|
+
|
|
1277
|
+
`cluster.nodes()` lists the nodes with their labels and what they announce (`announce()`), and the component emits
|
|
1278
|
+
`node-up`, `node-down` and `node-update`. The messages are sent as JSON; a message larger than the server carries (1 MiB
|
|
1279
|
+
by default on NATS) is sent in parts.
|
|
1280
|
+
|
|
1281
|
+
## 14. The signaling protocol
|
|
1282
|
+
|
|
1283
|
+
Clients in other languages and native applications speak the protocol directly. The requests are service calls whose
|
|
1284
|
+
pattern is `{method}`: over Socket.IO, the event `message` with `{method: 'webrtc.join', room: 'room-1'}` and an
|
|
1285
|
+
acknowledgement (`{success, error, payload}`); over JSON-RPC 2.0, a request whose method is `webrtc.join`. The server
|
|
1286
|
+
pushes the events to the members: Socket.IO events, or JSON-RPC notifications whose `params` hold the payload.
|
|
1287
|
+
|
|
1288
|
+
| Request | Parameters | Answer |
|
|
1289
|
+
|---|---|---|
|
|
1290
|
+
| `webrtc.join` | `room`, `metadata`, `session` | `{v, peerId, peers, iceServers, credentialTtl, heartbeatInterval, sfu}` |
|
|
1291
|
+
| `webrtc.signal` | `room`, `to`, and `description` (`{type, sdp}`) or `candidate` (`{candidate, sdpMid, sdpMLineIndex, usernameFragment}`) | `{delivered}` |
|
|
1292
|
+
| `webrtc.leave` | `room` | `{}` |
|
|
1293
|
+
| `webrtc.iceServers` | — | `{iceServers, credentialTtl}`: new credentials |
|
|
1294
|
+
| `webrtc.heartbeat` | — | `{}` |
|
|
1295
|
+
|
|
1296
|
+
| Event | Payload |
|
|
1297
|
+
|---|---|
|
|
1298
|
+
| `webrtc:peer-joined` | `{room, peerId, metadata, session, role, sfu}`: a member joined, or joined again |
|
|
1299
|
+
| `webrtc:peer-left` | `{room, peerId, reason}`: `left` when it asked, `disconnected` when its client went away |
|
|
1300
|
+
| `webrtc:signal` | `{room, from, session, description, streams}` or `{room, from, session, candidate}` |
|
|
1301
|
+
|
|
1302
|
+
- `session` is a random string a client chooses once. A member joining again with the same session (after a
|
|
1303
|
+
reconnection of its signaling) keeps the connections of the others; a new session (a reloaded page) replaces them.
|
|
1304
|
+
- `heartbeatInterval` is 0 over Socket.IO and JSON-RPC over WebSocket: the server sees their connections close. Over
|
|
1305
|
+
another transport, the client sends `webrtc.heartbeat` that often; after three intervals without a request, it leaves
|
|
1306
|
+
its rooms.
|
|
1307
|
+
- The member whose id comes first (by string comparison) is the polite one of a pair: when both send an offer at the
|
|
1308
|
+
same time, it gives up its own and answers the other one.
|
|
1309
|
+
- The fields of the media server, all set by the server: `role` is `sfu` for a media server and `whip` for a stream
|
|
1310
|
+
of WHIP (none for the clients); `sfu` is the id of the media server a member is connected to, and in the answer to
|
|
1311
|
+
`webrtc.join`, the one the client connects to. A client in a room with a media server connects to it only, and is
|
|
1312
|
+
the polite one with it. The descriptions of the media server carry `streams`: for each stream it sends, its `id` on
|
|
1313
|
+
that connection, the `peerId` of the member it comes from, and its id on the connection of that member (`stream`).
|
|
1314
|
+
- On the connection to the media server, the data channel 1023 (label `webrtc.control`, created by both sides with
|
|
1315
|
+
`negotiated: true` and this `id`) carries `{type: 'pause' | 'resume', mid}` in JSON: the server stops or starts
|
|
1316
|
+
sending the track of that transceiver. A data channel to a member through the server has the id of that member as
|
|
1317
|
+
its `protocol`; the server opens the channel on the side of that member with the id of the sender. The media servers
|
|
1318
|
+
of the nodes use the channel 1023 between them too, and channels whose `protocol` is `["sender", "member"]` in JSON.
|
|
1319
|
+
- The calls of `webrtcServiceEntrypoint()` (section 5) are JSON-RPC 2.0 messages on a data channel labeled `service`,
|
|
1320
|
+
ordered and reliable, which the caller creates. Each message is cut into frames as `ChunkedChannel` does: a frame is
|
|
1321
|
+
a binary message whose first byte holds flags (1 for the last frame of a message, 2 for a text message) and whose
|
|
1322
|
+
other bytes are the next part of the message (16 KiB at most from this package). A call is a text message: its
|
|
1323
|
+
frames carry the flags 2, and the last one 3. The responses and the events (notifications whose `params` are the
|
|
1324
|
+
arguments) come back the same way.
|
|
1325
|
+
- The errors carry the `errno` of the server's exception: `E_SIGNALING_DENIED`, `E_SIGNALING_LIMIT` (a full room, or
|
|
1326
|
+
a member no media server takes), `E_SIGNALING_NOT_A_MEMBER`, `E_SIGNALING_CLIENT_REQUIRED`, `E_INVALID_VALUE`.
|
|
1327
|
+
JSON-RPC over HTTP cannot carry the signaling: the events need a connection to be pushed through.
|
|
1328
|
+
|
|
1329
|
+
## 15. When a connection fails
|
|
1330
|
+
|
|
1331
|
+
`setDebugHandler()` gives the debug messages of ICE, DTLS, SCTP and TURN:
|
|
1332
|
+
|
|
1333
|
+
```typescript
|
|
1334
|
+
import {format} from 'node:util'
|
|
1335
|
+
import {setDebugHandler} from '@lakutata/webrtc'
|
|
1336
|
+
|
|
1337
|
+
setDebugHandler((namespace: string, args: unknown[]): void => console.debug(namespace, format(...args)))
|
|
1338
|
+
```
|
|
1339
|
+
|
|
1340
|
+
The worker threads of a media server send theirs to the same handler, each message starting with the number of its
|
|
1341
|
+
thread (`[thread 0] …`). A `select` of the media server or of WHEP that throws is reported under `webrtc/sfu` and
|
|
1342
|
+
`webrtc/whep`.
|
|
1343
|
+
|
|
1344
|
+
To check the TURN server, join a room from two clients with `iceTransportPolicy: 'relay'`: they connect only through
|
|
1345
|
+
the relays, and `turn.allocations` counts them. In Chrome, `chrome://webrtc-internals` shows the candidates and the
|
|
1346
|
+
pairs tried.
|
|
1347
|
+
|
|
1348
|
+
## 16. Limits
|
|
1349
|
+
|
|
1350
|
+
- Node.js does not capture, encode or decode audio and video: its tracks carry RTP packets, from an encoder or from
|
|
1351
|
+
another connection.
|
|
1352
|
+
- The throughput of one data channel between two Node.js processes on an Apple M4 is about 80 MiB/s for large
|
|
1353
|
+
messages, and 626,000 messages of 100 bytes per second; in an arm64 Linux virtual machine (Docker Desktop) on the
|
|
1354
|
+
same Mac, about 43 MiB/s, and 548,000 messages a second. `packages/webrtc/test-env/bench/engine.mjs` in the
|
|
1355
|
+
repository measures it on another machine.
|
|
1356
|
+
- The media server does not transcode, and sends each video as it is published: there is no simulcast yet, so a
|
|
1357
|
+
member with little bandwidth gets `congestion` events rather than a lower quality. Its capacity on one machine is in
|
|
1358
|
+
section 9; `packages/webrtc/test-env/bench/forward.mjs` in the repository measures it on another.
|
|
1359
|
+
- The media servers of a cluster send a track straight from the node of its publisher to each node wanting it, which
|
|
1360
|
+
suits rooms spread over tens of nodes; beyond that, the nodes would have to forward through each other.
|
|
1361
|
+
- Verified in October 2026 with Chrome 154, Firefox 153 and WebKit 26.5 (the builds of Playwright 1.62), Safari
|
|
1362
|
+
27.0.1 on macOS 27, Pion 4.2.25 (Go) and aiortc 1.15.0 (Python), on Node.js 24 (macOS, and Linux on Alpine), Bun
|
|
1363
|
+
1.4.3 (macOS, and Linux on Debian) and the main process of Electron 44. The scripts are in `packages/webrtc/test-env`
|
|
1364
|
+
of the repository.
|