@twilio/video-node-sdk 1.0.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,609 @@
1
+ # @twilio/video-node-sdk
2
+
3
+ [![CI](https://github.com/twilio/twilio-video-node/actions/workflows/ci.yml/badge.svg)](https://github.com/twilio/twilio-video-node/actions/workflows/ci.yml)
4
+
5
+ Server-side Node.js SDK for Twilio Video Group Rooms with raw media frame access. Built on a native C++ addon over WebRTC, it lets you push and receive decoded video and audio frames from Node.js on realtime.
6
+
7
+ **Note: this is a beta release of the Twilio Media SDK for Node.js. It is provided for evaluation purposes only and should not be used with production traffic. During the beta period this SDK is not HIPAA eligible.**
8
+
9
+ ## Installation
10
+
11
+ Install the provided `.tgz` file directly:
12
+
13
+ ```bash
14
+ npm install ./twilio-video-node-sdk-<version>.tgz
15
+ ```
16
+
17
+ The native binary is prebuilt and bundled — no build step required. Import it as `@twilio/video-node-sdk`.
18
+
19
+ **Requirements:**
20
+
21
+ - Node.js >= 24.0.0
22
+ - Linux x86-64, glibc >= 2.34 (Ubuntu 22.04+, Debian 12+)
23
+
24
+ Linux x86-64 is the only supported platform for the beta. Alpine/musl, arm64 and Windows are not supported, and macOS is development-only. See [Platform Support](#platform-support) for the full ABI requirements.
25
+
26
+ ### Access Token
27
+
28
+ The `connect()` function takes a standard Twilio Video Access Token with a
29
+ VideoGrant, the same token format used by the JavaScript SDK. Generate one
30
+ using the [`twilio`](https://www.npmjs.com/package/twilio) helper library. See
31
+ [User Identity and Access Tokens](https://www.twilio.com/docs/video/tutorials/user-identity-access-tokens)
32
+ for details.
33
+
34
+ ## Quick Start
35
+
36
+ ```js
37
+ const { connect, createLocalVideoTrack } = require('@twilio/video-node-sdk');
38
+
39
+ async function main() {
40
+ const videoTrack = createLocalVideoTrack('my-camera');
41
+
42
+ const room = await connect(token, {
43
+ name: 'my-room',
44
+ videoTracks: [videoTrack],
45
+ });
46
+
47
+ console.log('Connected:', room.name, room.sid);
48
+
49
+ // Push I420 video frames. connect() resolves only once the Room is connected;
50
+ // frames written before that are dropped and counted in getWriteStats().
51
+ videoTrack.write({
52
+ format: 'I420',
53
+ width: 1280,
54
+ height: 720,
55
+ y: { data: yPlane, stride: 1280, width: 1280, height: 720 },
56
+ u: { data: uPlane, stride: 640, width: 640, height: 360 },
57
+ v: { data: vPlane, stride: 640, width: 640, height: 360 },
58
+ });
59
+
60
+ async function trackSubscribed(track) {
61
+ if (track.kind !== 'video') return;
62
+ // Awaiting each frame is the backpressure. The loop ends by itself when the
63
+ // track is unsubscribed or the Room disconnects.
64
+ for await (const frame of track.frames()) {
65
+ console.log(`${frame.width}x${frame.height} @ ${frame.timestamp}us`);
66
+ frame.close?.();
67
+ }
68
+ }
69
+
70
+ function participantConnected(participant) {
71
+ participant.on('trackSubscribed', trackSubscribed);
72
+
73
+ participant.tracks.forEach(publication => {
74
+ if (publication.isSubscribed) {
75
+ trackSubscribed(publication.track);
76
+ }
77
+ });
78
+ }
79
+
80
+ // participantConnected does not fire for participants already in the Room, and a
81
+ // track can finish subscribing before this listener is attached. Seed from
82
+ // room.participants and check isSubscribed on the publications found there.
83
+ room.participants.forEach(participantConnected);
84
+ room.on('participantConnected', participantConnected);
85
+
86
+ room.on('disconnected', () => {
87
+ room.dispose();
88
+ });
89
+ }
90
+
91
+ main().catch(err => {
92
+ console.error('Error:', err);
93
+ process.exit(1);
94
+ });
95
+ ```
96
+
97
+ ## Releasing a Room
98
+
99
+ Call `room.dispose()` when you are done with a Room. Until you do, the process
100
+ does not exit on its own. `disconnect()` leaves the session but does not release
101
+ the native resources behind it.
102
+
103
+ The `disconnected` event is the clearest place to dispose, as in the quick
104
+ start.
105
+
106
+ For what each object owns, what teardown releases, and the ordering the SDK
107
+ guarantees during teardown, see [LIFECYCLE.md](LIFECYCLE.md).
108
+
109
+ ## Differences from the JavaScript SDK
110
+
111
+ This SDK shares the same Room/Participant/Track model and event names as the
112
+ [Twilio Video JavaScript SDK](https://www.twilio.com/docs/video/javascript),
113
+ but is designed for server-side media processing rather than browser-based
114
+ conferencing. Key differences:
115
+
116
+ - **No device capture.** `createLocalVideoTrack()` and `createLocalAudioTrack()`
117
+ return pushable tracks with no media constraints. You supply raw frames via
118
+ `track.write()` instead of capturing from a camera or microphone.
119
+
120
+ - **No rendering.** There is no `track.attach(element)`. Remote media arrives as
121
+ raw decoded frames (I420 video, PCM audio) through
122
+ `for await (const frame of track.frames())`.
123
+
124
+ - **Fixed audio input format.** `LocalAudioTrack.write()` accepts only 48 kHz
125
+ mono S16LE PCM. Received audio may vary in sample rate and channel count.
126
+
127
+ - **No adaptive simulcast or track priority.** Published video tracks always use
128
+ standard priority with simulcast disabled; the deprecated `TrackPriority` API
129
+ is not exposed. Bandwidth and remote render-size hints are configurable
130
+ instead via `bandwidthProfile`, `encodingParameters`, and
131
+ `RemoteVideoTrack.setContentPreferences()`.
132
+
133
+ - **Synchronous track creation.** `createLocalVideoTrack()` returns a track
134
+ immediately (no async device permissions). Tracks can be passed to `connect()`
135
+ or published later via `localParticipant.publishTrack()`.
136
+
137
+ ## API Overview
138
+
139
+ ### Top-level Functions
140
+
141
+ | Function | Description |
142
+ | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
143
+ | `connect(token, options?)` | Connect to a room. Returns `Promise<Room>`. |
144
+ | `createLocalVideoTrack(name?)` | Create a pushable local video track. |
145
+ | `createLocalAudioTrack(name?)` | Create a pushable local audio track. |
146
+ | `createLocalDataTrack(name \| options?)` | Create a local data track. Options are `name`, `ordered`, and one of `maxPacketLifeTime`/`maxRetransmits`. |
147
+ | `createLocalTracks(options?)` | Create local audio and/or video tracks. With no options, returns both. If either `audio` or `video` is specified, the other defaults to `false`. Each key accepts `true`/`false` or a per-track options object (e.g. `{ name }`). Returns `Promise<(LocalAudioTrack \| LocalVideoTrack)[]>`. |
148
+ | `twilioErrorFromCode(code, message?)` | Build a `TwilioError` (or matching subclass) from a numeric error code. |
149
+ | `setLogLevel(level)` | Set native log level. Accepts a name (`'off'` \| `'fatal'` \| `'error'` \| `'warning'` \| `'info'` \| `'debug'` \| `'trace'` \| `'all'`) or the equivalent number `0` (off) through `7` (all). |
150
+ | `getVersion()` | Returns the native SDK version string. |
151
+ | `MAX_QUEUE_CEILING` | Upper bound (`1024`) accepted for any `maxQueue`, on `frames()` and on `source.maxQueue`. |
152
+ | `SDK_LOCAL_CODE` | The `code` (`0`) carried by errors the SDK raises locally, which Twilio never assigns a code to. Match on the error class instead. |
153
+
154
+ ### Key Classes
155
+
156
+ | Export | Description |
157
+ | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
158
+ | `Room` | A connected video room. Emits events, exposes participants. |
159
+ | `LocalParticipant` | The local participant. Publish/unpublish tracks. |
160
+ | `RemoteParticipant` | A remote participant. Emits `trackSubscribed`/`trackUnsubscribed`. |
161
+ | `LocalVideoTrack` | Pushable video track (`write(frame)`). |
162
+ | `LocalAudioTrack` | Pushable audio track (`write(frame)`, `clearBuffer()`). |
163
+ | `LocalDataTrack` | Send arbitrary data (`send`). Create via `createLocalDataTrack(name \| options?)`. |
164
+ | `RemoteVideoTrack` | Receive video frames (`frames()`), `getStats()`, `frameDropped` event. |
165
+ | `RemoteAudioTrack` | Receive audio frames (`frames()`), `getStats()`, `frameDropped` event. |
166
+ | `RemoteDataTrack` | Receive data messages (`message` event). |
167
+ | `TrackPublication` | Base class for published tracks (`trackSid`, `trackName`, `kind`, `isTrackEnabled`). |
168
+ | `LocalTrackPublication` | Local publication. Exposes `track` and `unpublish()`. Subclassed per kind (`LocalVideoTrackPublication`, …). |
169
+ | `RemoteTrackPublication` | Remote publication. Exposes `track` and `isSubscribed`. Subclassed per kind (`RemoteVideoTrackPublication`, …). |
170
+ | `TwilioError` | Base error class, carrying a numeric `code`. One subclass per known Twilio code, plus SDK-local errors (`NativeBindingLoadError`, `RoomConnectTimeoutError`, …). |
171
+ | `ErrorCode` | Enum of Twilio Video error codes. |
172
+
173
+ ## Room Events
174
+
175
+ ### Room state at connect
176
+
177
+ `participantConnected` is not emitted for participants who were already in the Room when
178
+ `connect()` resolved. They are part of the Room's starting state: read them from
179
+ `room.participants`.
180
+
181
+ A participant who was already publishing emits `trackSubscribed` after `connect()` resolves.
182
+ Subscriptions that completed before the listener was attached are not replayed, and appear in
183
+ `participant.tracks` with `isSubscribed` set to `true`.
184
+
185
+ | Event | Handler Signature |
186
+ | ------------------------- | -------------------------------------------------- |
187
+ | `disconnected` | `(room: Room, error?: TwilioError) => void` |
188
+ | `connectFailure` | `(error: TwilioError) => void` |
189
+ | `reconnecting` | `(error?: TwilioError) => void` |
190
+ | `reconnected` | `() => void` |
191
+ | `participantConnected` | `(participant: RemoteParticipant) => void` |
192
+ | `participantDisconnected` | `(participant: RemoteParticipant) => void` |
193
+ | `participantReconnecting` | `(participant: RemoteParticipant) => void` |
194
+ | `participantReconnected` | `(participant: RemoteParticipant) => void` |
195
+ | `recordingStarted` | `() => void` |
196
+ | `recordingStopped` | `() => void` |
197
+ | `dominantSpeakerChanged` | `(participant: RemoteParticipant \| null) => void` |
198
+ | `transcription` | `(transcriptionJson: string) => void` |
199
+
200
+ The Room re-emits every track event in [RemoteParticipant Events](#remoteparticipant-events),
201
+ appending the `RemoteParticipant` that emitted it as the last argument. Handle every
202
+ participant's tracks from one place instead of attaching a listener to each participant.
203
+
204
+ ### RemoteParticipant Events
205
+
206
+ | Event | Handler Signature |
207
+ | ---------------------------- | --------------------------------------------------------------------------------------------------------------- |
208
+ | `trackSubscribed` | `(track: RemoteVideoTrack \| RemoteAudioTrack \| RemoteDataTrack, publication: RemoteTrackPublication) => void` |
209
+ | `trackUnsubscribed` | `(track: RemoteVideoTrack \| RemoteAudioTrack \| RemoteDataTrack, publication: RemoteTrackPublication) => void` |
210
+ | `trackSubscriptionFailed` | `(error: TwilioError, publication: RemoteTrackPublication) => void` |
211
+ | `trackPublished` | `(publication: RemoteTrackPublication) => void` |
212
+ | `trackUnpublished` | `(publication: RemoteTrackPublication) => void` |
213
+ | `trackEnabled` | `(publication: RemoteTrackPublication) => void` |
214
+ | `trackDisabled` | `(publication: RemoteTrackPublication) => void` |
215
+ | `videoTrackSwitchedOff` | `(track: RemoteVideoTrack) => void` |
216
+ | `videoTrackSwitchedOn` | `(track: RemoteVideoTrack) => void` |
217
+ | `networkQualityLevelChanged` | `(level: number) => void` |
218
+
219
+ ### LocalParticipant Events
220
+
221
+ | Event | Handler Signature |
222
+ | ---------------------------- | ------------------------------------------------------- |
223
+ | `trackPublished` | `(publication: LocalTrackPublication) => void` |
224
+ | `trackPublicationFailed` | `(error: TwilioError, localTrack?: LocalTrack) => void` |
225
+ | `networkQualityLevelChanged` | `(level: number) => void` |
226
+
227
+ ### TrackPublication
228
+
229
+ `LocalTrackPublication` exposes `track` (the local track instance) and an `unpublish()` method:
230
+
231
+ ```js
232
+ const pub = room.localParticipant.tracks.get(trackSid); // LocalTrackPublication
233
+ pub.unpublish(); // unpublishes the underlying track
234
+ ```
235
+
236
+ `RemoteTrackPublication` exposes `track` (the subscribed remote track, if any) and `isSubscribed`.
237
+
238
+ ### Room.getStats()
239
+
240
+ Returns a snapshot of WebRTC stats per peer connection. Rejects if the room is disconnected.
241
+
242
+ ```js
243
+ const reports = await room.getStats();
244
+ // reports[i]: {
245
+ // peerConnectionId, localAudioTrackStats, localVideoTrackStats,
246
+ // remoteAudioTrackStats, remoteVideoTrackStats
247
+ // }
248
+ ```
249
+
250
+ ## Track Types
251
+
252
+ ### LocalVideoTrack
253
+
254
+ Push raw I420 video frames into a room. Frames written before `connect()` resolves are dropped, and counted in `getWriteStats()`.
255
+
256
+ ```js
257
+ const track = createLocalVideoTrack('camera');
258
+ track.write({
259
+ format: 'I420', // optional; only 'I420' is accepted
260
+ width,
261
+ height, // both must be positive and even
262
+ y: { data: yBuffer, stride: yStride, width, height },
263
+ u: { data: uBuffer, stride: uStride, width: width / 2, height: height / 2 },
264
+ v: { data: vBuffer, stride: vStride, width: width / 2, height: height / 2 },
265
+ timestamp, // optional microseconds; defaults to monotonic now
266
+ rotation, // optional 0 | 90 | 180 | 270
267
+ });
268
+ track.enabled = false; // mute
269
+ ```
270
+
271
+ Buffers are copied synchronously, so they can be reused as soon as `write()` returns.
272
+
273
+ `write()` returns `false` when the frame was dropped rather than encoded - most often because the encoder sink has not attached yet, but also when libwebrtc's adapter rate-limits or rejects the resolution. It throws `TypeError`/`RangeError` on invalid input.
274
+
275
+ Optionally pin the frame size at creation, so a mismatched frame is rejected instead of silently rescaled:
276
+
277
+ ```js
278
+ const track = createLocalVideoTrack({
279
+ name: 'camera',
280
+ source: { type: 'raw', format: 'I420', width: 1280, height: 720, fps: 30 },
281
+ });
282
+ ```
283
+
284
+ Publish-side counters:
285
+
286
+ ```js
287
+ const { framesWritten, framesDropped, sendQueueDepth, maxQueue, lastTimestamp } =
288
+ track.getWriteStats();
289
+ ```
290
+
291
+ Video publish is synchronous - `write()` hands the frame straight to the encoder - so there is no SDK-side send queue and `sendQueueDepth`/`maxQueue` are always `0`. A `framesDropped` here means the frame was rejected, not shed from a queue.
292
+
293
+ ### LocalAudioTrack
294
+
295
+ Push raw PCM audio samples into a room. Format is fixed to **48kHz mono S16LE**.
296
+
297
+ ```js
298
+ const track = createLocalAudioTrack('mic');
299
+ const accepted = track.write({
300
+ pcm, // Buffer of interleaved int16 samples
301
+ frames, // samples per channel, e.g. 480 for a 10ms chunk
302
+ timestamp, // optional microseconds
303
+ });
304
+ ```
305
+
306
+ Unlike video, audio publish has a real send queue, drained one 10 ms chunk at a
307
+ time. It is bounded (~500 ms by default) so a producer running faster than real
308
+ time cannot accumulate latency. A `write()` is accepted only if it fits whole in
309
+ the remaining space; otherwise nothing is buffered, `write()` returns `false`,
310
+ and the rejection is counted. A single `write()` larger than the bound can never
311
+ fit, so size `maxQueue` to the largest burst you intend to publish:
312
+
313
+ ```js
314
+ const track = createLocalAudioTrack({
315
+ name: 'mic',
316
+ // maxQueue is in 10ms chunks: 200 => ~2s of smoothing. It binds the
317
+ // process-wide audio device, so it applies to every local audio track.
318
+ source: { type: 'raw', format: 'PCM_S16LE', sampleRate: 48000, channels: 1, maxQueue: 200 },
319
+ });
320
+ const { framesWritten, framesDropped, sendQueueDepth, maxQueue } = track.getWriteStats();
321
+ ```
322
+
323
+ Publishing at real-time cadence should never drop. A non-zero `framesDropped`
324
+ means the producer is outrunning the wire, or that a single `write()` was larger
325
+ than `maxQueue`.
326
+
327
+ The `timestamp` on an audio frame is observability-only. Audio publish is FIFO:
328
+ the device emits queued samples on its own 10 ms cadence, so the timestamp feeds
329
+ `lastTimestamp` and `timestampRegressions` and does not change what is sent.
330
+
331
+ `clearBuffer()` discards whatever is still queued and not yet sent. Use it when
332
+ the queued audio has become stale rather than merely late - barge-in, where the
333
+ speaker is interrupted and the rest of the utterance should never play, is the
334
+ usual case. Writes after it resume from an empty queue.
335
+
336
+ ```js
337
+ track.clearBuffer();
338
+ ```
339
+
340
+ ### LocalDataTrack
341
+
342
+ Send arbitrary string or binary messages. Delivery is reliable and ordered by default.
343
+
344
+ ```js
345
+ const track = createLocalDataTrack({ name: 'chat', ordered: true });
346
+ room.localParticipant.publishTrack(track);
347
+ track.send('hello');
348
+ track.send(Buffer.from([0x01, 0x02]));
349
+
350
+ // send() reports the outcome. The promise always resolves - never rejects - so
351
+ // a fire-and-forget send cannot produce an unhandled rejection.
352
+ const result = await track.send('important');
353
+ if (!result.ok) console.warn('send failed:', result.error);
354
+ ```
355
+
356
+ Messages larger than **64 KB** (`kMaxMessageSize`) are rejected synchronously
357
+ with a `RangeError` and never transmitted.
358
+
359
+ Pass `maxPacketLifeTime` (milliseconds) or `maxRetransmits` (a count) to trade reliability for
360
+ latency. The two are mutually exclusive, and each must be an integer from 0 to 65535.
361
+
362
+ ```js
363
+ const telemetry = createLocalDataTrack({ name: 'telemetry', maxPacketLifeTime: 500 });
364
+ telemetry.maxPacketLifeTime; // 500
365
+ telemetry.maxRetransmits; // null
366
+ telemetry.reliable; // false
367
+ ```
368
+
369
+ `maxPacketLifeTime` and `maxRetransmits` are `number | null`, reading back as `null` when the
370
+ limit was not set. `reliable` is `true` only when neither is set.
371
+
372
+ ### RemoteVideoTrack
373
+
374
+ Receive decoded I420 video frames from a remote participant.
375
+
376
+ ```js
377
+ for await (const frame of track.frames()) {
378
+ // frame: {
379
+ // format: 'I420',
380
+ // width, height,
381
+ // y, u, v, // I420Plane: { data: Buffer, stride, width, height }
382
+ // timestamp: number, // microseconds
383
+ // captureTimestamp?: number,
384
+ // rtpTimestamp?: number,
385
+ // frameId: number, // SDK-generated, monotonic per track
386
+ // rotation?: 0 | 90 | 180 | 270,
387
+ // close?(): void, // optional prompt release
388
+ // }
389
+ frame.close?.();
390
+ }
391
+ // The loop ends on unsubscribe or Room disconnect. `break` releases the track.
392
+
393
+ // Hint the desired render dimensions to the SFU. Width/height must be positive
394
+ // integers. Only takes effect when the room was connected with a
395
+ // bandwidthProfile that has `contentPreferencesMode: 'manual'`.
396
+ track.setContentPreferences({ renderDimensions: { width: 320, height: 240 } });
397
+
398
+ // `isSwitchedOff` is `true` when the SFU has stopped delivering this track
399
+ // (e.g. due to bandwidth-profile constraints). Pair with the
400
+ // `videoTrackSwitchedOff` / `videoTrackSwitchedOn` events on RemoteParticipant.
401
+ track.isSwitchedOff;
402
+ ```
403
+
404
+ ### RemoteAudioTrack
405
+
406
+ Receive decoded PCM audio frames from a remote participant.
407
+
408
+ ```js
409
+ for await (const frame of track.frames()) {
410
+ // frame: {
411
+ // format: 'PCM_S16LE',
412
+ // sampleRate, channels, frames,
413
+ // pcm: Buffer, // interleaved int16 samples
414
+ // timestamp: number, // microseconds
415
+ // frameId: number, // SDK-generated, monotonic per track
416
+ // close?(): void,
417
+ // }
418
+ }
419
+ ```
420
+
421
+ ### RemoteDataTrack
422
+
423
+ Receive string or binary messages from a remote participant.
424
+
425
+ ```js
426
+ track.on('message', (data, track) => {
427
+ /* data is string | Buffer; track is the RemoteDataTrack it arrived on */
428
+ });
429
+ ```
430
+
431
+ `maxPacketLifeTime`, `maxRetransmits`, `reliable`, and `ordered` report how the publisher
432
+ configured delivery. A publisher's limit of `65535` reads back as `null`, because a subscribed
433
+ track reports it the same way it reports an unset limit; `reliable` still distinguishes the two.
434
+
435
+ For the precise guarantees - buffer ownership, where frames are dropped, drop
436
+ policy and ordering, timestamp rules, and the publish invariants - see
437
+ [FRAME_CONTRACT.md](FRAME_CONTRACT.md).
438
+
439
+ ## Frame Formats
440
+
441
+ ### Video (`VideoFrame` / `VideoFrameInput`)
442
+
443
+ I420 planar layout. Each plane is an `I420Plane`: `{ data: Buffer, stride, width, height }`, where `stride` is bytes per row (≥ the plane's width, padded for alignment).
444
+
445
+ | Plane | Logical size | `data` size | Description |
446
+ | ----- | ------------------------ | ----------------------- | ---------------- |
447
+ | Y | `width × height` | `y.stride × height` | Luminance |
448
+ | U | `⌈width/2⌉ × ⌈height/2⌉` | `u.stride × ⌈height/2⌉` | Chrominance (Cb) |
449
+ | V | `⌈width/2⌉ × ⌈height/2⌉` | `v.stride × ⌈height/2⌉` | Chrominance (Cr) |
450
+
451
+ Publish and receive use the **same** planar shape: each of `y`/`u`/`v` is an `I420Plane` (`{ data, stride, width, height }`). A received frame can be written straight back out without reshaping.
452
+
453
+ Timestamps are plain numbers of **microseconds** (`timestamp`). Microsecond resolution stays exact in a JS number for roughly 285 years, and the underlying engine reports microseconds natively. `rotation` is `0 | 90 | 180 | 270`.
454
+
455
+ ### Audio (`AudioFrame` / `AudioFrameInput`)
456
+
457
+ Interleaved 16-bit signed little-endian PCM in a single `Buffer`.
458
+
459
+ - **Inputs** to `LocalAudioTrack.write()` are fixed at **48kHz mono** — only `pcm` and `frames` are accepted.
460
+ - **Received `AudioFrame`s** include `sampleRate`, `channels`, `frames`, `pcm`, `timestamp` (microseconds), and `frameId`.
461
+
462
+ ## Configuration
463
+
464
+ ### ConnectOptions
465
+
466
+ ```ts
467
+ {
468
+ name?: string; // Room name
469
+ videoTracks?: LocalVideoTrack[]; // Tracks to publish on connect
470
+ audioTracks?: LocalAudioTrack[];
471
+ dataTracks?: LocalDataTrack[];
472
+ enableInsights?: boolean;
473
+ enableAutomaticSubscription?: boolean;
474
+ enableDominantSpeaker?: boolean;
475
+ networkQuality?: boolean | { local?: 1; remote?: 0 | 1 };
476
+ preferredAudioCodecs?: ('opus' | 'PCMU')[];
477
+ preferredVideoCodecs?: 'VP8'[];
478
+ videoEncodingMode?: 'auto';
479
+ bandwidthProfile?: BandwidthProfileOptions;
480
+ receiveTranscriptions?: boolean;
481
+ region?: string; // e.g. 'us1', 'au1'
482
+ iceOptions?: IceOptions;
483
+ encodingParameters?: EncodingParameters;
484
+ connectionTimeout?: number; // ms; default 30000, 0 waits indefinitely
485
+ }
486
+ ```
487
+
488
+ ### BandwidthProfileOptions
489
+
490
+ ```ts
491
+ {
492
+ video?: {
493
+ mode?: 'collaboration' | 'grid' | 'presentation';
494
+ maxSubscriptionBitrate?: number; // bits per second
495
+ trackSwitchOffMode?: 'detected' | 'predicted' | 'disabled';
496
+ clientTrackSwitchOffControl?: 'auto' | 'manual';
497
+ contentPreferencesMode?: 'auto' | 'manual';
498
+ };
499
+ }
500
+ ```
501
+
502
+ ### EncodingParameters
503
+
504
+ ```ts
505
+ {
506
+ maxAudioBitrate?: number; // bits per second
507
+ maxVideoBitrate?: number; // bits per second
508
+ }
509
+ ```
510
+
511
+ ### IceOptions
512
+
513
+ ```ts
514
+ {
515
+ transportPolicy?: 'all' | 'relay'; // 'relay' forces TURN
516
+ iceServers?: IceServer[]; // { urls: string[]; username?: string; credential?: string }
517
+ }
518
+ ```
519
+
520
+ ## Platform Support
521
+
522
+ - **Node.js** >= 24.0.0
523
+ - **OS** Linux x86-64
524
+ - **Distros** Ubuntu 22.04+ and Debian 12+.
525
+ - **CPU** x86-64 only. There is no arm64 build, so on Apple Silicon Node must run under Rosetta.
526
+
527
+ The prebuilt native addon is linked against glibc and requires:
528
+
529
+ | Requirement | Minimum |
530
+ | --------------------- | ------- |
531
+ | glibc | 2.34 |
532
+ | libstdc++ (`GLIBCXX`) | 3.4.30 |
533
+ | C++ ABI (`CXXABI`) | 1.3.11 |
534
+
535
+ It also links `libX11.so.6`, which WebRTC requires unconditionally. Install your distro's X11 client library (`libx11-6` on Debian and Ubuntu) even on headless servers.
536
+
537
+ Alpine and other musl-based distros are not supported: the addon is glibc-only. Windows is not supported. macOS x64 builds and runs for local development, but is not a supported target and is not tested as one. See [DEVELOPER_GUIDE.md](DEVELOPER_GUIDE.md).
538
+
539
+ ## Examples
540
+
541
+ The example applications listed below demonstrate various ways to use the SDK for audio or video processing. They load credentials from a `.env` file at the repo root. Copy the template, fill in your credentials, and run:
542
+
543
+ ```bash
544
+ cp .env.example .env
545
+ # edit .env: set TWILIO_ACCOUNT_SID / TWILIO_API_KEY / TWILIO_API_SECRET
546
+ node examples/virtual_camera.js [room-name]
547
+ ```
548
+
549
+ `.env` is gitignored, so your real credentials are never committed.
550
+
551
+ See the [`examples/`](https://github.com/twilio/twilio-video-node/tree/main/examples) directory:
552
+
553
+ | Example | Description |
554
+ | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
555
+ | [`virtual_camera.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/virtual_camera.js) | Decodes an MP4 with ffmpeg and pushes I420 frames to a room. |
556
+ | [`video_mirror.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/video_mirror.js) | Receives remote video frames and pushes them back as-is. |
557
+ | [`audio_push.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/audio_push.js) | Generates a sine wave tone and pushes PCM audio to a room. |
558
+ | [`data_channel.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/data_channel.js) | Two participants exchange string and binary messages via data tracks. |
559
+ | [`voice_agent.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/voice_agent.js) | Bridges room audio to the OpenAI Realtime API for a spoken voice agent (requires `OPENAI_API_KEY`). |
560
+ | [`cv_object_detection.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/cv_object_detection.js) | Runs YOLOX object detection on a participant's webcam and re-publishes the video with bounding boxes. |
561
+ | [`cv_face_analysis.js`](https://github.com/twilio/twilio-video-node/blob/main/examples/cv_face_analysis.js) | Analyzes a participant's face — presence and an attention estimate (head orientation) — drawn on the video. |
562
+
563
+ The computer-vision examples (`cv_*.js`) run local ONNX models via
564
+ [`onnxruntime-node`](https://www.npmjs.com/package/onnxruntime-node) and draw with
565
+ [`@napi-rs/canvas`](https://www.npmjs.com/package/@napi-rs/canvas). These two are
566
+ large and only these examples need them, so they live in `examples/package.json`
567
+ rather than the SDK's own dependencies — install them separately:
568
+
569
+ ```bash
570
+ npm install --prefix examples
571
+ ```
572
+
573
+ No cloud service or API key is needed: each example analyzes the first
574
+ participant's video and expresses its result on a re-published video track. Run
575
+ them against any room you also join from a browser, publishing your webcam.
576
+
577
+ ### Downloading the Computer Vision models
578
+
579
+ The ONNX model files are not shipped with the repo — download the ones you need
580
+ and save them to `examples/.models/`. The examples print these same instructions
581
+ if a model is missing.
582
+
583
+ ```bash
584
+ mkdir -p examples/.models
585
+
586
+ # cv_object_detection.js — YOLOX-nano (~3.7 MB)
587
+ curl -L -o examples/.models/yolox_nano.onnx \
588
+ "https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_nano.onnx"
589
+
590
+ # cv_face_analysis.js — RTMO-t (a zip containing end2end.onnx, ~27 MB extracted)
591
+ curl -L -o examples/.models/rtmo-t.zip \
592
+ "https://download.openmmlab.com/mmpose/v1/projects/rtmo/onnx_sdk/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.zip"
593
+ unzip -j examples/.models/rtmo-t.zip '*end2end.onnx' -d examples/.models
594
+ mv examples/.models/end2end.onnx examples/.models/rtmo-t.onnx
595
+
596
+ # verify the downloads (the examples also check this on startup)
597
+ echo "c789161ed43c8269fcd4e67c67eeeb4e80c622da2eb296a20bc6007bd18a0b7d examples/.models/yolox_nano.onnx" | shasum -a 256 -c
598
+ echo "20aad6e2e42359cac1c5b4a0b2da00e29bfe91a72a782fdcf287d273a04c1b24 examples/.models/rtmo-t.onnx" | shasum -a 256 -c
599
+ ```
600
+
601
+ Both models are **Apache-2.0** licensed and downloaded from their projects'
602
+ official channels — [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) by
603
+ Megvii and [RTMO](https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo)
604
+ (OpenMMLab mmpose). Each example verifies its model's SHA-256 on startup and
605
+ refuses to run a file that doesn't match.
606
+
607
+ ## License
608
+
609
+ See [LICENSE.md](./LICENSE.md).