@webex/web-client-media-engine 1.41.1 → 1.42.0

package/README.md

@@ -0,0 +1,183 @@
+# Web Client Media Engine (WCME)
+
+Web Client Media Engine is common web code for interacting with the multistream media server.
+
+![WCME class diagram](uml.svg)
+
+## Setup
+
+1. Run `yarn` to install dependencies.
+1. Run `yarn watch` to build and watch for updates.
+1. Run `yarn test` to build, run tests, lint, and run test coverage.
+
+## Multistream Connection
+
+The `MultistreamConnection` object is the primary interface for clients to interact with the media server. It maintains a list of transceivers, as well as handles media requests to and from the media server. Each `MultistreamConnection` has a single RTCPeerConnection that is shared across all `MediaType`s. It also creates a RTCDataChannel to send and receive JMP messages.
+
+### Example
+
+Establish a multistream connection and start sending audio/video:
+
+```typescript
+const multistreamConnection = new MultistreamConnection();
+
+const offer = await multistreamConnection.createOffer();
+// after sending offer to server and receiving answer
+await multistreamConnection.setAnswer(answer);
+
+const localAudioTrack = createMicrophoneTrack();
+multistreamConnection.publishTrack(localAudioTrack);
+
+const localVideoTrack = createCameraTrack();
+multistreamConnection.publishTrack(localVideoTrack);
+```
+
+## SDP Management
+
+Clients want to be able to send audio and video (limited to main audio, main video, content audio, and content video) as well as receive any number of audio and video streams. Unified plan in WebRTC means there's a limitation that only a single stream can be signaled per mline. The media server, however, only supports four mlines (main audio, main video, content audio, and content video), so in order to receive more we need to manipulate the SDP in a way that works for both the browser and the server.
+
+To accomplish this, there are mlines that only the browser knows about -- these mlines are "filtered out" before sending an offer to the server, and corresponding mlines are "injected" into the answer from the server. This works for a few reasons:
+
+- The browser is configured to use one ICE bundle group for each media type: this means that the additional mlines don't require extra ICE connections.
+- The browser only needs packets tagged in a certain way to receive them correctly. The media server supports this tagging, so packets are routed to the correct RTCRtpReceiver.
+- JMP signaling allows signaling to the media server how to tag the packets, so the client can tell the server how to tag media such that it's received correctly.
+
+An example (truncated) client SDP:
+
+```
+  v=0
+  ...
+  // An audio mline for sending main audio
+  m=audio 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:0
+  a=sendrecv
+  a=jmp
+  a=jmp-source:0 csi=677569024
+  // A video mline for sending main video
+  m=video 63722 UDP/TLS/RTP/SAVPF 127 125 108 124 123 35 114
+  a=mid:1
+  a=sendrecv
+  a=jmp
+  a=jmp-source:1 csi=677569025
+  // An audio mline for sending content audio
+  m=audio 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:2
+  a=sendrecv
+  a=content:slides
+  a=jmp
+  a=jmp-source:2 csi=777569024
+  // A video mline for sending content video
+  m=video 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:3
+  a=sendrecv
+  a=content:slides
+  a=jmp
+  a=jmp-source:3 csi=777569025
+  // 3 recvonly audio mlines for receiving audio
+  m=audio 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:4
+  a=recvonly
+  m=audio 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:5
+  a=recvonly
+  m=audio 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:6
+  a=recvonly
+  // 3 recvonly video mlines for receiving video
+  m=video 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:7
+  a=recvonly
+  m=video 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:8
+  a=recvonly
+  m=video 56200 UDP/TLS/RTP/SAVPF 111
+  a=mid:9
+  a=recvonly
+  // mline for datachannel
+  m=application 55527 UDP/DTLS/SCTP webrtc-datachannel
+  a=mid:10
+```
+
+In the above SDP, only mlines 0, 1, 2, 3 and 10 will be sent to Homer. The rest are only seen by the client and are used to generate RTCRtpTransceivers to act as handles for media received on those mlines. When the answer from Homer is received, it will only contain mlines 0, 1, 2, 3 and 10. The client will then generate synthetic answer mlines to match the "hidden" ones (4, 5, 6, 7, 8, and 9) before setting the remote description.
+
+Mlines 4, 5, 6, 7, 8, and 9 serve only as ways for generating tracks associated with a MID on the client. To request media on those tracks, the client sends a `MediaRequest` (described below) with the appropriate MID.
+
+`MultistreamConnection` handles all the SDP manipulation required to accomplish the above for both the offer and the answer. It also performs additional preprocessing on the offer such as injecting content types and JMP attributes.
+
+## Transceivers
+
+In WebRTC, RTCRtpTransceivers represent a pairing of send and receive SRTP streams between the client and server. WCME defines two classes of transceivers: `SendOnlyTransceiver` and `RecvOnlyTransceiver`. Each `MediaType` (`VideoMain`, `AudioMain`, `VideoSlides`, or `AudioSlides`) can only have one `SendOnlyTransceiver` but may have multiple `RecvOnlyTransceiver`s. `MultistreamConnection` maintains a list of all transceivers in a connection per `MediaType`.
+
+Although `SendOnlyTransceiver`s are only used for sending, its underlying RTCRtpTransceiver direction is set to "sendrecv" in order to make it compatible with Homer. Each `SendOnlyTransceiver` maintains the state of the sending track -- whether or not it is published, whether or not it has been requested by a remote peer -- as well as handles replacing the existing track with a new one (e.g. when switching camera devices).
+
+`RecvOnlyTransceiver`s maintain the state of receiving tracks. Each `RecvOnlyTransceiver` is associated with a single `ReceiveSlot` (described below).
+
+## ReceiveSlot
+
+WebRTC clients need tracks to be able to receive remote media, but creating a track for every remote participant's media would result in a huge SDP, and would require that the SDP was updated every time a client joined or left the session. So instead of allocating an RTCRtpReceiver for every remote participant, the receivers are treated as "slots" on which a remote participant (CSI) can be requested. `ReceiveSlot`s have an ID (which is the MID of the corresponding mline), and media is requested via JMP using these IDs. For example, if a client wants to receive the 3 most active speakers' audio, then it needs to create only 3 main audio receive slots, even if there are 25 participants in the meeting.
+
+The media received on a `ReceiveSlot` can change over time, depending on the policy requested on that slot and/or future media requests on that slot. `Source Indication` messages are used by the server to notify the client which CSI is being received on a `ReceiveSlot` at a given time.
+
+## Requesting Media
+
+The `requestMedia` API is used to request media from the media server. It takes in a `MediaType` (`VideoMain`, `AudioMain`, `VideoSlides`, or `AudioSlides`) and an array of `MediaRequest` objects. A `MediaRequest` object consists of a policy (`ActiveSpeaker` or `ReceiverSelected`), some policy-specific information, and an array of `ReceiveSlot`s on which the requested media will be received.
+
+```typescript
+requestMedia(mediaType: MediaType, mediaRequests: MediaRequest[]): void
+```
+
+The `MediaRequest` object is an abstraction on the JMP SCR object, and allows crafting different combinations of policies and `ReceiveSlot`s to achieve different behaviors. A `MediaRequest` consists of:
+
+- Policy
+- PolicySpecificInfo
+- ReceiveSlot[]
+
+Details for these fields are the same as they are for the JMP SCR. Information can be found [here](https://confluence-eng-gpk2.cisco.com/conf/pages/viewpage.action?spaceKey=WMT&title=JMP+-+Json+Multistream+Protocol).
+
+### Examples
+
+Requesting the audio of the 3 active speakers:
+
+```typescript
+// Create the receive slots
+const audioSlot1 = await createReceiveSlot(MediaType.AudioMain);
+const audioSlot2 = await createReceiveSlot(MediaType.AudioMain);
+const audioSlot3 = await createReceiveSlot(MediaType.AudioMain);
+requestMedia(MediaType.AudioMain, [
+  new MediaRequest(Policy.ActiveSpeaker, new ActiveSpeakerInfo(100, false, false, true), [
+    audioSlot1,
+    audioSlot2,
+    audioSlot3,
+  ]),
+]);
+```
+
+Requesting a the video of a specific CSI:
+
+```typescript
+// Create the receive slots
+const videoSlot1 = await createReceiveSlot(MediaType.VideoMain);
+requestMedia(MediaType.VideoMain, [
+  new MediaRequest(Policy.ReceiverSelected, new ReceiverSelectedInfo(csiToSelect), [videoSlot1]),
+]);
+```
+
+## Utilities
+
+WCME also defines several utility files with exported functions that may be useful in both WCME and in other code:
+
+- `sdp-utils`: Contains functions that munge and manipulate SDP offer/answer descriptions.
+- `ua-utils`: Contains functions to help identify the current user agent (i.e. browser name and version).
+
+## Logging
+
+Logging is done through the `js-logger` library and the `Logger` class is exported to be used with other repositories. This can be done by importing and setting a handler for `Logger`.
+
+```ts
+import { Logger } from '@webex/web-client-media-engine';
+
+Logger.setHandler((msgs, context) => {
+  // do something with logs
+  console.log(context.name, msgs);
+});
+```