@webex/web-client-media-engine 3.5.1 → 3.6.0

package/README.md

@@ -3,6 +3,7 @@
 Web Client Media Engine is common web code for interacting with the multistream media server.
 
 ![WCME class diagram](uml.svg)
+_UML diagram generated with [tplant](https://github.com/bafolts/tplant)._
 
 ## Setup
 
@@ -25,11 +26,13 @@ const offer = await multistreamConnection.createOffer();
 // after sending offer to server and receiving answer
 await multistreamConnection.setAnswer(answer);
 
-const localAudioTrack = createMicrophoneTrack();
-multistreamConnection.publishTrack(localAudioTrack);
+const audioSendSlot = multistreamConnection.createSendSlot(MediaType.AudioMain);
+const localAudioStream = createMicrophoneStream(LocalMicrophoneStream);
+audioSendSlot.publishStream(localAudioStream);
 
-const localVideoTrack = createCameraTrack();
-multistreamConnection.publishTrack(localVideoTrack);
+const videoSendSlot = multistreamConnection.createSendSlot(MediaType.VideoMain);
+const localVideoStream = createCameraStream(LocalCameraStream);
+videoSendSlot.publishStream(localVideoStream);
 ```
 
 ## SDP Management
@@ -102,21 +105,29 @@ In the above SDP, only mlines 0, 1, 2, 3 and 10 will be sent to Homer. The rest
 
 Mlines 4, 5, 6, 7, 8, and 9 serve only as ways for generating tracks associated with a MID on the client. To request media on those tracks, the client sends a `MediaRequest` (described below) with the appropriate MID.
 
-`MultistreamConnection` handles all the SDP manipulation required to accomplish the above for both the offer and the answer. It also performs additional preprocessing on the offer such as injecting content types and JMP attributes.
+`MultistreamConnection` handles all the SDP manipulation required to accomplish the above for both the offer and the answer. It also performs additional preprocessing on the offer such as injecting content types and JMP attributes. All of this is handled by the appropriate SDP "mungers" (`IngressSdpMunger` and `EgressSdpMunger`), which are called before setting the local offer, sending the offer to the remote server, and setting the remote answer.
 
-## Transceivers
+## Transceivers and Slots
 
 In WebRTC, RTCRtpTransceivers represent a pairing of send and receive SRTP streams between the client and server. WCME defines two classes of transceivers: `SendOnlyTransceiver` and `RecvOnlyTransceiver`. Each `MediaType` (`VideoMain`, `AudioMain`, `VideoSlides`, or `AudioSlides`) can only have one `SendOnlyTransceiver` but may have multiple `RecvOnlyTransceiver`s. `MultistreamConnection` maintains a list of all transceivers in a connection per `MediaType`.
 
-Although `SendOnlyTransceiver`s are only used for sending, its underlying RTCRtpTransceiver direction is set to "sendrecv" in order to make it compatible with Homer. Each `SendOnlyTransceiver` maintains the state of the sending track -- whether or not it is published, whether or not it has been requested by a remote peer -- as well as handles replacing the existing track with a new one (e.g. when switching camera devices).
+Although `SendOnlyTransceiver`s are only used for sending, its underlying RTCRtpTransceiver direction is set to "sendrecv" in order to make it compatible with Homer. Each `SendOnlyTransceiver` maintains the state of the sending stream -- whether or not it is published, whether or not it has been requested by a remote peer -- as well as handles replacing the existing stream with a new one (e.g. when switching camera devices).
 
-`RecvOnlyTransceiver`s maintain the state of receiving tracks. Each `RecvOnlyTransceiver` is associated with a single `ReceiveSlot` (described below).
+Likewise, `RecvOnlyTransceiver`s maintain the state of receiving streams.
 
-## ReceiveSlot
+Clients should never have to interact with transceivers directly. Instead, all interactions with transceivers are handled via "slots". Each `SendOnlyTransceiver` is associated with a single `SendSlot`, while each `RecvOnlyTransceiver` is associated with a single `ReceiveSlot`.
 
-WebRTC clients need tracks to be able to receive remote media, but creating a track for every remote participant's media would result in a huge SDP, and would require that the SDP was updated every time a client joined or left the session. So instead of allocating an RTCRtpReceiver for every remote participant, the receivers are treated as "slots" on which a remote participant (CSI) can be requested. `ReceiveSlot`s have an ID (which is the MID of the corresponding mline), and media is requested via JMP using these IDs. For example, if a client wants to receive the 3 most active speakers' audio, then it needs to create only 3 main audio receive slots, even if there are 25 participants in the meeting.
+### Receive Slots
 
-The media received on a `ReceiveSlot` can change over time, depending on the policy requested on that slot and/or future media requests on that slot. `Source Indication` messages are used by the server to notify the client which CSI is being received on a `ReceiveSlot` at a given time.
+WebRTC clients need to be able to receive remote media, but creating a stream for every remote participant's media would result in a huge SDP, and would require that the SDP was updated every time a client joined or left the session. So instead of allocating an RTCRtpReceiver for every remote participant, the receivers are treated as "slots" on which a remote participant (CSI) can be requested. `ReceiveSlot`s have an ID (which is the MID of the corresponding mline), and media is requested via JMP using these IDs. For example, if a client wants to receive the 3 most active speakers' audio, then it needs to create only 3 main audio receive slots, even if there are 25 participants in the meeting.
+
+The media received on a `ReceiveSlot` can change over time, depending on the policy requested on that slot and/or future media requests on that slot. `sourceAdvertisement` messages are used by the server to notify the client which CSI is being received on a `ReceiveSlot` at a given time.
+
+### Send Slots
+
+Similarly, send slots are used to handle sending local media.
+
+By default, all `SendOnlyTransceiver`s are inactive (that is, the direction in the SDP offer is set to "inactive") until a `SendSlot` is created, during which the transceiver can be activated (the direction set to "sendrecv"). Only one `SendSlot` per media type should be created at a time, and `SendSlot`s can be activated or deactivated on the fly.
 
 ## Requesting Media
 
@@ -126,13 +137,13 @@ The `requestMedia` API is used to request media from the media server. It takes
 requestMedia(mediaType: MediaType, mediaRequests: MediaRequest[]): void
 ```
 
-The `MediaRequest` object is an abstraction on the JMP SCR object, and allows crafting different combinations of policies and `ReceiveSlot`s to achieve different behaviors. A `MediaRequest` consists of:
+The `MediaRequest` object is an abstraction on the JMP media request object, and allows crafting different combinations of policies and `ReceiveSlot`s to achieve different behaviors. A `MediaRequest` consists of:
 
 - Policy
 - PolicySpecificInfo
 - ReceiveSlot[]
 
-Details for these fields are the same as they are for the JMP SCR. Information can be found [here](https://confluence-eng-gpk2.cisco.com/conf/pages/viewpage.action?spaceKey=WMT&title=JMP+-+Json+Multistream+Protocol).
+Details for these fields are the same as they are for the JMP media request. Information can be found [here](https://confluence-eng-gpk2.cisco.com/conf/pages/viewpage.action?spaceKey=WMT&title=JMP+-+Json+Multistream+Protocol).
 
 ### Examples
 
@@ -162,13 +173,15 @@ requestMedia(MediaType.VideoMain, [
 ]);
 ```
 
-## Utilities
+## Utilities and Other Exports
 
 WCME also defines several utility files with exported functions that may be useful in both WCME and in other code:
 
 - `sdp-utils`: Contains functions that munge and manipulate SDP offer/answer descriptions.
 - `ua-utils`: Contains functions to help identify the current user agent (i.e. browser name and version).
 
+In addition to WCME exports, all necessary imports from `webrtc-core` and `json-multistream` are also re-exported for your convenience, so they can be used in other code without the need to import from those repositories separately.
+
 ## Logging
 
 Logging is done through the `js-logger` library and the `Logger` class is exported to be used with other repositories. This can be done by importing and setting a handler for `Logger`.
@@ -181,3 +194,5 @@ Logger.setHandler((msgs, context) => {
   console.log(context.name, msgs);
 });
 ```
+
+Loggers from `webrtc-core` and `json-multistream` are also re-exported if you want to handle logs from those repositories separately.

package/dist/cjs/index.js

@@ -6942,6 +6942,17 @@ var map2obj = function (report) {
     });
     return o;
 };
+var dumpStream = function (stream) { return ({
+    id: stream.id,
+    tracks: stream.getTracks().map(function (track) { return ({
+        id: track.id,
+        kind: track.kind,
+        label: track.label,
+        enabled: track.enabled,
+        muted: track.muted,
+        readyState: track.readyState
+    }); })
+}); };
 var persistedKeys = ['type', 'id', 'timestamp'];
 /**
  * Check to see if the report consists of more than just the persisted metadata.
@@ -7036,6 +7047,7 @@ var rtcStats = function (pc, logger, intervalTime, statsPreProcessor) {
     var trace = function (name, payload, timestamp) {
         logger({ timestamp: timestamp ? Math.round(timestamp) : Date.now(), name: name, payload: payload });
     };
+    var origPeerConnection = window.RTCPeerConnection;
     pc.addEventListener('icecandidate', function (e) {
         if (e.candidate) {
             trace('onicecandidate', makeEvent(JSON.stringify(e.candidate)));
@@ -7068,6 +7080,139 @@ var rtcStats = function (pc, logger, intervalTime, statsPreProcessor) {
     pc.addEventListener('datachannel', function (event) {
         trace('ondatachannel', makeEvent("".concat(event.channel.id, ": ").concat(event.channel.label)));
     });
+    ['createDataChannel', 'close'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                trace(method, makeEvent(method));
+                return nativeMethod.apply(this, arguments);
+            };
+        }
+    });
+    ['addStream', 'removeStream'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                var stream = arguments[0];
+                var streamInfo = stream
+                    .getTracks()
+                    .map(function (t) { return "".concat(t.kind, ":").concat(t.id); })
+                    .join(',');
+                trace(method, makeEvent("".concat(stream.id, " ").concat(streamInfo)));
+                return nativeMethod.apply(this, arguments);
+            };
+        }
+    });
+    ['addTrack'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                var track = arguments[0];
+                var streams = [].slice.call(arguments, 1);
+                trace(method, makeEvent("".concat(track.kind, ":").concat(track.id, " ").concat(streams.map(function (s) { return "stream:".concat(s.id); }).join(';') || '-')));
+                return nativeMethod.apply(this, arguments);
+            };
+        }
+    });
+    ['removeTrack'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                var track = arguments[0].track;
+                trace(method, makeEvent(track ? "".concat(track.kind, ":").concat(track.id) : 'null'));
+                return nativeMethod.apply(this, arguments);
+            };
+        }
+    });
+    ['createOffer', 'createAnswer'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                var opts;
+                var args = arguments;
+                if (arguments.length === 1 && typeof arguments[0] === 'object') {
+                    // eslint-disable-next-line prefer-destructuring
+                    opts = arguments[0];
+                }
+                else if (arguments.length === 3 && typeof arguments[2] === 'object') {
+                    // eslint-disable-next-line prefer-destructuring
+                    opts = arguments[2];
+                }
+                trace(method, makeEvent(opts || ''));
+                return nativeMethod.apply(this, opts ? [opts] : undefined).then(function (description) {
+                    trace("".concat(method, "OnSuccess"), makeEvent(description.sdp));
+                    if (args.length > 0 && typeof args[0] === 'function') {
+                        args[0].apply(null, [description]);
+                        return undefined;
+                    }
+                    return description;
+                }, function (err) {
+                    trace("".concat(method, "OnFailure"), makeEvent(err.toString()));
+                    if (args.length > 1 && typeof args[1] === 'function') {
+                        args[1].apply(null, [err]);
+                        return;
+                    }
+                    throw err;
+                });
+            };
+        }
+    });
+    ['setLocalDescription', 'setRemoteDescription', 'addIceCandidate'].forEach(function (method) {
+        var nativeMethod = origPeerConnection.prototype[method];
+        if (nativeMethod) {
+            origPeerConnection.prototype[method] = function () {
+                var args = arguments;
+                trace(method, makeEvent(method === 'addIceCandidate' ? arguments[0] : arguments[0].sdp));
+                return nativeMethod.apply(this, [arguments[0]]).then(function () {
+                    trace("".concat(method, "OnSuccess"), makeEvent('success'));
+                    if (args.length >= 2 && typeof args[1] === 'function') {
+                        args[1].apply(null, []);
+                        return undefined;
+                    }
+                    return undefined;
+                }, function (err) {
+                    trace("".concat(method, "OnFailure"), makeEvent(err.toString()));
+                    if (args.length >= 3 && typeof args[2] === 'function') {
+                        args[2].apply(null, [err]);
+                        return undefined;
+                    }
+                    throw err;
+                });
+            };
+        }
+    });
+    if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
+        var origGetUserMedia_1 = navigator.mediaDevices.getUserMedia.bind(navigator.mediaDevices);
+        var gum = function () {
+            trace('navigator.mediaDevices.getUserMedia', makeEvent(JSON.stringify(arguments[0])));
+            return origGetUserMedia_1
+                .apply(navigator.mediaDevices, arguments)
+                .then(function (stream) {
+                trace('navigator.mediaDevices.getUserMediaOnSuccess', makeEvent(JSON.stringify(dumpStream(stream))));
+                return stream;
+            }, function (err) {
+                trace('navigator.mediaDevices.getUserMediaOnFailure', makeEvent(err.name));
+                return Promise.reject(err);
+            });
+        };
+        navigator.mediaDevices.getUserMedia = gum.bind(navigator.mediaDevices);
+    }
+    if (navigator.mediaDevices && navigator.mediaDevices.getDisplayMedia) {
+        var origGetDisplayMedia_1 = navigator.mediaDevices.getDisplayMedia.bind(navigator.mediaDevices);
+        var gdm = function () {
+            trace('navigator.mediaDevices.getDisplayMedia', makeEvent(JSON.stringify(arguments[0])));
+            return origGetDisplayMedia_1
+                .apply(navigator.mediaDevices, arguments)
+                .then(function (stream) {
+                trace('navigator.mediaDevices.getDisplayMediaOnSuccess', makeEvent(JSON.stringify(dumpStream(stream))));
+                return stream;
+            }, function (err) {
+                trace('navigator.mediaDevices.getDisplayMediaOnFailure', makeEvent(err.name));
+                return Promise.reject(err);
+            });
+        };
+        navigator.mediaDevices.getDisplayMedia = gdm.bind(navigator.mediaDevices);
+    }
     var interval = window.setInterval(function () {
         if (pc.signalingState === 'closed') {
             window.clearInterval(interval);
@@ -10277,6 +10422,7 @@ class SendOnlyTransceiver extends Transceiver {
         this.streamMuteStateChange = new TypedEvent();
         this.streamPublishStateChange = new TypedEvent();
         this.negotiationNeeded = new TypedEvent();
+        this.namedMediaGroupsChange = new TypedEvent();
         this.requested = false;
         this.requestedIdEncodingParamsMap = new Map();
         this.csi = csi;
@@ -10351,6 +10497,13 @@ class SendOnlyTransceiver extends Transceiver {
             }
         });
     }
+    setNamedMediaGroups(namedMediaGroups) {
+        if (this.mediaType !== exports.MediaType.AudioMain) {
+            throw new Error(`This interface does not allow media types other than AudioMain: ${this.mediaType}`);
+        }
+        this.namedMediaGroups = namedMediaGroups;
+        this.namedMediaGroupsChange.emit();
+    }
     publishStream(stream) {
         return this.replacePublishedStream(stream);
     }
@@ -10454,6 +10607,9 @@ class SendSlot {
             return this.sendTransceiver.unpublishStream();
         });
     }
+    setNamedMediaGroups(namedMediaGroups) {
+        this.sendTransceiver.setNamedMediaGroups(namedMediaGroups);
+    }
     get active() {
         return this.sendTransceiver.active;
     }
@@ -13893,6 +14049,9 @@ class MultistreamConnection extends EventEmitter$2 {
                 this.queueLocalOfferAnswer();
             }
         });
+        transceiver.namedMediaGroupsChange.on(() => {
+            this.sendSourceAdvertisement(mediaType);
+        });
         this.sendTransceivers.set(mediaType, transceiver);
         this.createJmpSession(mediaType);
     }
@@ -14082,7 +14241,11 @@ class MultistreamConnection extends EventEmitter$2 {
             };
         }
         else {
-            task = () => { var _a; return (_a = this.jmpSessions.get(mediaType)) === null || _a === void 0 ? void 0 : _a.sendSourceAdvertisement(1, numLiveSources); };
+            task = () => {
+                var _a;
+                return (_a = this.jmpSessions
+                    .get(mediaType)) === null || _a === void 0 ? void 0 : _a.sendSourceAdvertisement(1, numLiveSources, mediaType === exports.MediaType.AudioMain ? transceiver.namedMediaGroups : []);
+            };
         }
         if (((_b = this.dataChannel) === null || _b === void 0 ? void 0 : _b.readyState) === 'open') {
             task();