twilio-video 2.18.1 → 2.18.2

package/CHANGELOG.md

@@ -2,6 +2,14 @@ The Twilio Programmable Video SDKs use [Semantic Versioning](http://www.semver.o
 
 **Version 1.x reached End of Life on September 8th, 2021.** See the changelog entry [here](https://www.twilio.com/changelog/end-of-life-complete-for-unsupported-versions-of-the-programmable-video-sdk). Support for the 1.x version ended on December 4th, 2020.
 
+2.18.2 (December 15, 2021)
+==========================
+
+Bug Fixes
+---------
+
+- Fixed a bug where setting `clientTrackSwitchOffControl` to `auto` caused the RemoteVideoTracks to get switched off while playing in picture-in-picture mode. Note that this fix does not apply to Firefox as it does not yet implement [picture-in-picture](https://developer.mozilla.org/en-US/docs/Web/API/Picture-in-Picture_API) APIs. (VIDEO-6677)
+
 2.18.1 (October 29, 2021)
 =========================
 

package/COMMON_ISSUES.md

@@ -36,10 +36,10 @@ before opening a new issue. We recommend regularly upgrading to the latest versi
 
 ### Chrome mobile
 <details>
-<summary>Chrome on iOS 15.1 crashes when publishing or muting a VideoTrack that is using an H264 codec</summary>
+<summary>Android 12: Video distortion on Chrome when hardware acceleration is enabled</summary>
 <p>
 
-   This issue happens due to a regression on iOS 15.1 where using an H264 codec causes the browser to crash when joining a room or when muting the VideoTrack. Please use VP8 as a workaround for now. See more details [here](https://github.com/twilio/twilio-video.js/issues/1611).
+   This is a VP8 encoder issue on Android 12. Please see this [github ticket](https://github.com/twilio/twilio-video.js/issues/1627) and this [Chrome bug](https://bugs.chromium.org/p/chromium/issues/detail?id=1237677) for more details.
 </p>
 </details>
 <details>
@@ -61,13 +61,12 @@ before opening a new issue. We recommend regularly upgrading to the latest versi
 
 ### Safari desktop
 <details>
-<summary>Safari 15.1 crashes when muting a VideoTrack that is using an H264 codec</summary>
+<summary>Browser crashes when muting a VideoTrack that is using an H264 codec on Safari 15.1</summary>
 <p>
 
-   This issue happens due to a regression on Safari 15.1 where muting a VideoTrack that is using an H264 codec causes the browser to crash. Please use VP8 as a workaround for now. See more details [here](https://github.com/twilio/twilio-video.js/issues/1611).
+   Due to a regression on Safari 15.1, the browser crashes when a VideoTrack is muted that is using an H264 codec. Please use VP8 as a workaround for now. See more details [here](https://github.com/twilio/twilio-video.js/issues/1611).
 </p>
 </details>
-
 <details>
 <summary>Failures to publish tracks on Safari 15</summary>
 <p>
@@ -114,17 +113,101 @@ before opening a new issue. We recommend regularly upgrading to the latest versi
 
 ### Safari mobile
 <details>
-<summary>iOS 15.1: Browser crashes when publishing or muting a VideoTrack that is using an H264 codec</summary>
+<summary>iOS 15: VideoTracks goes black and the page freezes on certain interruptions</summary>
+<p>
+
+   Certain interruptions such as incoming calls, backgrounding the browser or switching between apps causes VideoTracks on Chrome and Safari on iOS 15.1 to go black. Sometimes, the whole page also freezes and become unresponsive causing audio and video to cut off. These issues are regressions on iOS 15.1. See the following bugs for more details.
+
+   * [Page freezing](https://bugs.webkit.org/show_bug.cgi?id=230922#c12)
+   * [VideoTrack going black](https://bugs.webkit.org/show_bug.cgi?id=232599)
+
+   A workaround can be implemented to prevent the VideoTrack from going black. This workaround however doesn't prevent the issue where sometimes the page freezes. It is recommended to apply this workaround on Chrome and Safari on iOS 15.1.
+
+  ```js
+  // Keeps track of video elements and their event listeners
+  const videoElements = {};
+
+  // Listen to onPlay and onPause events and intelligently re-attach the video element
+  function shimVideoElement(track, el) {
+    let wasInterrupted = false;
+
+    const onPause = () => {
+      wasInterrupted = true;
+    };
+
+    const onPlay = () => {
+      if (wasInterrupted) {
+        track.detach(el);
+        track.attach(el);
+        wasInterrupted = false;
+      }
+    };
+
+    el.addEventListener('pause', onPause);
+    el.addEventListener('play', onPlay);
+
+    // Track this element so we can remove the listeners
+    videoElements[el] = { onPause, onPlay };
+  }
+  ```
+
+  Apply the workaround after attaching the video element.
+
+  ```js
+  videoTrack.attach(videoElement);
+  shimVideoElement(videoTrack, videoElement);
+  ```
+
+  Remove the listeners before detaching the video element.
+
+  ```js
+  const { onPause, onPlay } = videoElements[videoElement];
+  videoElement.removeEventListener('pause', onPause);
+  videoElement.removeEventListener('play', onPlay);
+  ```
+</p>
+</details>
+<details>
+<summary>iOS 15: Browser crashes when publishing or muting a VideoTrack that is using an H264 codec</summary>
 <p>
 
-   This issue happens due to a regression on iOS 15.1 where using an H264 codec causes the browser to crash when joining a room or when muting the VideoTrack. Please use VP8 as a workaround for now. See more details [here](https://github.com/twilio/twilio-video.js/issues/1611).
+   Chrome and Safari on iOS 15.1 crashes when a VideoTrack is muted or published using an H264 codec. This issue happens due to a regression on iOS 15.1. Please use VP8 as a workaround for now. See more details [here](https://github.com/twilio/twilio-video.js/issues/1611).
 </p>
 </details>
 <details>
 <summary>iOS 15: Low audio volume in Safari</summary>
 <p>
 
-   Safari on iOS version 15, sometimes routes audio to the earpiece and not the speakers by default. Which customers some time perceive as low audio volume. Find more details [here](https://github.com/twilio/twilio-video.js/issues/1586) and in this [WebKit bug](https://bugs.webkit.org/show_bug.cgi?id=230902).
+   Safari on iOS version 15, sometimes routes audio to the earpiece and not the speakers by default. Which customers some time perceive as low audio volume. Find more details [here](https://github.com/twilio/twilio-video.js/issues/1586) and in this [WebKit bug](https://bugs.webkit.org/show_bug.cgi?id=230902). As a workaround, you can pipe all remote audio tracks into a single audio context for iOS 15. Using a gain node, you can increase the gain value to increase the audio volume levels. See example below.
+
+   ```js
+   // Make sure to reuse the audioContext object as browsers
+   // have limits to the number of AudioContext instances you can create.
+   const audioContext = new (window.AudioContext || window.webkitAudioContext)();
+
+   function attachAudioTrack(remoteAudioTrack) {
+     const audioNode = audioContext.createMediaStreamSource(new MediaStream([remoteAudioTrack.mediaStreamTrack]));
+     const gainNode = audioContext.createGain();
+     
+     // Adjust this value depending on your customers' preference
+     gainNode.gain.value = 20;
+     
+     audioNode.connect(gainNode);
+     gainNode.connect(audioContext.destination);
+   }
+
+   // Attach the RemoteAudioTrack once received.
+   attachAudioTrack(remoteAudioTrack);
+   ```
+
+   This workaround has the following potential side effects.
+
+   * There is a possibility of the introduction of echo. Please adjust the gain value and check for echo while testing the workaround.
+   * The output volume might end up really high if the user switches headsets.
+   * The default volume might end up really high once Apple rolls out the fix for this issue.
+
+   Keeping the side effects in mind, you might need to adjust your UI to improve the experience. For example, you can turn off this workaround by default and have a "call to action" in your UI that allows the user to turn the volume up if they cannot hear any audio. This button will then apply the workaround. Another option is to listen for `devicechange` events to determine if the user switches headsets. When this happens, you will have the ability to reset the gain value.
+
 </p>
 </details>
 <details>

package/README.md

@@ -74,7 +74,7 @@ Releases of twilio-video.js are hosted on a CDN, and you can include these
 directly in your web app using a <script> tag.
 
 ```html
-<script src="//sdk.twilio.com/js/video/releases/2.18.1/twilio-video.min.js"></script>
+<script src="//sdk.twilio.com/js/video/releases/2.18.2/twilio-video.min.js"></script>
 
 ```
 

package/dist/twilio-video.js

@@ -1,4 +1,4 @@
-/*! twilio-video.js 2.18.1
+/*! twilio-video.js 2.18.2
 
 The following license applies to all parts of this software except as
 documented below.
@@ -3469,6 +3469,8 @@ function restartWhenInadvertentlyStopped(localMediaTrack) {
             // after the MediaStreamTrack is re-acquired.
             var promise = (trackChangeInProgress && trackChangeInProgress.promise) || Promise.resolve();
             return promise.finally(function () { return localMediaRestartDeferreds.resolveDeferred(kind); });
+        }).catch(function (ex) {
+            log.error("error in maybeRestart: " + ex.message);
         });
     }
     function onMute() {
@@ -5414,6 +5416,12 @@ var RemoteVideoTrack = /** @class */ (function (_super) {
             _invisibleElements: {
                 value: new WeakSet(),
             },
+            _elToPipCallbacks: {
+                value: new WeakMap(),
+            },
+            _elToPipWindows: {
+                value: new WeakMap(),
+            },
             _turnOffTimer: {
                 value: new Timeout(function () {
                     _this._setRenderHint({ enabled: false });
@@ -5507,6 +5515,45 @@ var RemoteVideoTrack = /** @class */ (function (_super) {
         }
         return this;
     };
+    RemoteVideoTrack.prototype._unObservePip = function (el) {
+        var pipCallbacks = this._elToPipCallbacks.get(el);
+        if (pipCallbacks) {
+            el.removeEventListener('enterpictureinpicture', pipCallbacks.onEnterPip);
+            el.removeEventListener('leavepictureinpicture', pipCallbacks.onLeavePip);
+            this._elToPipCallbacks.delete(el);
+        }
+    };
+    RemoteVideoTrack.prototype._observePip = function (el) {
+        var _this = this;
+        var pipCallbacks = this._elToPipCallbacks.get(el);
+        if (!pipCallbacks) {
+            var onEnterPip = function (event) { return _this._onEnterPip(event, el); };
+            var onLeavePip = function (event) { return _this._onLeavePip(event, el); };
+            var onResizePip = function (event) { return _this._onResizePip(event, el); };
+            el.addEventListener('enterpictureinpicture', onEnterPip);
+            el.addEventListener('leavepictureinpicture', onLeavePip);
+            this._elToPipCallbacks.set(el, { onEnterPip: onEnterPip, onLeavePip: onLeavePip, onResizePip: onResizePip });
+        }
+    };
+    RemoteVideoTrack.prototype._onEnterPip = function (event, videoEl) {
+        this._log.debug('onEnterPip');
+        var pipWindow = event.pictureInPictureWindow;
+        this._elToPipWindows.set(videoEl, pipWindow);
+        var onResizePip = this._elToPipCallbacks.get(videoEl).onResizePip;
+        pipWindow.addEventListener('resize', onResizePip);
+        maybeUpdateEnabledHint(this);
+    };
+    RemoteVideoTrack.prototype._onLeavePip = function (event, videoEl) {
+        this._log.debug('onLeavePip');
+        this._elToPipWindows.delete(videoEl);
+        var onResizePip = this._elToPipCallbacks.get(videoEl).onResizePip;
+        var pipWindow = event.pictureInPictureWindow;
+        pipWindow.removeEventListener('resize', onResizePip);
+        maybeUpdateEnabledHint(this);
+    };
+    RemoteVideoTrack.prototype._onResizePip = function () {
+        maybeUpdateDimensionHint(this);
+    };
     RemoteVideoTrack.prototype.attach = function (el) {
         var result = _super.prototype.attach.call(this, el);
         if (this._clientTrackSwitchOffControl === 'auto') {
@@ -5519,6 +5566,7 @@ var RemoteVideoTrack = /** @class */ (function (_super) {
         if (this._enableDocumentVisibilityTurnOff) {
             this._documentVisibilityTurnOffCleanup = this._documentVisibilityTurnOffCleanup || setupDocumentVisibilityTurnOff(this);
         }
+        this._observePip(result);
         return result;
     };
     RemoteVideoTrack.prototype.detach = function (el) {
@@ -5529,6 +5577,7 @@ var RemoteVideoTrack = /** @class */ (function (_super) {
             _this._intersectionObserver.unobserve(element);
             _this._resizeObserver.unobserve(element);
             _this._invisibleElements.delete(element);
+            _this._unObservePip(element);
         });
         if (this._attachments.size === 0) {
             if (this._documentVisibilityTurnOffCleanup) {
@@ -5627,32 +5676,39 @@ function setupDocumentVisibilityTurnOff(removeVideoTrack) {
         documentVisibilityMonitor.offVisibilityChange(1, onVisibilityChanged);
     };
 }
-function maybeUpdateEnabledHint(removeVideoTrack) {
-    if (removeVideoTrack._clientTrackSwitchOffControl !== 'auto') {
+function maybeUpdateEnabledHint(remoteVideoTrack) {
+    if (remoteVideoTrack._clientTrackSwitchOffControl !== 'auto') {
         return;
     }
-    var visibleElements = removeVideoTrack._getAllAttachedElements().filter(function (el) { return !removeVideoTrack._invisibleElements.has(el); });
-    var enabled = document.visibilityState === 'visible' && visibleElements.length > 0;
+    var visibleElements = remoteVideoTrack._getAllAttachedElements().filter(function (el) { return !remoteVideoTrack._invisibleElements.has(el); });
+    var pipWindows = remoteVideoTrack._getAllAttachedElements().filter(function (el) { return remoteVideoTrack._elToPipWindows.has(el); });
+    // even when document is invisible we may have track playing in pip window.
+    var enabled = pipWindows.length > 0 || (document.visibilityState === 'visible' && visibleElements.length > 0);
     if (enabled === true) {
-        removeVideoTrack._turnOffTimer.clear();
-        removeVideoTrack._setRenderHint({ enabled: true });
+        remoteVideoTrack._turnOffTimer.clear();
+        remoteVideoTrack._setRenderHint({ enabled: true });
     }
-    else if (!removeVideoTrack._turnOffTimer.isSet) {
+    else if (!remoteVideoTrack._turnOffTimer.isSet) {
         // set the track to be turned off after some delay.
-        removeVideoTrack._turnOffTimer.start();
+        remoteVideoTrack._turnOffTimer.start();
     }
 }
-function maybeUpdateDimensionHint(removeVideoTrack) {
-    if (removeVideoTrack._contentPreferencesMode !== 'auto') {
+function maybeUpdateDimensionHint(remoteVideoTrack) {
+    if (remoteVideoTrack._contentPreferencesMode !== 'auto') {
         return;
     }
-    var visibleElements = removeVideoTrack._getAllAttachedElements().filter(function (el) { return !removeVideoTrack._invisibleElements.has(el); });
-    if (visibleElements.length > 0) {
-        var _a = __read(visibleElements.sort(function (el1, el2) {
+    var visibleElements = remoteVideoTrack._getAllAttachedElements().filter(function (el) { return !remoteVideoTrack._invisibleElements.has(el); });
+    var pipElements = remoteVideoTrack._getAllAttachedElements().map(function (el) {
+        var pipWindow = remoteVideoTrack._elToPipWindows.get(el);
+        return pipWindow ? { clientHeight: pipWindow.height, clientWidth: pipWindow.width } : { clientHeight: 0, clientWidth: 0 };
+    });
+    var totalElements = visibleElements.concat(pipElements);
+    if (totalElements.length > 0) {
+        var _a = __read(totalElements.sort(function (el1, el2) {
             return el2.clientHeight + el2.clientWidth - el1.clientHeight - el1.clientWidth - 1;
         }), 1), _b = _a[0], clientHeight = _b.clientHeight, clientWidth = _b.clientWidth;
         var renderDimensions = { height: clientHeight, width: clientWidth };
-        removeVideoTrack._setRenderHint({ renderDimensions: renderDimensions });
+        remoteVideoTrack._setRenderHint({ renderDimensions: renderDimensions });
     }
 }
 /**
@@ -11533,19 +11589,17 @@ var MediaSignaling = /** @class */ (function (_super) {
         var receiverPromise = this._getReceiver(id).then(function (receiver) {
             if (receiver.kind !== 'data') {
                 _this._log.error('Expected a DataTrackReceiver');
-                throw new Error('Expected a DataTrackReceiver');
             }
             if (_this._receiverPromise !== receiverPromise) {
                 return;
             }
             try {
                 _this._transport = receiver.toDataTransport();
+                _this.emit('ready', _this._transport);
             }
             catch (ex) {
-                _this._log.error('Failed to toDataTransport');
-                throw new Error('Failed to toDataTransport');
+                _this._log.error("Failed to toDataTransport: " + ex.message);
             }
-            _this.emit('ready', _this._transport);
             receiver.once('close', function () { return _this._teardown(); });
         });
         this._receiverPromise = receiverPromise;
@@ -12843,7 +12897,9 @@ var PeerConnectionV2 = /** @class */ (function (_super) {
             log.debug("Starting ICE reconnect timeout: " + delay);
             this._iceReconnectTimeout.start();
         }
-        this.offer();
+        this.offer().catch(function (ex) {
+            log.error("offer failed in _initiateIceRestart with: " + ex.message);
+        });
     };
     /**
      * Schedule an ICE Restart.
@@ -31275,7 +31331,7 @@ module.exports={
   "name": "twilio-video",
   "title": "Twilio Video",
   "description": "Twilio Video JavaScript Library",
-  "version": "2.18.1",
+  "version": "2.18.2",
   "homepage": "https://twilio.com",
   "author": "Mark Andrus Roberts <mroberts@twilio.com>",
   "contributors": [
@@ -31370,7 +31426,8 @@ module.exports={
     "lint:js": "eslint ./lib ./test/*.js ./docker/**/*.js ./test/framework/*.js ./test/lib/*.js ./test/integration/** ./test/unit/** ",
     "lint:ts": "eslint ./tsdef/*.ts ./lib/**/*.ts",
     "lint": "npm-run-all lint:js lint:ts",
-    "test:unit": "npm-run-all build:es5 test:unit:js test:unit:ts",
+    "printVersion": "node --version && npm --version",
+    "test:unit": "npm-run-all printVersion build:es5 test:unit:js test:unit:ts",
     "test:unit:js": "nyc --report-dir=./coverage/js --extension=.js --include=lib/**/* --reporter=html --reporter=lcov --reporter=text mocha ./test/unit/index.js",
     "test:unit:ts": "nyc --report-dir=./coverage/ts --extension=.ts --include=lib/**/* --reporter=html --reporter=lcov --reporter=text mocha -r ts-node/register ./test/unit/index.ts",
     "test:serversiderender": "mocha ./test/serversiderender/index.js",