@tencent-rtc/mcp 1.3.4 → 1.3.6

package/resource/doc/rtcengine/web/plugin/26-advanced-publish-cdn-stream.md

@@ -0,0 +1,451 @@
+## Feature Description
+
+This document describes how to use the CDNStreaming plugin to implement cloud-based stream mixing, relaying streams to CDN, and pushing mixed streams back to TRTC rooms.
+
+## Prerequisites
+
+1. [TRTC SDK](https://www.npmjs.com/package/trtc-sdk-v5) version must be >= 5.1.
+2. Enable [Tencent Cloud Live service](https://console.tencentcloud.com/live/domainmanage).
+3. In the [TRTC Console - Advanced Features](https://console.trtc.io/features), click "Relay to CDN", and enable the "Enable relay to CDN" switch.
+   - When the "Global auto relay" switch is enabled, all upstream audio and video streams in TRTC will be automatically relayed to Tencent Cloud CDN ("Global Auto relay" mode).
+   - When the "Global auto relay" switch is disabled, you can use this plugin or the server REST API to relay specific audio and video streams to the CDN and specify the playback address ("Designated Stream relay" mode).
+  
+  <div align="center">
+    <img src="assets/en-cdn-relay-switch.png" style="max-width: 1000px;" />
+  </div>
+
+## Getting Started
+
+### Install & Register
+
+```js
+import TRTC from 'trtc-sdk-v5';
+import { CDNStreaming, PublishMode } from 'trtc-sdk-v5/plugins/cdn-streaming';
+
+const trtc = TRTC.create({ plugins: [CDNStreaming] });
+```
+
+### Plugin Modes Overview
+
+This plugin provides four streaming modes:
+
+| Mode | PublishMode | Description |
+|------|-------------|-------------|
+| A | `PublishMainStreamToCDN` | Relay camera and microphone stream to CDN |
+| B | `PublishSubStreamToCDN` | Relay screen sharing stream to CDN |
+| C | `PublishMixStreamToCDN` | Mix multiple streams and relay to CDN |
+| D | `PublishMixStreamToRoom` | Mix multiple streams and push back to TRTC room |
+
+For each mode, the following lifecycle applies:
+
+- `trtc.startPlugin('CDNStreaming', options)` - Start the plugin
+- `trtc.updatePlugin('CDNStreaming', options)` - Update parameters (0 or N times)
+- `trtc.stopPlugin('CDNStreaming', options)` - Stop the plugin
+
+For detailed `options` parameters, see [CDNStreamingOptions](./global.html#CDNStreamingOptions).
+
+## Relay Single Stream to CDN
+
+Relay camera stream (Mode A) or screen sharing stream (Mode B) to Tencent Cloud CDN.
+
+### Step 1: Select Relay Mode
+
+In [TRTC Console - Advanced Features](https://console.trtc.io/features), select your relay mode:
+
+**Global Auto-Relay Mode:**
+
+Streams will be **automatically** pushed to CDN with default playback address:
+
+`http://[playback-domain]/live/[streamId].flv`
+
+- `[playback-domain]`: The playback domain configured in [Domain Management](https://console.tencentcloud.com/live/livestat)
+- Default `[streamId]` for camera: `${sdkAppId}_${roomId}_${userId}_main`
+- Default `[streamId]` for screen sharing: `${sdkAppId}_${roomId}_${userId}_aux`
+- When roomId or userId contains special characters, streamId uses MD5: `${sdkAppId}_${md5(${roomId}_${userId}_main)}`
+
+**Manual Relay Mode:**
+
+Streams will **not** be automatically pushed. Follow Step 2 to manually enable streaming.
+
+### Step 2: Start Relay
+
+```javascript
+// Start relaying camera stream
+const options = {
+  target: {
+    publishMode: PublishMode.PublishMainStreamToCDN
+  }
+}
+try {
+  await trtc.startPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming start failed', error);
+}
+```
+
+Playback address: `http://[playback-domain]/live/${sdkAppId}_${roomId}_${userId}_main.flv`
+
+### Step 3: Update StreamId (Optional)
+
+```javascript
+// Customize the playback address
+const options = {
+  target: {
+    publishMode: PublishMode.PublishMainStreamToCDN,
+    streamId: 'stream001'
+  }
+}
+try {
+  await trtc.updatePlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming update failed', error);
+}
+```
+
+New playback address: `http://[playback-domain]/live/stream001.flv`
+
+### Step 4: Stop Relay
+
+```javascript
+const options = {
+  target: {
+    publishMode: PublishMode.PublishMainStreamToCDN,
+  }
+}
+try {
+  await trtc.stopPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming stop failed', error);
+}
+```
+
+You can view the current stream status through the [Stream Management Interface](https://console.tencentcloud.com/live/domainmanage).
+
+## Relay Mixed Stream to CDN
+
+Mix multiple audio and video streams in the room into one stream and relay to CDN.
+
+For other platform implementations, please refer to [Cloud-Based Stream Mixing and Transcoding](https://trtc.io/document/47858).
+
+### How It Works
+
+This plugin sends a command to Tencent Cloud's transcoding server to mix multiple streams. You can customize:
+- `encoding`: Output stream quality (resolution, bitrate, framerate)
+- `mix`: Layout configuration for mixed streams
+
+<img src="https://cloudcache.intl.tencent-cloud.com/cms/backend-cms/fb99be746bf211efbd54525400f69702.png" style="width: 1000px">
+
+### Example Workflow
+
+1. Call `startPlugin`, the mixing layout is as shown in Figure A
+2. Call `updatePlugin` to add screen sharing, the mixing layout is as shown in Figure B
+3. Call `stopPlugin` when mixing is no longer needed
+
+<img src="https://write-document-release-1258344699.cos.ap-guangzhou.tencentcos.cn/100025252093/8f489b66a80411f0bf2352540044a08e.png" style="width: 600px">
+
+### Configuration Guide
+
+1. Set `target.publishMode` to `PublishMode.PublishMixStreamToCDN`
+
+2. Configure `encoding` for output quality: videoWidth, videoHeight, videoBitrate, videoFramerate, videoGOP, audioSampleRate, audioBitrate, audioChannels
+
+3. Configure `mix.backgroundColor` and `mix.backgroundImage` for background
+   > Background images need to be uploaded in [TRTC Console - Advanced Features](https://console.trtc.io/features) > Material management. Use the "Image ID" as string for backgroundImage (e.g., "63").
+   > <img src="assets/en-cdn-console-image.png" style="width: 600px">
+
+4. Configure `mix.audioMixUserList` for audio mixing (empty = mix all users)
+
+5. Configure `mix.videoLayoutList` for video layout
+
+> Coordinate system: the origin (0, 0) is the **top-left corner** of the output canvas, in pixels. locationX/locationY is the top-left position of each stream; width/height is the layout region size. Coordinates/sizes are tied to encoding.videoWidth/videoHeight - if you change resolution, scale layout parameters accordingly to avoid misalignment.
+>
+> For full parameter details, see [CDNStreamingOptions](./global.html#CDNStreamingOptions).
+
+### Start Mixing
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToCDN,
+    streamId: 'mix-stream',
+  },
+  encoding: {
+    videoWidth: 1280,
+    videoHeight: 720,
+    videoBitrate: 1500,
+    videoFramerate: 15,
+  },
+  mix: {
+    audioMixUserList: [
+      { userId: 'current_user_id' }
+    ],
+    videoLayoutList: [
+      {
+        fixedVideoUser: { userId: 'current_user_id' },
+        width: 640,
+        height: 480,
+        locationX: 0,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_MAIN,
+        zOrder: 1
+      },
+    ]
+  }
+};
+try {
+  await trtc.startPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming start failed', error);
+}
+```
+
+Playback address: `http://[playback-domain]/live/mix-stream.flv`
+
+### Update Layout
+
+```javascript
+// Add screen sharing to the mix
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToCDN,
+    streamId: 'mix-stream',
+  },
+  encoding: {
+    videoWidth: 1280,
+    videoHeight: 720,
+    videoBitrate: 1500,
+    videoFramerate: 15,
+  },
+  mix: {
+    audioMixUserList: [
+      { userId: 'current_user_id' }
+    ],
+    videoLayoutList: [
+      {
+        fixedVideoUser: { userId: 'current_user_id' },
+        width: 640,
+        height: 480,
+        locationX: 0,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_MAIN,
+        zOrder: 1
+      },
+      {
+        fixedVideoUser: { userId: 'current_user_id' },
+        width: 640,
+        height: 480,
+        locationX: 640,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_SUB,
+        zOrder: 1
+      },
+    ]
+  }
+}
+try {
+  await trtc.updatePlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming update failed', error);
+}
+```
+
+### Stop Mixing
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToCDN,
+  }
+}
+try {
+  await trtc.stopPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming stop failed', error);
+}
+```
+
+## Push to Third-party CDN
+
+Push streams to a CDN from any cloud service provider.
+
+### Step 1: Get CDN Push Address
+
+Obtain a CDN stream push address (url) from your cloud service provider.
+
+### Step 2: Get bizId and appId
+
+In [TRTC Console - Advanced Features](https://console.trtc.io/features), find your push domain (format: `xxxxx.push.tlivecloud.com`):
+- `xxxxx` is the `bizId`
+- Use your SDKAppId as `appId`
+
+<div align="center">
+  <img src="assets/en-cdn-stream-manager.png" style="width: 600px">
+</div>
+
+### Step 3: Start Publishing
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMainStreamToCDN,
+    appId: 0,    // Your appId
+    bizId: 0,    // Your bizId
+    url: ''      // Your CDN push address
+  }
+};
+try {
+  await trtc.startPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming start customCDN failed', error);
+}
+```
+
+### Stop Publishing
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMainStreamToCDN,
+  }
+}
+try {
+  await trtc.stopPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming stop customCDN failed ', error);
+}
+```
+
+## Push Back to TRTC Room
+
+Mix streams and push the mixed stream back to a specified TRTC room as a robot user.
+
+> **Note:**
+> - A stream-pulling robot with userId `mcu_robot_${roomId}_${userId}` will automatically enter the current room. Avoid userId conflicts.
+> - This feature will incur transcoding fees for stream mixing.
+> - When `audioMixUserList` is empty, all users are mixed by default. Pass a non-existent userId if you don't need audio mixing.
+> - For full parameter details, see [CDNStreamingOptions](./global.html#CDNStreamingOptions).
+
+### Start Push Back
+
+```javascript
+// User user_123 in room 3333 initiates this task
+// Mixed stream will be published as robot push_back_user_321 in room 4444
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToRoom,
+    robotUser: {
+      userId: 'push_back_user_321', // Recommend dynamic generation
+      roomId: 4444,
+    }
+  },
+  encoding: {
+    videoWidth: 1280,
+    videoHeight: 720,
+    videoBitrate: 1500,
+    videoFramerate: 15,
+  },
+  mix: {
+    audioMixUserList: [
+      { userId: 'user_123', roomId: 3333 }
+    ],
+    videoLayoutList: [
+      {
+        fixedVideoUser: { userId: 'user_123', roomId: 3333 },
+        width: 640,
+        height: 480,
+        locationX: 0,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_MAIN,
+        zOrder: 1
+      },
+      {
+        fixedVideoUser: { userId: 'user_123', roomId: 3333 },
+        width: 640,
+        height: 480,
+        locationX: 640,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_SUB,
+        zOrder: 1
+      },
+    ]
+  }
+};
+try {
+  await trtc.startPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming start push stream to room failed', error);
+}
+```
+
+### Update Layout
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToRoom
+  },
+  mix: {
+    videoLayoutList: [
+      {
+        fixedVideoUser: { userId: 'user_123', roomId: 3333 },
+        width: 640,
+        height: 480,
+        locationX: 640,
+        locationY: 0,
+        fixedVideoStreamType: TRTC.TYPE.STREAM_TYPE_MAIN,
+        zOrder: 1
+      }
+    ]
+  }
+};
+try {
+  await trtc.updatePlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming update push stream to room failed', error);
+}
+```
+
+### Stop Push Back
+
+```javascript
+let options = {
+  target: {
+    publishMode: PublishMode.PublishMixStreamToRoom,
+  }
+}
+try {
+  await trtc.stopPlugin('CDNStreaming', options);
+} catch (error) {
+  console.error('CDNStreaming stop push stream to room failed ', error);
+}
+```
+
+## API Description
+
+For `startPlugin('CDNStreaming', options)` and `updatePlugin('CDNStreaming', options)`, parameter types can be found in [CDNStreamingOptions](./global.html#CDNStreamingOptions).
+
+For `stopPlugin('CDNStreaming', options)`, the parameter only needs to fill in the `options.target.publishMode` field.
+
+## Notes
+
+1. To record mixed audio streams, please refer to [Cloud Recording](https://trtc.io/document/45169).
+
+2. Cloud-based stream mixing and transcoding requires decoding and re-encoding the audio and video streams input to the MCU cluster, which will incur additional service costs. Therefore, TRTC will charge users who use the MCU cluster for cloud-based stream mixing and transcoding an additional value-added fee. Cloud-based stream mixing and transcoding fees are based on the **resolution of the transcoded output** and **transcoding duration**. The higher the resolution of the transcoded output and the longer the transcoding time, the higher the cost. For details, please refer to [Cloud-Based Stream Mixing and Transcoding Billing Description](https://trtc.io/document/47858?product=rtcengine&menulabel=core%20sdk&platform=android#25e78f25-3514-4608-a85f-f6db86685684).
+
+3. For audio and video duration billing, please refer to [Billing Description](https://trtc.io/zh/document/47858?product=rtcengine&menulabel=core%20sdk&platform=android#25e78f25-3514-4608-a85f-f6db86685684).
+
+4. After leaving the room, billing will not continue, but when re-entering the room, it will automatically continue to start. If you want to directly terminate CDN relay and mixing, please execute stopPlugin for the corresponding publishMode.
+
+## FAQ
+
+Q. Unable to stop streaming?
+
+Please check if "Global Auto-Relay" is enabled in the console. In this mode, streaming cannot be stopped through the API.
+
+Q. How to view all current streams?
+
+You can view the current stream status through the [Stream Management Interface](https://console.tencentcloud.com/live/streammanage).
+
+Q. Screen sharing is blurry?
+
+The SDK uses `1080p` parameter configuration by default to capture screen sharing. Please refer to the interface: {@link TRTC#startScreenShare startScreenShare}

package/resource/doc/rtcengine/web/plugin/27-advanced-small-stream.md

@@ -0,0 +1,151 @@
+
+## Function Description
+
+In the multi-person video call scenario, as the number of streaming users increases, both bandwidth consumption and device performance consumption will rise for users who are subscribing streams. If no optimization is done, the experience on low-end and mid-range devices will be greatly compromised.
+
+This article mainly introduces the best practices for multi-person video scenarios, using the "small streams + only subscribe video streams from the visible area" feature to reduce bandwidth consumption and ensure the best call experience.
+
+## Enable small stream
+
+The small stream refers to encoding two video streams at the same time when the camera is turned on. One is a normal resolution video (referred to as the main stream), and the other is a low-resolution video (referred to as the small stream). At the receiving end, you can choose to receive the main stream or the small stream to save bandwidth.
+
+Applicable scenarios: The small stream is suitable for multi-person audio and video communication scenarios. Generally, in the page layout of multi-person communication scenarios, there will be "a big view and a lot of small view". The video in the big view is the main stream, and the video in the small view is the small stream. This function will increase the upstream bandwidth slightly, but it can greatly save the downstream bandwidth.
+
+### Implementation Steps
+
+#### Enable small streams on the publishing end
+
+When calling [startLocalVideo](./TRTC.html#startLocalVideo), set the small parameter to enable small stream encoding. In an environment that does not support enabling small streams, the small parameter will not take effect and will not report an error.
+
+```js
+await trtc.startLocalVideo({ option: { small: '120p' }});
+```
+
+Since version 5.3.0+, you can turn on/off dynamically the small stream.
+
+```js
+await trtc.startLocalVideo();
+
+// turn on small stream
+await trtc.updateLocalVideo({ option: { small: '120p' }});
+
+// turn off small stream
+await trtc.updateLocalVideo({ option: { small: false }});
+```
+
+#### Subscribe the big stream or the small stream on the receiving end
+
+When calling [startRemoteVideo](./TRTC.html#startRemoteVideo), set the small parameter to subscribe the small stream. Otherwise, the big stream is subscribed by default. When subscribing the small stream, there is no need to check whether the remote user has published the small stream in advance. If the remote end does not publish the small stream, the SDK will automatically subscribe the big stream.
+
+```js
+trtc.on(TRTC.EVENT.REMOTE_VIDEO_AVAILABLE, ({ userId, streamType }) => {
+  trtc.startRemoteVideo({ 
+    view: '', // Pass in the elementId or element instance object of the video container.
+    userId, 
+    streamType, 
+    option: { small: true }
+  });
+})
+```
+
+Switch to the specified userId's small stream when filling in the small parameter during updateRemoteVideo. It is commonly used to implement the following functions: when the user clicks the "small view", play remote video on "big view"; when the user clicks the "big view", play remote video on "small view".
+
+```js
+// Switch to "big view" playback when the user clicks the "small view".
+await trtc.updateRemoteVideo({ 
+  view: '', // Pass in the elementId or element instance object of the "big view" video container.
+  userId: '', // Pass in the userId of the remote user.
+  streamType: TRTC.TYPE.STREAM_TYPE_MAIN, 
+  option: { small: false }
+})
+// Switch to "small view" playback when the user clicks the "big view".
+await trtc.updateRemoteVideo({ 
+  view: '', // Pass in the elementId or element instance object of the "small view" video container.
+  userId: '', // Pass in the userId of the remote user.
+  streamType: TRTC.TYPE.STREAM_TYPE_MAIN, 
+  option: { small: true }})
+```
+
+### Note
+
+1. TRTC's video stream has a main stream and an sub stream. The main stream includes the big stream and the small stream, and the sub stream is generally used for screen sharing.
+2. When subscribing the stream, the main stream defaults to subscribe the big stream, and the small parameter can be used to subscribe the small stream.
+
+    Supports subscribe the big stream + sub stream, small stream + sub stream, and does not support subscribe the big stream + small stream at the same time. The big stream and the small stream are played in the same video view.
+
+3. Only when the publishing end enables the small stream, the remote stream will be switched to the small stream when the small parameter is specified. If there is no small stream, the SDK will automatically subscribe the big stream.
+4. There is currently no event notification for successful switching between big and small streams.
+5. The small stream resolution cannot be larger than the resolution of the big stream. Otherwise, the setting will be invalid and the original small stream resolution (default 120p) will be kept.
+6. The actual encoding resolution of the small stream will be modified according to the aspect ratio of the big stream, and the aspect ratio of the big stream and the small stream will be consistent.
+
+## Enable "receiveWhenViewVisible"
+
+In general, in a multi-person video scenario, due to limited page space, the streaming end will not display all videos on the page. Therefore, for those remote videos that do not need to be displayed on the page, they do not need to be streamed. Only when the view is visible should the video stream be pulled. This is to save traffic and reduce performance overhead.
+
+Implementing this feature is not easy. Starting with version 5.4.0, the SDK has built-in support for this feature. You only need to set the `receiveWhenViewVisible` and `viewRoot` parameters when calling [startRemoteVideo](./TRTC.html#startRemoteVideo) to enable this feature, which is very convenient and fast. The SDK will automatically check the visibility state of the `view` passed in when startRemoteVideo. Video streaming will start when the `view` is visible and stop when `view` is not visible.
+
+### Sample code
+
+Assuming your DOM structure for playing videos is as follows:
+
+```html
+<div id='view-list'>
+  <div id='view1_main'></div>
+  <div id='view2_main'></div>
+  <div id='view3_main'></div>
+</div>
+```
+
+```js
+trtc.on(TRTC.EVENT.REMOTE_VIDEO_AVAILABLE, ({ userId, streamType }) => {
+  trtc.startRemoteVideo({ 
+    view: 'view1_main',
+    userId, 
+    streamType, 
+    option: { 
+      small: true,
+      receiveWhenViewVisible: true, // Enable "only subscribe video streams from the visible area"
+      viewRoot:  document.getElementById('view-list') // Pass in the first-level parent element of all view list, which is used to calculate the reference relationship between view and viewRoot and check whether the view is visible. The default value is document.body.
+    }
+  });
+})
+```
+
+### Add loading effect for better user experience
+
+After enabling this feature, when the user scrolls through the video list:
+
+- When the view changes from visible to invisible, the SDK will stop streaming.
+- When the view changes from invisible to visible, the SDK will resume streaming.
+
+Since resuming streaming until the video playback is successful is an asynchronous process, a black screen may appear during this process. To provide a better user experience, we recommend adding a loading effect to the video area during this asynchronous process. You can refer to the following code to determine whether to display the loading effect.
+
+```js
+let isLoading = true;
+
+trtc.on(TRTC.EVENT.VIDEO_PLAY_STATE_CHANGED, ({ userId, streamType, state }) => {
+  // '' is local video
+  if (userId === '') return;
+
+  // set isLoading to false when remote video is playing.
+  isLoading = state !== 'PLAYING';
+
+  // You can confirm which remote user's main/sub stream playback state has changed by userId and streamType.
+});
+await trtc.startRemoteVideo({  });
+
+trtc.stopRemoteVideo()
+isLoading = false;
+```
+
+## FAQ
+
+1. How does the SDK confirm whether a view is visible?
+
+    When any pixel point of the view appears in the viewRoot area, the view is considered visible; conversely, when all pixel points of the view are not in the viewRoot area, the view is considered invisible.
+
+2. Do I need to consider browser compatibility for big and small streams?
+
+    Although some browsers do not support enabling big and small streams (such as all browsers on iOS systems and Chrome versions below 63), enabling small stream has no side effects. If the environment does not support it, the SDK will only use the main stream, and the call can proceed normally. You do not need to write additional if-else statements.
+
+    You can call {@link TRTC.isSupported TRTC.isSupported} to get checkResult.detail.isSmallStreamSupported to confirm whether the current environment supports enabling small stream.