@tencent-rtc/mcp 1.3.4 → 1.3.6

package/resource/doc/rtcengine/web/plugin/42-advanced-realtime-transcriber.md

@@ -0,0 +1,315 @@
+## Overview
+
+This document describes how to use the Realtime Transcriber plugin to enable real-time speech-to-text (ASR) and translation capabilities. This plugin can convert users' speech in a room to text in real-time and supports translation into multiple languages.
+
+## Prerequisites
+
+- TRTC Web SDK version >= 5.15.0
+
+## Implementation
+
+### Import and Register Plugin
+
+```javascript
+import TRTC from 'trtc-sdk-v5';
+import RealtimeTranscriber from 'trtc-sdk-v5/plugins/realtime-transcriber';
+
+const trtc = TRTC.create({ plugins: [RealtimeTranscriber] });
+```
+
+### Start Plugin
+
+Starting the plugin returns a `transcriberRobotId`, which is used to stop the transcription task later.
+
+```javascript
+const transcriberRobotId = await trtc.startPlugin('RealtimeTranscriber', {
+  sourceLanguage: 'zh',
+  translationLanguages: ['en', 'ja'],
+  userIdsToTranscribe: 'all',
+});
+```
+
+### Listen for Transcription Messages
+
+Listen to the `REALTIME_TRANSCRIBER_MESSAGE` event to receive transcription results.
+
+```javascript
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (event) => {
+  console.log(`${event.speakerUserId}: ${event.sourceText}`);
+  if (event.translationTexts) {
+    Object.entries(event.translationTexts).forEach(([lang, text]) => {
+      console.log(`  ${lang}: ${text}`);
+    });
+  }
+});
+```
+
+### Listen for State Changes
+
+Listen to the `REALTIME_TRANSCRIBER_STATE_CHANGED` event to receive transcription task state changes.
+
+```javascript
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_STATE_CHANGED, (event) => {
+  console.log(`Transcription state: ${event.state}, robotId: ${event.transcriberRobotId}`);
+});
+```
+
+### Stop Plugin
+
+```javascript
+await trtc.stopPlugin('RealtimeTranscriber', { transcriberRobotId });
+```
+
+## Usage Examples
+
+### ASR Scenario: Transcribe Audio to Text
+
+```javascript
+// Start transcription task
+const robotId = await trtc.startPlugin('RealtimeTranscriber', {
+  sourceLanguage: 'zh'
+});
+
+// Listen for transcription messages
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (data) => {
+  console.log(data.speakerUserId + ' says: ' + data.sourceText);
+});
+```
+
+### Translation Scenario: Translate Specific User's Speech into Multiple Languages
+
+```javascript
+// Start transcription task, transcribe userA's speech and translate to French and English
+const robotId = await trtc.startPlugin('RealtimeTranscriber', {
+  sourceLanguage: 'zh',
+  translationLanguages: ['fr', 'en'],
+  userIdsToTranscribe: 'userA'
+});
+
+// Listen for transcription messages
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (data) => {
+  console.log(`👤 ${data.speakerUserId}:`);
+  console.log(`  Original: ${data.sourceText}`);
+  
+  if (data.translationTexts) {
+    Object.entries(data.translationTexts).forEach(([lang, text]) => {
+      console.log(`  ${lang.toUpperCase()}: ${text}`);
+    });
+  }
+});
+```
+
+### Transcribe Multiple Specific Users
+
+```javascript
+const robotId = await trtc.startPlugin('RealtimeTranscriber', {
+  sourceLanguage: 'en',
+  translationLanguages: ['zh', 'ja'],
+  userIdsToTranscribe: ['userA', 'userB', 'userC']
+});
+```
+
+## API Reference
+
+### trtc.startPlugin('RealtimeTranscriber', options)
+
+Starts a real-time transcription task.
+
+**options: RealtimeTranscriberStartOptions**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| sourceLanguage | `string` | Required | Source language code, currently supports only `'zh'` and `'en'` |
+| translationLanguages | `string` \| `string[]` | Optional | Target translation language(s), supports single language or array, e.g., `'en'` or `['en', 'ja']` |
+| userIdsToTranscribe | `string` \| `string[]` | Optional | User ID(s) to transcribe, defaults to `'all'` for all users, can also specify single or multiple user IDs |
+| transcriberRobotId | `string` | Optional | Custom transcriber robot ID, auto-generated if not provided |
+
+**Returns: Promise\<string\>**
+
+Returns the `transcriberRobotId` for the transcription task, used to stop the task later.
+
+**Example:**
+
+```javascript
+const robotId = await trtc.startPlugin('RealtimeTranscriber', {
+  sourceLanguage: 'zh',
+  translationLanguages: ['en', 'ja'],
+  userIdsToTranscribe: 'all',
+});
+```
+
+### trtc.stopPlugin('RealtimeTranscriber', options)
+
+Stops a real-time transcription task.
+
+**options**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| transcriberRobotId | `string` | Required | The transcriber robot ID returned when starting |
+
+**Example:**
+
+```javascript
+await trtc.stopPlugin('RealtimeTranscriber', { transcriberRobotId: robotId });
+```
+
+## Event Reference
+
+### TRTC.EVENT.REALTIME_TRANSCRIBER_STATE_CHANGED
+
+Transcription task state change event.
+
+**RealtimeTranscriberStateEvent**
+
+| Name | Type | Description |
+|------|------|-------------|
+| state | `'started'` \| `'stopped'` | Transcription task state |
+| roomId | `string` \| `number` | Room ID |
+| transcriberRobotId | `string` | Transcriber robot ID |
+| sourceLanguage | `string` | Source language (optional) |
+| error | `number` | Error code (optional) |
+| errorMessage | `string` | Error message (optional) |
+
+**Example:**
+
+```javascript
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_STATE_CHANGED, (event) => {
+  if (event.state === 'started') {
+    console.log('Transcription task started');
+  } else if (event.state === 'stopped') {
+    console.log('Transcription task stopped');
+  }
+});
+```
+
+### TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE
+
+Transcription message event, triggered when new transcription results are available.
+
+**RealtimeTranscriberMessageEvent**
+
+| Name | Type | Description |
+|------|------|-------------|
+| segmentId | `string` | Speech segment ID, used to identify a continuous speech segment |
+| speakerUserId | `string` | Speaker's user ID |
+| sourceText | `string` | Transcribed source text |
+| translationTexts | `Array<{ language: string, text: string }>` | Translation texts array, e.g., `[{ language: 'en', text: 'hello' }, { language: 'ja', text: 'こんにちは' }]` |
+| timestamp | `number` | Timestamp |
+| isCompleted | `boolean` | Whether the speech segment is completed. `false` indicates intermediate result (real-time update), `true` indicates final result |
+| robotId | `string` | Transcriber robot ID |
+
+**Example:**
+
+```javascript
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (event) => {
+  const { speakerUserId, sourceText, translationTexts, isCompleted } = event;
+  
+  // Display real-time transcription result
+  console.log(`[${isCompleted ? 'Completed' : 'In Progress'}] ${speakerUserId}: ${sourceText}`);
+  
+  // Display translation results
+  if (translationTexts) {
+    translationTexts.forEach(({ language, text }) => {
+      console.log(`  ${language}: ${text}`);
+    });
+  }
+});
+
+// Remove listener
+trtc.off(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, handler);
+```
+
+## Supported Languages
+
+### ASR Source Languages
+
+| Language | Code |
+|----------|------|
+| Chinese | `zh` |
+| English | `en` |
+
+<!--
+| Traditional Chinese | `zh-tw` |
+| Chinese Dialects | `zh-dialect` |
+| Cantonese | `zh-yue` |
+| Vietnamese | `vi` |
+| Japanese | `ja` |
+| Korean | `ko` |
+| Indonesian | `id` |
+| Thai | `th` |
+| Portuguese | `pt` |
+| Turkish | `tr` |
+| Arabic | `ar` |
+| Spanish | `es` |
+| Hindi | `hi` |
+| French | `fr` |
+| Malay | `ms` |
+| Filipino | `fil` |
+| German | `de` |
+| Italian | `it` |
+| Russian | `ru` |
+| Swedish | `sv` |
+| Danish | `da` |
+| Norwegian | `no` |
+-->
+
+### Translation Target Languages
+
+| Language | Code |
+|----------|------|
+| Chinese | `zh` |
+| English | `en` |
+| Vietnamese | `vi` |
+| Japanese | `ja` |
+| Korean | `ko` |
+| Indonesian | `id` |
+| Thai | `th` |
+| Portuguese | `pt` |
+| Arabic | `ar` |
+| Spanish | `es` |
+| French | `fr` |
+| Malay | `ms` |
+| German | `de` |
+| Italian | `it` |
+| Russian | `ru` |
+
+## FAQ
+
+**1. How to distinguish between intermediate and final results?**
+
+Use the `isCompleted` field in the `REALTIME_TRANSCRIBER_MESSAGE` event. When `isCompleted` is `false`, it indicates an intermediate result (real-time update); when `isCompleted` is `true`, it indicates the speech segment is completed with the final result.
+
+```javascript
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (event) => {
+  if (event.isCompleted) {
+    // Final result, can be stored or displayed
+    saveSubtitle(event);
+  } else {
+    // Intermediate result, can update UI in real-time
+    updateLiveCaption(event);
+  }
+});
+```
+
+**2. How to merge subtitles by segmentId?**
+
+Each continuous speech segment has the same `segmentId`, which can be used to update and merge subtitles:
+
+```javascript
+const subtitleMap = new Map();
+
+trtc.on(TRTC.EVENT.REALTIME_TRANSCRIBER_MESSAGE, (event) => {
+  subtitleMap.set(event.segmentId, event);
+  
+  if (event.isCompleted) {
+    // Speech segment completed, can be archived
+    archiveSubtitle(subtitleMap.get(event.segmentId));
+  }
+});
+```
+
+**3. How to save bandwidth?**
+
+When you don't need to receive transcription messages, remove all listeners for the `REALTIME_TRANSCRIBER_MESSAGE` event, and the plugin will automatically pause receiving messages to save bandwidth. When listeners are added again, message receiving will automatically resume.
+

package/resource/doc/rtcengine/web/plugin/43-advanced-face-detection.md

@@ -0,0 +1,180 @@
+## Function Description
+
+This document details how to use the TRTC Web SDK's Face Detection plugin to monitor real-time face presence changes in camera video streams. The plugin employs intelligent debounce logic, triggering callbacks only when face appearance/disappearance states genuinely change, preventing false positives caused by lighting variations or brief obstructions.
+
+## Prerequisites
+
+- TRTC Web SDK version >= 5.15.0
+- Camera video stream must be enabled first.
+
+### Platform Compatibility
+
+| Platform | Operating System | Browser Version | Notes |
+| :--- | :--- | :--- | :--- |
+| Web | Windows | Chrome 90+, Firefox 90+, Edge 97+ | Recommended: Latest Chrome browser (with hardware acceleration enabled) |
+| Web | Mac | Chrome 98+, Firefox 96+, Safari 15+ | 2019 or newer MacBook recommended |
+| Web | Android | Chrome、Firefox| Use of Chrome or Firefox is recommended |
+| Web | iOS | Chrome、Safari、Firefox | Use of Chrome or Safari is recommended |
+
+## Implementation Steps
+
+### Deploying Static Resources
+
+The plugin relies on static resource files. To ensure the browser can load and run these files correctly, you need to complete the following steps:
+
+1.  Publish the `node_modules/trtc-sdk-v5/assets` directory to your CDN or static resource server.
+2.  Pass your CDN address to the `assetsPath` parameter when creating the TRTC instance, e.g., `TRTC.create({ assetsPath: 'https://xxxx/assets' })`. The SDK will load the necessary resource files on demand.
+
+> - If your SDK version is lower than `v5.10.0`, you need to download and extract https://web.sdk.qcloud.com/trtc/webrtc/v5/assets/assets.zip, then publish it to your CDN or static resource server.
+> - If the Host URL of the files differs from your web application's Host URL, you need to enable CORS policy for the file domain.
+> - Do not host the directory files under an HTTP service, as loading HTTP resources under an HTTPS domain will be blocked by the browser's security policy.
+
+### Registering the Plugin
+
+```javascript
+import { FaceDetection } from 'trtc-sdk-v5/plugins/video-effect/face-detection';
+
+const trtc = TRTC.create({ 
+  plugins: [FaceDetection], 
+  assetsPath: 'https://xxxx/assets'  // Replace with your static resource address
+});
+```
+
+### Starting the Local Camera
+
+```javascript
+await trtc.startLocalVideo();
+```
+
+### Enabling the Face Detection Plugin
+
+```javascript
+await trtc.startPlugin('FaceDetection', {
+  onFaceDetectionStateChanged: (hasFace) => {
+    if (hasFace) {
+      console.log('Face detected');
+      // Execute business logic for face appearance
+    } else {
+      console.log('Face disappeared');
+      // Execute business logic for face disappearance
+    }
+  },
+  detectionInterval: 300,    // Detection interval in ms (Optional, default: 300)
+  minConfidence: 0.8,        // Minimum confidence threshold (Optional, default: 0.8)
+  missingTimeout: 1000       // Disappearance confirmation time in ms (Optional, default: 1000)
+});
+```
+
+### Updating Parameters as Needed
+
+```javascript
+// Update detection parameters - increase sensitivity
+await trtc.updatePlugin('FaceDetection', {
+  onFaceDetectionStateChanged: (hasFace) => {
+    console.log('Updated detection state:', hasFace);
+  },
+  detectionInterval: 200,    // Shorter interval for higher real-time performance
+  minConfidence: 0.7,        // Lower threshold for higher sensitivity
+  missingTimeout: 1500       // Longer timeout to reduce false negatives
+});
+```
+
+### Disabling Face Detection
+
+```javascript
+await trtc.stopPlugin('FaceDetection');
+```
+
+## Core Features
+
+### Intelligent Debounce Logic
+
+The plugin uses an **asymmetric debounce mechanism** to ensure state change accuracy:
+
+- **Face appears (true)**: Triggered immediately for real-time response.
+- **Face disappears (false)**: Triggered after a delay, requiring the `missingTimeout` condition to be met for confirmation.
+
+This design effectively prevents false positives caused by:
+- Brief head turns or looking down
+- Sudden lighting changes
+- Temporary obstructions
+
+### Video Stream State Synchronization
+
+The plugin automatically handles local video stream state changes:
+
+| Scenario | Plugin Behavior | Callback Trigger |
+| :--- | :--- | :--- |
+| Video muted (Mute) | Pauses detection | Triggers once with `hasFace: false` |
+| Video resumed (Unmute) | Resumes detection | Triggers callback based on actual state |
+
+## API Documentation
+
+### trtc.startPlugin('FaceDetection', options)
+
+Used to start the Face Detection plugin.
+
+#### options
+
+| Name | Type | Attributes | Description |
+| :--- | :--- | :--- | :--- |
+| `onFaceDetectionStateChanged` | `(hasFace: boolean) => void` | **Required** | Callback function for face state changes (triggered only on genuine state change) |
+| `detectionInterval` | number | Optional | Detection interval time (ms), range: 100-5000, default: 300 |
+| `minConfidence` | number | Optional | Minimum face confidence threshold, range: 0-1, default: 0.8 |
+| `missingTimeout` | number | Optional | Face disappearance confirmation time (ms), range: 0-5000, default: 1000 |
+
+**Example:**
+
+```javascript
+await trtc.startPlugin('FaceDetection', {
+  onFaceDetectionStateChanged: (hasFace) => {
+    if (hasFace) {
+      console.log('Face appeared');
+    } else {
+      console.log('Face disappeared');
+    }
+  },
+  detectionInterval: 300,
+  minConfidence: 0.8,
+  missingTimeout: 1000
+});
+```
+
+### trtc.updatePlugin('FaceDetection', options)
+
+Used to modify face detection parameters.
+
+#### options
+
+| Name | Type | Attributes | Description |
+| :--- | :--- | :--- | :--- |
+| `onFaceDetectionStateChanged` | `(hasFace: boolean) => void` | Optional | Callback function for face state changes |
+| `detectionInterval` | number | Optional | Detection interval time (ms), range: 100-5000 |
+| `minConfidence` | number | Optional | Minimum face confidence threshold, range: 0-1 |
+| `missingTimeout` | number | Optional | Face disappearance confirmation time (ms), range: 0-5000 |
+
+**Example:**
+
+```javascript
+await trtc.updatePlugin('FaceDetection', {
+  detectionInterval: 200,
+  minConfidence: 0.7,
+  missingTimeout: 1500
+});
+```
+
+### trtc.stopPlugin('FaceDetection')
+
+Stops the Face Detection plugin.
+
+## FAQs
+
+**1. How to adjust detection sensitivity?**
+
+Adjust sensitivity by combining these parameters:
+- **Increase sensitivity**: Lower `minConfidence`, shorten `detectionInterval`
+- **Decrease sensitivity**: Raise `minConfidence`, lengthen `missingTimeout`
+
+**2. Do I need to restart detection after the video stream recovers?**
+
+No. The plugin automatically handles video stream state changes and resumes detection upon recovery.