agora-python-server-sdk 2.1.5__tar.gz → 2.1.6__tar.gz

{agora_python_server_sdk-2.1.5/agora_python_server_sdk.egg-info → agora_python_server_sdk-2.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: agora_python_server_sdk
-Version: 2.1.5
+Version: 2.1.6
 Summary: A Python SDK for Agora Server
 Home-page: https://github.com/AgoraIO-Extensions/Agora-Python-Server-SDK
 Classifier: Intended Audience :: Developers
@@ -29,7 +29,7 @@ Description-Content-Type: text/markdown
   - CentOS 7.0 and above
   
 - Supported Mac versions:
-  - MacOS 13 and above
+  - MacOS 13 and above(only for coding and testing)
 
 - Python version:
   - Python 3.10 and above
@@ -51,6 +51,28 @@ python agora_rtc/examples/example_audio_pcm_send.py --appId=xxx --channelId=xxx
 
 # Change log
 
+## 2024.12.09 Release 2.1.6
+- New Features:
+  -- Added AudioVadManager to manage VAD (Voice Activity Detection) instances.
+  -- Integrated VAD functionality into the SDK. Developers no longer need to worry about how to use VAD; they only need to focus on setting appropriate parameters. Reference: sample_audio_vad.py
+- Changes:
+  -- In register_audio_frame_observer, two new parameters have been added to set the VAD parameters. Reference: sample_audio_vad.py
+  -- In on_playback_audio_frame_before_mixing, two new return values have been added: vad_result_state and vad_result_bytearray.
+    state:
+    < 0: No internal automatic VAD applied
+    0: No speaking
+    1: Started speaking
+    2: Speaking
+    3: Stopped speaking
+    vad_result_bytearray: The result processed by VAD, returned when VAD is active.
+    If automatic VAD is enabled:
+    Developers should use vad_result_bytearray for subsequent business processing (e.g., sending to ASR/STT), rather than using the raw frame data.
+    Reference: sample_audio_vad.py
+- Optimizations:
+  -- Replaced the use of pacer with AudioConsumer for pushing PCM audio.
+- Updates:
+  -- Updated the samples related to Pacer and VAD.
+
 ## 2024.12.03 release Version 2.1.5
 - Modifications:
   - LocalUser/audioTrack:
@@ -138,3 +160,122 @@ Judge the value of state according to the returned state, and do corresponding p
   # Source code: audio_consumer.py
   # Sample code: example_audio_consumer.py
 ### How to release resources?
+## 如何释放资源？
+    localuser.unpublish_audio(audio_track)
+    localuser.unpublish_video(video_track)
+    audio_track.set_enabled(0)
+    video_track.set_enabled(0)
+
+    localuser.unregister_audio_frame_observer()
+    localuser.unregister_video_frame_observer()
+    localuser.unregister_local_user_observer()
+
+    connection.disconnect()
+    connection.unregister_observer()
+
+    localuser.release()
+    connection.release()
+
+    
+    audio_track.release()
+    video_track.release()
+    pcm_data_sender.release()
+    video_data_sender.release()
+    audio_consumer.release()
+
+    media_node_factory.release()
+    agora_service.release()
+    
+    #set to None
+    audio_track = None
+    video_track = None
+    audio_observer = None
+    video_observer = None
+    local_observer = None
+    localuser = None
+    connection = None
+    agora_service = None
+
+## Interrupt Handling in AI Scenarios
+# Definition of Interrupt
+In human-machine dialogue, an interrupt refers to the situation where a user suddenly interrupts the robot's response, requesting the robot to stop its current response immediately and shift to answer the user's new question. This behavior is called an interrupt.
+
+# Trigger Conditions for Interrupts
+Interrupts can be defined in different ways depending on the product. There are generally two modes:
+
+- Mode 1: Voice Activation Mode
+When it detects that the user is speaking, the interrupt strategy is triggered. For example, when the system recognizes speech, it triggers the interrupt strategy to stop the robot's response.
+
+- Mode 2: ASR Activation Mode
+When the system detects that the user is speaking and receives a result from ASR (Automatic Speech Recognition) or STT (Speech-to-Text), the interrupt strategy is triggered.
+
+# Advantages of Different Interrupt Strategies
+Voice Activation Interrupt
+
+Advantages:
+Reduces the user's wait time and the likelihood of interrupts, as the robot will stop its response immediately when the user starts speaking, eliminating the need for the user to wait for the robot to finish speaking.
+Disadvantages:
+Since this is voice-activated, it may be triggered by meaningless audio signals, depending on the accuracy of the VAD (Voice Activity Detection). For example, if someone is typing on the keyboard while the AI is speaking, it might trigger the interrupt incorrectly.
+ASR Activation Interrupt
+
+Advantages:
+Reduces the probability of unnecessary interrupts because the interrupt strategy is triggered only after ASR or STT has recognized the user’s speech.
+Disadvantages:
+Since this is ASR/STT-triggered, it requires converting the audio signal into text, which introduces a delay before the interrupt can be processed.
+- Recommended Mode
+If the VAD can filter out non-speech signals and only triggers when human speech is detected, the Voice Activation Mode is recommended. This mode is also suitable when the delay in processing the interrupt is not a major concern.
+
+If the interrupt delay is not sensitive, the ASR Activation Mode is recommended. This mode can filter out non-speech signals more effectively and reduce the probability of an unintended interrupt.
+
+How to Implement Interrupts? What Actions Are Required?
+In a human-machine dialogue system, conversations are typically structured in "rounds," where each round consists of a question from the user, followed by a response from the robot, and so on. For each round, we can assign a roundId, incrementing it with each new round. A round consists of the following stages:
+
+VAD (Voice Activity Detection):
+This marks the start of the dialogue, where the system detects the beginning and end of the user's speech. It then passes this information to the ASR for further processing.
+
+ASR (Automatic Speech Recognition):
+This phase involves recognizing the user's speech and converting it into text, which is then passed to the LLM (Large Language Model).
+
+LLM (Large Language Model):
+This is the generation phase, where the LLM processes the recognized user input and generates a response.
+
+TTS (Text-to-Speech):
+In this phase, the LLM’s response is converted into an audio format.
+
+RTC Streaming:
+The generated audio is streamed via RTC (Real-Time Communication) to be played back to the user.
+
+Therefore, an interrupt happens when, in the next round (roundId+1), either through Voice Activation (triggered by the VAD phase) or ASR Activation (triggered when ASR recognizes the user’s speech), the following actions must be performed:
+
+Stop the LLM Generation in the current round (roundId).
+Stop the TTS Synthesis in the current round (roundId).
+Stop the RTC Streaming in the current round (roundId).
+API Call References:
+Call: AudioConsumer.clear()
+Call: LocalAudioTrack.clear_sender_buffer()
+Business Layer: Clear any remaining TTS-related data (if applicable)
+
+
+## When to Pass LLM Results to TTS for Synthesis?
+LLM (Large Language Model) results are returned asynchronously and in a streaming manner. When should the results from the LLM be passed to TTS (Text-to-Speech) for synthesis?
+
+Two main factors need to be considered:
+
+Ensure that the TTS synthesized speech is unambiguous:
+The speech synthesized by TTS must be clear, complete, and continuous. For example, if the LLM returns the text: "中间的首都是北京吗？", and we pass it to TTS as:
+
+"中",
+"国首",
+"是北",
+"京吗？",
+This would result in ambiguous synthesis because there are no spaces between certain words (e.g., between "中" and "国", "首" and "是", and "京" and "吗"). Proper segmentation must be ensured to avoid such ambiguities.
+Minimize overall processing delay:
+If the LLM results are passed to TTS only after the entire response is generated, the speech synthesis will be unambiguous and continuous. However, this approach introduces significant delay, which negatively affects the user experience.
+
+Recommended Approach
+To achieve a balance between clarity and minimal delay, the following steps should be followed:
+
+Store the LLM results in a cache as they are received.
+Perform a reverse scan of the cached data to find the most recent punctuation mark.
+Truncate the data from the start to the most recent punctuation mark and pass it to TTS for synthesis.
+Remove the truncated data from the cache. The remaining data should be moved to the beginning of the cache and continue waiting for additional data from the LLM.

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/README.md

@@ -14,7 +14,7 @@
   - CentOS 7.0 and above
   
 - Supported Mac versions:
-  - MacOS 13 and above
+  - MacOS 13 and above(only for coding and testing)
 
 - Python version:
   - Python 3.10 and above
@@ -36,6 +36,28 @@ python agora_rtc/examples/example_audio_pcm_send.py --appId=xxx --channelId=xxx
 
 # Change log
 
+## 2024.12.09 Release 2.1.6
+- New Features:
+  -- Added AudioVadManager to manage VAD (Voice Activity Detection) instances.
+  -- Integrated VAD functionality into the SDK. Developers no longer need to worry about how to use VAD; they only need to focus on setting appropriate parameters. Reference: sample_audio_vad.py
+- Changes:
+  -- In register_audio_frame_observer, two new parameters have been added to set the VAD parameters. Reference: sample_audio_vad.py
+  -- In on_playback_audio_frame_before_mixing, two new return values have been added: vad_result_state and vad_result_bytearray.
+    state:
+    < 0: No internal automatic VAD applied
+    0: No speaking
+    1: Started speaking
+    2: Speaking
+    3: Stopped speaking
+    vad_result_bytearray: The result processed by VAD, returned when VAD is active.
+    If automatic VAD is enabled:
+    Developers should use vad_result_bytearray for subsequent business processing (e.g., sending to ASR/STT), rather than using the raw frame data.
+    Reference: sample_audio_vad.py
+- Optimizations:
+  -- Replaced the use of pacer with AudioConsumer for pushing PCM audio.
+- Updates:
+  -- Updated the samples related to Pacer and VAD.
+
 ## 2024.12.03 release Version 2.1.5
 - Modifications:
   - LocalUser/audioTrack:
@@ -123,3 +145,122 @@ Judge the value of state according to the returned state, and do corresponding p
   # Source code: audio_consumer.py
   # Sample code: example_audio_consumer.py
 ### How to release resources?
+## 如何释放资源？
+    localuser.unpublish_audio(audio_track)
+    localuser.unpublish_video(video_track)
+    audio_track.set_enabled(0)
+    video_track.set_enabled(0)
+
+    localuser.unregister_audio_frame_observer()
+    localuser.unregister_video_frame_observer()
+    localuser.unregister_local_user_observer()
+
+    connection.disconnect()
+    connection.unregister_observer()
+
+    localuser.release()
+    connection.release()
+
+    
+    audio_track.release()
+    video_track.release()
+    pcm_data_sender.release()
+    video_data_sender.release()
+    audio_consumer.release()
+
+    media_node_factory.release()
+    agora_service.release()
+    
+    #set to None
+    audio_track = None
+    video_track = None
+    audio_observer = None
+    video_observer = None
+    local_observer = None
+    localuser = None
+    connection = None
+    agora_service = None
+
+## Interrupt Handling in AI Scenarios
+# Definition of Interrupt
+In human-machine dialogue, an interrupt refers to the situation where a user suddenly interrupts the robot's response, requesting the robot to stop its current response immediately and shift to answer the user's new question. This behavior is called an interrupt.
+
+# Trigger Conditions for Interrupts
+Interrupts can be defined in different ways depending on the product. There are generally two modes:
+
+- Mode 1: Voice Activation Mode
+When it detects that the user is speaking, the interrupt strategy is triggered. For example, when the system recognizes speech, it triggers the interrupt strategy to stop the robot's response.
+
+- Mode 2: ASR Activation Mode
+When the system detects that the user is speaking and receives a result from ASR (Automatic Speech Recognition) or STT (Speech-to-Text), the interrupt strategy is triggered.
+
+# Advantages of Different Interrupt Strategies
+Voice Activation Interrupt
+
+Advantages:
+Reduces the user's wait time and the likelihood of interrupts, as the robot will stop its response immediately when the user starts speaking, eliminating the need for the user to wait for the robot to finish speaking.
+Disadvantages:
+Since this is voice-activated, it may be triggered by meaningless audio signals, depending on the accuracy of the VAD (Voice Activity Detection). For example, if someone is typing on the keyboard while the AI is speaking, it might trigger the interrupt incorrectly.
+ASR Activation Interrupt
+
+Advantages:
+Reduces the probability of unnecessary interrupts because the interrupt strategy is triggered only after ASR or STT has recognized the user’s speech.
+Disadvantages:
+Since this is ASR/STT-triggered, it requires converting the audio signal into text, which introduces a delay before the interrupt can be processed.
+- Recommended Mode
+If the VAD can filter out non-speech signals and only triggers when human speech is detected, the Voice Activation Mode is recommended. This mode is also suitable when the delay in processing the interrupt is not a major concern.
+
+If the interrupt delay is not sensitive, the ASR Activation Mode is recommended. This mode can filter out non-speech signals more effectively and reduce the probability of an unintended interrupt.
+
+How to Implement Interrupts? What Actions Are Required?
+In a human-machine dialogue system, conversations are typically structured in "rounds," where each round consists of a question from the user, followed by a response from the robot, and so on. For each round, we can assign a roundId, incrementing it with each new round. A round consists of the following stages:
+
+VAD (Voice Activity Detection):
+This marks the start of the dialogue, where the system detects the beginning and end of the user's speech. It then passes this information to the ASR for further processing.
+
+ASR (Automatic Speech Recognition):
+This phase involves recognizing the user's speech and converting it into text, which is then passed to the LLM (Large Language Model).
+
+LLM (Large Language Model):
+This is the generation phase, where the LLM processes the recognized user input and generates a response.
+
+TTS (Text-to-Speech):
+In this phase, the LLM’s response is converted into an audio format.
+
+RTC Streaming:
+The generated audio is streamed via RTC (Real-Time Communication) to be played back to the user.
+
+Therefore, an interrupt happens when, in the next round (roundId+1), either through Voice Activation (triggered by the VAD phase) or ASR Activation (triggered when ASR recognizes the user’s speech), the following actions must be performed:
+
+Stop the LLM Generation in the current round (roundId).
+Stop the TTS Synthesis in the current round (roundId).
+Stop the RTC Streaming in the current round (roundId).
+API Call References:
+Call: AudioConsumer.clear()
+Call: LocalAudioTrack.clear_sender_buffer()
+Business Layer: Clear any remaining TTS-related data (if applicable)
+
+
+## When to Pass LLM Results to TTS for Synthesis?
+LLM (Large Language Model) results are returned asynchronously and in a streaming manner. When should the results from the LLM be passed to TTS (Text-to-Speech) for synthesis?
+
+Two main factors need to be considered:
+
+Ensure that the TTS synthesized speech is unambiguous:
+The speech synthesized by TTS must be clear, complete, and continuous. For example, if the LLM returns the text: "中间的首都是北京吗？", and we pass it to TTS as:
+
+"中",
+"国首",
+"是北",
+"京吗？",
+This would result in ambiguous synthesis because there are no spaces between certain words (e.g., between "中" and "国", "首" and "是", and "京" and "吗"). Proper segmentation must be ensured to avoid such ambiguities.
+Minimize overall processing delay:
+If the LLM results are passed to TTS only after the entire response is generated, the speech synthesis will be unambiguous and continuous. However, this approach introduces significant delay, which negatively affects the user experience.
+
+Recommended Approach
+To achieve a balance between clarity and minimal delay, the following steps should be followed:
+
+Store the LLM results in a cache as they are received.
+Perform a reverse scan of the cached data to find the most recent punctuation mark.
+Truncate the data from the start to the most recent punctuation mark and pass it to TTS for synthesis.
+Remove the truncated data from the cache. The remaining data should be moved to the beginning of the cache and continue waiting for additional data from the LLM.

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora/rtc/_ctypes_handle/_audio_frame_observer.py

@@ -5,6 +5,7 @@ import ctypes
 from ..audio_frame_observer import *
 import logging
 logger = logging.getLogger(__name__)
+from  ..audio_vad_manager import AudioVadManager
 #from ..audio_sessionctrl import *
 
 ON_RECORD_AUDIO_FRAME_CALLBACK = ctypes.CFUNCTYPE(ctypes.c_int, AGORA_HANDLE, ctypes.c_char_p, ctypes.POINTER(AudioFrameInner))
@@ -35,7 +36,7 @@ class AudioFrameObserverInner(ctypes.Structure):
         ("on_get_ear_monitoring_audio_frame_param", ON_GET_EAR_MONITORING_AUDIO_FRAME_PARAM_CALLBACK)
     ]
 
-    def __init__(self, observer: IAudioFrameObserver, local_user: 'LocalUser'):
+    def __init__(self, observer: IAudioFrameObserver, local_user: 'LocalUser', enable_vad: int, vad_configure):
         self.observer = observer
         self.local_user = local_user
         self.on_record_audio_frame = ON_RECORD_AUDIO_FRAME_CALLBACK(self._on_record_audio_frame)
@@ -45,6 +46,8 @@ class AudioFrameObserverInner(ctypes.Structure):
         self.on_playback_audio_frame_before_mixing = ON_PLAYBACK_AUDIO_FRAME_BEFORE_MIXING_CALLBACK(self._on_playback_audio_frame_before_mixing)
         self.on_get_audio_frame_position = ON_GET_AUDIO_FRAME_POSITION_CALLBACK(self._on_get_audio_frame_position)
         self._session_ctrl_manager = None #SessionCtrlManager()
+        self._vad_instance_manager = AudioVadManager(vad_configure) if enable_vad else None
+        self._enable_vad = True if enable_vad > 0 else False
 
         # self.on_get_playback_audio_frame_param = ON_GET_PLAYBACK_AUDIO_FRAME_PARAM_CALLBACK(self._on_get_playback_audio_frame_param)
         # self.on_get_record_audio_frame_param = ON_GET_RECORD_AUDIO_FRAME_PARAM_CALLBACK(self._on_get_record_audio_frame_param)
@@ -91,7 +94,15 @@ class AudioFrameObserverInner(ctypes.Structure):
 
         user_id_str = user_id.decode('utf-8')
         frame = audio_frame_inner.contents.get()
-        ret = self.observer.on_playback_audio_frame_before_mixing(self.local_user, channel_id_str, user_id_str, frame)
+        # make a map: key{channel_id, user_id}, value: vadv2 instance
+        # and call back in this call back
+        # when to create and remove the key from map?
+        # in _del_ function to release the vadv2 instance
+        if self._enable_vad:
+            vad_result_state, vad_result_bytes = self._vad_instance_manager.process(channel_id_str, user_id_str, frame)
+            ret = self.observer.on_playback_audio_frame_before_mixing(self.local_user, channel_id_str, user_id_str, frame, vad_result_state, vad_result_bytes)
+        else:
+            ret = self.observer.on_playback_audio_frame_before_mixing(self.local_user, channel_id_str, user_id_str, frame, -1, None)
         return ret
 
     def _on_get_audio_frame_position(self, local_user_handle):
@@ -113,3 +124,10 @@ class AudioFrameObserverInner(ctypes.Structure):
     def _on_get_ear_monitoring_audio_frame_param(self, local_user_handle) -> AudioParams:
         logger.debug(f"AudioFrameObserverInner _on_get_ear_monitoring_audio_frame_param: {local_user_handle}")
         return self.observer.on_get_ear_monitoring_audio_frame_param(self.local_user)
+    def clear(self):
+        #disalbe vad
+        self._enable_vad = False
+        if self._vad_instance_manager:
+            self._vad_instance_manager.release()
+        self._vad_instance_manager = None
+        pass

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora/rtc/audio_frame_observer.py

@@ -14,7 +14,7 @@ class IAudioFrameObserver:
     def on_ear_monitoring_audio_frame(self, agora_local_user, frame):
         return 1
 
-    def on_playback_audio_frame_before_mixing(self, agora_local_user, channelId, uid, frame: AudioFrame):
+    def on_playback_audio_frame_before_mixing(self, agora_local_user, channelId, uid, frame: AudioFrame, vad_result_state:int, vad_result_bytearray:bytearray):
         return 1
 
     def on_get_audio_frame_position(self, agora_local_user):

agora_python_server_sdk-2.1.6/agora/rtc/audio_vad_manager.py

@@ -0,0 +1,59 @@
+import ctypes
+from .agora_base import *
+import ctypes
+from .audio_frame_observer import *
+from .voice_detection import AudioVadV2, AudioVadConfigV2
+import logging
+from threading import Lock
+logger = logging.getLogger(__name__)
+
+# 需要考虑不同的vad configure：也就是说需要触发什么时候做设置？？
+#可以做一个对外的接口：update（channel_id, user_id, vad_config）
+# 然后一个总的configure接口：update_all(channel_id, vad_config)
+# 默认配置用service 的configure
+class AudioVadManager():
+    def __init__(self, configure: AudioVadConfigV2) -> None:
+        self._instance_map = {} # set to dict
+        self._vad_config = configure
+        self._lock = Lock()
+        self._is_init = True
+        pass
+    def _make_key(self, channel_id: str, user_id: str) -> str:
+        return channel_id + user_id
+    def get_vad_instance(self, channel_id: str, user_id: str) -> AudioVadV2:
+        key = self._make_key(channel_id, user_id)
+        with self._lock:
+            return self._instance_map.get(key, None)
+    #note: inner function, not thread safe, but should be ok, since it is called by other thread safe function
+    def _add_vad_instance(self, channel_id: str, user_id: str) -> int:
+        key = self._make_key(channel_id, user_id)
+        self._instance_map[key] = AudioVadV2(self._vad_config)
+        return 0
+        pass
+    def del_vad_instance(self, channel_id: str, user_id: str) -> None:
+        key = self._make_key(channel_id, user_id)
+        with self._lock:
+            self._instance_map.pop(key, None)
+    def get_vad_instance(self, channel_id: str, user_id: str) -> AudioVadV2:
+        key = self._make_key(channel_id, user_id)
+        with self._lock:
+            return self._instance_map.get(key, None)
+    def process(self, channel_id: str, user_id: str, frame: AudioFrame) -> tuple[int, bytearray]:
+        if self._is_init is False:
+            return -2, None
+        vad_instance = self.get_vad_instance(channel_id, user_id)
+        if vad_instance is not None:
+            return vad_instance.process(frame)
+        else:
+            #add new one
+            self._add_vad_instance(channel_id, user_id)
+            return -1, None
+        pass
+    def release(self) -> None:
+        print("____release vad manager: ", len(self._instance_map))
+        if self._is_init is False:
+            return
+        self._is_init = False
+        with self._lock:
+            self._instance_map.clear()
+        pass

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora/rtc/local_user.py

@@ -413,10 +413,10 @@ class LocalUser:
         ret = agora_local_user_set_playback_audio_frame_before_mixing_parameters(self.user_handle, channels, sample_rate_hz)
         return ret
 
-    def register_audio_frame_observer(self, observer: IAudioFrameObserver):
+    def register_audio_frame_observer(self, observer: IAudioFrameObserver,  enable_vad: int, vad_configure):
         if self.audio_frame_observer:
             self.unregister_audio_frame_observer()
-        self.audio_frame_observer = AudioFrameObserverInner(observer, self)
+        self.audio_frame_observer = AudioFrameObserverInner(observer, self, enable_vad, vad_configure)
         ret = agora_local_user_register_audio_frame_observer(self.user_handle, self.audio_frame_observer)
         return ret
 
@@ -424,6 +424,8 @@ class LocalUser:
         ret = 0
         if self.audio_frame_observer:
             ret = agora_local_user_unregister_audio_frame_observer(self.user_handle)
+            # clear observerInner related resoure
+            self.audio_frame_observer.clear()
         self.audio_frame_observer = None
         return ret
 

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora/rtc/utils/audio_consumer.py

@@ -69,7 +69,7 @@ class AudioConsumer:
     def consume(self):
         print("consume begin")
         if self._init == False:
-            return
+            return -1
         now = time.time()*1000
         elapsed_time = int(now - self._start_time)
         expected_total_packages = int(elapsed_time//10)
@@ -78,7 +78,7 @@ class AudioConsumer:
 
         if besent_packages > 18 and data_len //self._bytes_per_frame < 18: #for fist time, if data_len is not enough, just return and wait for next time
             #print("-----underflow data")
-            return
+            return -2
         if besent_packages > 18: #rest to start state, push 18 packs in Start_STATE
             self._reset()
             besent_packages = min(18, data_len//self._bytes_per_frame)
@@ -89,7 +89,7 @@ class AudioConsumer:
         act_besent_packages = (int)(min(besent_packages, data_len//self._bytes_per_frame))
         #print("consume 1:", act_besent_packages, data_len)
         if act_besent_packages < 1:
-            return
+            return 0
        
         #construct an audio frame to push
         #frame = PcmAudioFrame()
@@ -104,6 +104,7 @@ class AudioConsumer:
             self._consumed_packages += act_besent_packages
 
         self._pcm_sender.send_audio_pcm_data(self._frame)
+        return self._consumed_packages
         #print(f"act_besent_packages: {now},{now - self._start_time}, {besent_packages}, {act_besent_packages},{self._consumed_packages},{data_len}")
         pass
 

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora/rtc/voice_detection.py

@@ -115,7 +115,7 @@ class AudioVadV2():
         
         ratios = self._calculate_sliding_window_ratio(queue, self._trend_window)
         # 计算趋势
-        print(ratios)
+        #print(ratios)
         return 1 if ratios[1] > ratios[0] else 0
 
     #get silence count from deque: totalcount, silenct_count
@@ -156,7 +156,7 @@ class AudioVadV2():
                
                 #and clear pre &start
                 self._clear_queue(self._stop_queue)
-                print("start speaking:", len(self._stop_queue))
+                #print("start speaking:", len(self._stop_queue))
             
         return state, bytes
         
@@ -165,7 +165,7 @@ class AudioVadV2():
         #如果数据满，怎判断是否触发stop
         state = self._cur_state
         size, full = self._push_to_stop(data)
-        print(f"stop: {size}, {full}")
+        #print(f"stop: {size}, {full}")
 
         
         if full == True:

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6/agora_python_server_sdk.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: agora_python_server_sdk
-Version: 2.1.5
+Version: 2.1.6
 Summary: A Python SDK for Agora Server
 Home-page: https://github.com/AgoraIO-Extensions/Agora-Python-Server-SDK
 Classifier: Intended Audience :: Developers
@@ -29,7 +29,7 @@ Description-Content-Type: text/markdown
   - CentOS 7.0 and above
   
 - Supported Mac versions:
-  - MacOS 13 and above
+  - MacOS 13 and above(only for coding and testing)
 
 - Python version:
   - Python 3.10 and above
@@ -51,6 +51,28 @@ python agora_rtc/examples/example_audio_pcm_send.py --appId=xxx --channelId=xxx
 
 # Change log
 
+## 2024.12.09 Release 2.1.6
+- New Features:
+  -- Added AudioVadManager to manage VAD (Voice Activity Detection) instances.
+  -- Integrated VAD functionality into the SDK. Developers no longer need to worry about how to use VAD; they only need to focus on setting appropriate parameters. Reference: sample_audio_vad.py
+- Changes:
+  -- In register_audio_frame_observer, two new parameters have been added to set the VAD parameters. Reference: sample_audio_vad.py
+  -- In on_playback_audio_frame_before_mixing, two new return values have been added: vad_result_state and vad_result_bytearray.
+    state:
+    < 0: No internal automatic VAD applied
+    0: No speaking
+    1: Started speaking
+    2: Speaking
+    3: Stopped speaking
+    vad_result_bytearray: The result processed by VAD, returned when VAD is active.
+    If automatic VAD is enabled:
+    Developers should use vad_result_bytearray for subsequent business processing (e.g., sending to ASR/STT), rather than using the raw frame data.
+    Reference: sample_audio_vad.py
+- Optimizations:
+  -- Replaced the use of pacer with AudioConsumer for pushing PCM audio.
+- Updates:
+  -- Updated the samples related to Pacer and VAD.
+
 ## 2024.12.03 release Version 2.1.5
 - Modifications:
   - LocalUser/audioTrack:
@@ -138,3 +160,122 @@ Judge the value of state according to the returned state, and do corresponding p
   # Source code: audio_consumer.py
   # Sample code: example_audio_consumer.py
 ### How to release resources?
+## 如何释放资源？
+    localuser.unpublish_audio(audio_track)
+    localuser.unpublish_video(video_track)
+    audio_track.set_enabled(0)
+    video_track.set_enabled(0)
+
+    localuser.unregister_audio_frame_observer()
+    localuser.unregister_video_frame_observer()
+    localuser.unregister_local_user_observer()
+
+    connection.disconnect()
+    connection.unregister_observer()
+
+    localuser.release()
+    connection.release()
+
+    
+    audio_track.release()
+    video_track.release()
+    pcm_data_sender.release()
+    video_data_sender.release()
+    audio_consumer.release()
+
+    media_node_factory.release()
+    agora_service.release()
+    
+    #set to None
+    audio_track = None
+    video_track = None
+    audio_observer = None
+    video_observer = None
+    local_observer = None
+    localuser = None
+    connection = None
+    agora_service = None
+
+## Interrupt Handling in AI Scenarios
+# Definition of Interrupt
+In human-machine dialogue, an interrupt refers to the situation where a user suddenly interrupts the robot's response, requesting the robot to stop its current response immediately and shift to answer the user's new question. This behavior is called an interrupt.
+
+# Trigger Conditions for Interrupts
+Interrupts can be defined in different ways depending on the product. There are generally two modes:
+
+- Mode 1: Voice Activation Mode
+When it detects that the user is speaking, the interrupt strategy is triggered. For example, when the system recognizes speech, it triggers the interrupt strategy to stop the robot's response.
+
+- Mode 2: ASR Activation Mode
+When the system detects that the user is speaking and receives a result from ASR (Automatic Speech Recognition) or STT (Speech-to-Text), the interrupt strategy is triggered.
+
+# Advantages of Different Interrupt Strategies
+Voice Activation Interrupt
+
+Advantages:
+Reduces the user's wait time and the likelihood of interrupts, as the robot will stop its response immediately when the user starts speaking, eliminating the need for the user to wait for the robot to finish speaking.
+Disadvantages:
+Since this is voice-activated, it may be triggered by meaningless audio signals, depending on the accuracy of the VAD (Voice Activity Detection). For example, if someone is typing on the keyboard while the AI is speaking, it might trigger the interrupt incorrectly.
+ASR Activation Interrupt
+
+Advantages:
+Reduces the probability of unnecessary interrupts because the interrupt strategy is triggered only after ASR or STT has recognized the user’s speech.
+Disadvantages:
+Since this is ASR/STT-triggered, it requires converting the audio signal into text, which introduces a delay before the interrupt can be processed.
+- Recommended Mode
+If the VAD can filter out non-speech signals and only triggers when human speech is detected, the Voice Activation Mode is recommended. This mode is also suitable when the delay in processing the interrupt is not a major concern.
+
+If the interrupt delay is not sensitive, the ASR Activation Mode is recommended. This mode can filter out non-speech signals more effectively and reduce the probability of an unintended interrupt.
+
+How to Implement Interrupts? What Actions Are Required?
+In a human-machine dialogue system, conversations are typically structured in "rounds," where each round consists of a question from the user, followed by a response from the robot, and so on. For each round, we can assign a roundId, incrementing it with each new round. A round consists of the following stages:
+
+VAD (Voice Activity Detection):
+This marks the start of the dialogue, where the system detects the beginning and end of the user's speech. It then passes this information to the ASR for further processing.
+
+ASR (Automatic Speech Recognition):
+This phase involves recognizing the user's speech and converting it into text, which is then passed to the LLM (Large Language Model).
+
+LLM (Large Language Model):
+This is the generation phase, where the LLM processes the recognized user input and generates a response.
+
+TTS (Text-to-Speech):
+In this phase, the LLM’s response is converted into an audio format.
+
+RTC Streaming:
+The generated audio is streamed via RTC (Real-Time Communication) to be played back to the user.
+
+Therefore, an interrupt happens when, in the next round (roundId+1), either through Voice Activation (triggered by the VAD phase) or ASR Activation (triggered when ASR recognizes the user’s speech), the following actions must be performed:
+
+Stop the LLM Generation in the current round (roundId).
+Stop the TTS Synthesis in the current round (roundId).
+Stop the RTC Streaming in the current round (roundId).
+API Call References:
+Call: AudioConsumer.clear()
+Call: LocalAudioTrack.clear_sender_buffer()
+Business Layer: Clear any remaining TTS-related data (if applicable)
+
+
+## When to Pass LLM Results to TTS for Synthesis?
+LLM (Large Language Model) results are returned asynchronously and in a streaming manner. When should the results from the LLM be passed to TTS (Text-to-Speech) for synthesis?
+
+Two main factors need to be considered:
+
+Ensure that the TTS synthesized speech is unambiguous:
+The speech synthesized by TTS must be clear, complete, and continuous. For example, if the LLM returns the text: "中间的首都是北京吗？", and we pass it to TTS as:
+
+"中",
+"国首",
+"是北",
+"京吗？",
+This would result in ambiguous synthesis because there are no spaces between certain words (e.g., between "中" and "国", "首" and "是", and "京" and "吗"). Proper segmentation must be ensured to avoid such ambiguities.
+Minimize overall processing delay:
+If the LLM results are passed to TTS only after the entire response is generated, the speech synthesis will be unambiguous and continuous. However, this approach introduces significant delay, which negatively affects the user experience.
+
+Recommended Approach
+To achieve a balance between clarity and minimal delay, the following steps should be followed:
+
+Store the LLM results in a cache as they are received.
+Perform a reverse scan of the cached data to find the most recent punctuation mark.
+Truncate the data from the start to the most recent punctuation mark and pass it to TTS for synthesis.
+Remove the truncated data from the cache. The remaining data should be moved to the beginning of the cache and continue waiting for additional data from the LLM.

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/agora_python_server_sdk.egg-info/SOURCES.txt

@@ -10,6 +10,7 @@ agora/rtc/audio_encoded_frame_sender.py
 agora/rtc/audio_frame_observer.py
 agora/rtc/audio_pcm_data_sender.py
 agora/rtc/audio_sessionctrl.py
+agora/rtc/audio_vad_manager.py
 agora/rtc/local_audio_track.py
 agora/rtc/local_user.py
 agora/rtc/local_user_observer.py

{agora_python_server_sdk-2.1.5 → agora_python_server_sdk-2.1.6}/setup.py

@@ -45,7 +45,7 @@ class CustomInstallCommand(install):
 
 setup(
     name='agora_python_server_sdk',
-    version='2.1.5',
+    version='2.1.6',
     description='A Python SDK for Agora Server',
     long_description=open('README.md').read(),
     long_description_content_type='text/markdown',