deeptalk-asd 0.2.0__tar.gz

deeptalk_asd-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 huyyxy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

deeptalk_asd-0.2.0/PKG-INFO

@@ -0,0 +1,17 @@
+Metadata-Version: 2.4
+Name: deeptalk-asd
+Version: 0.2.0
+Summary: DeepTalk Active Speaker Detection
+Requires-Python: >=3.8
+License-File: LICENSE
+Requires-Dist: python-dotenv
+Requires-Dist: termcolor
+Requires-Dist: tornado
+Requires-Dist: python_speech_features
+Requires-Dist: numpy==1.26.4
+Requires-Dist: opencv-python
+Requires-Dist: scipy
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Dynamic: license-file
+Dynamic: requires-python

deeptalk_asd-0.2.0/README.md

@@ -0,0 +1,98 @@
+# DeepTalk-ASD
+
+[LR-ASD](https://github.com/Junhua-Liao/LR-ASD) is a SOTA Active Speaker Detection (ASD) model. While it offers exceptional performance, the official open-source project relies on GPUs and relatively older face detection models. This project aims to provide a production-ready, out-of-the-box ASD system.
+
+---
+
+DeepTalk-ASD is an efficient Active Speaker Detection (ASD) system. By fusing audio and video features, it determines in real-time which face in a video frame is speaking.
+
+## Key Features
+
+- **Multimodal Fusion**: Combines face detection, voice activity detection (VAD), and speaker identification models.
+- **Modular Design**: 
+    - **FaceDetector**: Detects and tracks faces (supports InspireFace).
+    - **TurnDetector**: Audio VAD detection (supports Silero VAD).
+    - **SpeakerDetector**: Audio-visual feature fusion and decision making (based on LR-ASD).
+- **High Performance**: Supports ONNX inference, suitable for real-time applications.
+- **Easy to Use**: Provides command-line demos supporting real-time camera input and video file processing.
+
+## System Architecture
+
+The system is orchestrated through three main sub-components:
+1. **FaceDetector**: Responsible for locating faces in each video frame.
+2. **TurnDetector**: Responsible for determining if the current audio stream contains speech.
+3. **SpeakerDetector**: The core decision layer that calculates speaking probability based on VAD results and face image sequences.
+
+## Quick Start
+
+### 1. Environment Setup
+
+Python 3.8 or higher is recommended.
+
+```bash
+# Clone the repository
+git clone <repository_url>
+cd DeepTalk-ASD
+
+# Install dependencies
+python3 -m pip install -r requirements.txt
+
+# Install the project in editable mode
+pip3 install -e .
+```
+
+### 2. Verify Installation
+
+Check the installation via the Python interactive environment:
+
+```python
+python3
+>>> import deeptalk_asd
+>>> print(deeptalk_asd.__version__)
+```
+
+### 3. Model Weights
+
+Ensure the `weights` directory contains the necessary model files. The project includes support for converting models to ONNX.
+- `Pikachu`: InspireFace related models.
+- `silero_vad.onnx`: Silero VAD model.
+- LR-ASD related models: `audio_frontend.onnx`, `visual_frontend.onnx`, `av_backend.onnx`.
+
+### 4. Running Demos
+
+#### Video File Processing Demo
+```bash
+python3 demo/video_asd_demo.py --input demo/demo.mp4 --display
+```
+
+#### Real-time Camera Demo
+```bash
+python3 demo/realtime_asd_demo.py
+```
+
+## Configuration
+
+The components can be flexibly configured using the factory method:
+
+```python
+from deeptalk_asd import ASDDetectorFactory
+
+config = {
+    "face_detector": {"type": "inspireface"},
+    "turn_detector": {"type": "silero-vad", "model_path": "weights/silero_vad.onnx"},
+    "speaker_detector": {"type": "LR-ASD-ONNX", "onnx_dir": "weights"}
+}
+
+factory = ASDDetectorFactory(**config)
+asd = factory.create()
+```
+
+## License
+
+The code in this project is licensed under the **MIT License**. However, please note that the integrated models and their related code are subject to their respective licenses:
+
+1. **InspireFace**: Core code is MIT, but the provided pre-trained models are typically restricted to **non-commercial research use**. For commercial use, please refer to [InsightFace](https://github.com/deepinsight/insightface) documentation.
+2. **Silero VAD**: Licensed under the **MIT License**. Please refer to [Silero VAD](https://github.com/snakers4/silero-vad).
+3. **LR-ASD**: Licensed under the **MIT License**. Please refer to [LR-ASD](https://github.com/Junhua-Liao/LR-ASD).
+
+While you may redistribute the code of this project under the MIT license, you must explicitly inform users in the documentation that: **When using specific pre-trained models (especially face detection models), users must comply with the non-commercial restrictions of the original authors.** If commercialization is required, users should replace them with commercially-friendly models or contact the original authors for authorization.

deeptalk_asd-0.2.0/deeptalk_asd.egg-info/PKG-INFO

@@ -0,0 +1,17 @@
+Metadata-Version: 2.4
+Name: deeptalk-asd
+Version: 0.2.0
+Summary: DeepTalk Active Speaker Detection
+Requires-Python: >=3.8
+License-File: LICENSE
+Requires-Dist: python-dotenv
+Requires-Dist: termcolor
+Requires-Dist: tornado
+Requires-Dist: python_speech_features
+Requires-Dist: numpy==1.26.4
+Requires-Dist: opencv-python
+Requires-Dist: scipy
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Dynamic: license-file
+Dynamic: requires-python

deeptalk_asd-0.2.0/deeptalk_asd.egg-info/SOURCES.txt

@@ -0,0 +1,36 @@
+LICENSE
+README.md
+pyproject.toml
+setup.py
+deeptalk_asd.egg-info/PKG-INFO
+deeptalk_asd.egg-info/SOURCES.txt
+deeptalk_asd.egg-info/dependency_links.txt
+deeptalk_asd.egg-info/requires.txt
+deeptalk_asd.egg-info/top_level.txt
+src/__init__.py
+src/asd.py
+src/asd_factory.py
+src/asd_interface.py
+src/audio_frame.py
+src/deeptalk_logger.py
+src/video_frame.py
+src/face_detector/__init__.py
+src/face_detector/face_compare_helper.py
+src/face_detector/face_info.py
+src/face_detector/factory.py
+src/face_detector/inspireface_detector.py
+src/face_detector/interface.py
+src/speaker_detector/__init__.py
+src/speaker_detector/factory.py
+src/speaker_detector/interface.py
+src/speaker_detector/lrasd_onnx.py
+src/turn_detector/__init__.py
+src/turn_detector/factory.py
+src/turn_detector/interface.py
+src/turn_detector/silero_vad_turn_detector.py
+src/turn_detector/utterance.py
+src/turn_detector/resampler/__init__.py
+src/turn_detector/resampler/audio_resampler.py
+src/turn_detector/resampler/audio_resampler_factory.py
+src/turn_detector/resampler/rosa_audio_resampler.py
+src/turn_detector/resampler/sox_audio_resampler.py

deeptalk_asd-0.2.0/deeptalk_asd.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

deeptalk_asd-0.2.0/deeptalk_asd.egg-info/requires.txt

@@ -0,0 +1,9 @@
+python-dotenv
+termcolor
+tornado
+python_speech_features
+numpy==1.26.4
+opencv-python
+scipy
+pandas
+tqdm

deeptalk_asd-0.2.0/deeptalk_asd.egg-info/top_level.txt

@@ -0,0 +1 @@
+deeptalk_asd

deeptalk_asd-0.2.0/pyproject.toml

@@ -0,0 +1,20 @@
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "deeptalk-asd"
+version = "0.2.0"
+description = "DeepTalk Active Speaker Detection"
+requires-python = ">=3.8"
+dependencies = [
+    "python-dotenv",
+    "termcolor",
+    "tornado",
+    "python_speech_features",
+    "numpy==1.26.4",
+    "opencv-python",
+    "scipy",
+    "pandas",
+    "tqdm",
+]

deeptalk_asd-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

deeptalk_asd-0.2.0/setup.py

@@ -0,0 +1,29 @@
+from setuptools import setup, find_packages
+
+sub_packages = find_packages(where="src")
+
+# Build package_dir mapping: each sub-package must be explicitly mapped
+# e.g. "deeptalk_asd.face_detector" -> "src/face_detector"
+package_dir = {"deeptalk_asd": "src"}
+for pkg in sub_packages:
+    package_dir[f"deeptalk_asd.{pkg}"] = f"src/{pkg.replace('.', '/')}"
+
+setup(
+    name="deeptalk_asd",
+    version="0.2.0",
+    description="DeepTalk Active Speaker Detection",
+    package_dir=package_dir,
+    packages=["deeptalk_asd"] + [f"deeptalk_asd.{pkg}" for pkg in sub_packages],
+    python_requires=">=3.8",
+    install_requires=[
+        "python-dotenv",
+        "termcolor",
+        "tornado",
+        "python_speech_features",
+        "numpy==1.26.4",
+        "opencv-python",
+        "scipy",
+        "pandas",
+        "tqdm",
+    ],
+)

deeptalk_asd-0.2.0/src/__init__.py

@@ -0,0 +1,26 @@
+"""
+DeepTalk-ASD: 活动说话者检测模块
+"""
+
+from .asd_factory import ASDDetectorFactory
+from .asd_interface import ASDInterface
+from .audio_frame import AudioFrame
+from .video_frame import VideoFrame, VideoBufferType, VideoRotation, VideoCodec, VideoStreamType
+from .face_detector.face_info import FaceProfile, FaceRectangle, HeadPose
+from .turn_detector.utterance import Utterance, TurnState
+
+__all__ = [
+    "ASDDetectorFactory",
+    "ASDInterface",
+    "AudioFrame",
+    "VideoFrame",
+    "VideoBufferType",
+    "VideoRotation",
+    "VideoCodec",
+    "VideoStreamType",
+    "FaceProfile",
+    "FaceRectangle",
+    "HeadPose",
+    "Utterance",
+    "TurnState",
+]

deeptalk_asd-0.2.0/src/asd.py

@@ -0,0 +1,133 @@
+"""
+文件名: asd.py
+描述:
+    ASDInterface 的实现，编排 FaceDetector、TurnDetector、SpeakerDetector 三个子组件
+    完成活动说话者检测的完整流程。
+"""
+import cv2
+import numpy as np
+from typing import List
+
+from .asd_interface import ASDInterface
+from .video_frame import VideoFrame
+from .audio_frame import AudioFrame
+from .face_detector.face_info import FaceProfile
+from .turn_detector.interface import TurnDetectorInterface
+from .turn_detector.utterance import Utterance, TurnState
+from .face_detector.interface import FaceDetectorInterface
+from .speaker_detector.interface import SpeakerDetectorInterface
+
+from .deeptalk_logger import DeepTalkLogger
+
+logger = DeepTalkLogger(__name__)
+
+
+class ASD(ASDInterface):
+    """活动说话者检测实现类
+
+    编排三个子组件：
+    - FaceDetector: 从视频帧中检测人脸
+    - TurnDetector: 从音频帧中检测语音轮次 (VAD)
+    - SpeakerDetector: 融合音视频特征，判定活跃说话者
+    """
+
+    def __init__(
+        self,
+        face_detector: FaceDetectorInterface,
+        turn_detector: TurnDetectorInterface,
+        speaker_detector: SpeakerDetectorInterface,
+        **kwargs,
+    ):
+        """
+        初始化活动说话者检测系统
+
+        参数:
+            face_detector: 人脸检测器实例
+            turn_detector: 语音轮次检测器实例
+            speaker_detector: 说话者检测器实例
+            kwargs: 可选参数，保留用于未来扩展
+        """
+        super().__init__(**kwargs)
+        self._face_detector = face_detector
+        self._turn_detector = turn_detector
+        self._speaker_detector = speaker_detector
+
+    def append_video(self, video_frame: VideoFrame, create_time: float = None) -> List[FaceProfile]:
+        """
+        添加视频帧，视频帧率缺省为25
+
+        流程:
+            1. 使用 FaceDetector 检测视频帧中的人脸
+            2. 将每个人脸的灰度图与 track_id 传给 SpeakerDetector
+
+        参数:
+            video_frame: 当前视频帧
+            create_time: 此块视频帧的创建时间
+        """
+        # 1. 人脸检测
+        face_profiles = self._face_detector.detect(video_frame)
+
+        # 2. 转换为 SpeakerDetector 需要的格式: [{'id': track_id, 'image': face_gray}, ...]
+        frame_faces = []
+        for profile in face_profiles:
+            face_image = profile.face_image
+            if face_image is not None:
+                # 转换为灰度图
+                if len(face_image.shape) == 3:
+                    face_gray = cv2.cvtColor(face_image, cv2.COLOR_BGR2GRAY)
+                else:
+                    face_gray = face_image
+                    face_gray = cv2.resize(face_gray, (224, 224))
+                    face_gray = face_gray[56:168, 56:168]  # 中心裁剪 112x112
+                frame_faces.append({
+                    'id': profile.track_id,
+                    'image': face_gray,
+                })
+
+        # 3. 传入 SpeakerDetector
+        if frame_faces:
+            self._speaker_detector.append_video(frame_faces, create_time)
+        return face_profiles
+
+    def append_audio(self, audio_frame: AudioFrame, create_time: float = None) -> Utterance:
+        """
+        添加音频块到处理队列
+
+        流程:
+            1. 将音频 PCM 数据传给 SpeakerDetector
+            2. 使用 TurnDetector 进行 VAD 检测
+
+        参数:
+            audio_frame: 当前音频帧
+            create_time: 此块音频帧的创建时间
+
+        返回:
+            轮次检测结果 (Utterance)
+        """
+        # 1. 将音频数据传入 SpeakerDetector
+        audio_data = bytes(audio_frame.data)
+        self._speaker_detector.append_audio(audio_data, create_time)
+
+        # 2. TurnDetector 进行 VAD 轮次检测
+        utterance = self._turn_detector.detect(audio_frame)
+        if utterance.turn_state in [TurnState.TURN_START, TurnState.TURN_CONFIRMED, TurnState.TURN_END, TurnState.TURN_REJECTED]:
+            logger.critical(f"Utterance: {utterance.turn_state.name}")
+        return utterance
+
+    def evaluate(self, start_time: float = None, end_time: float = None):
+        """
+        评估当前活动说话者
+
+        参数:
+            start_time: 评估起始时间（time.perf_counter 时间戳），可选
+            end_time: 评估结束时间（time.perf_counter 时间戳），可选
+
+        返回:
+            最新时间点的活动说话者的 track_id 和置信度得分
+            格式: {track_id: score, ...}
+        """
+        return self._speaker_detector.evaluate(start_time, end_time)
+
+    def reset(self):
+        """重置系统状态"""
+        self._speaker_detector.reset()

deeptalk_asd-0.2.0/src/asd_factory.py

@@ -0,0 +1,89 @@
+"""
+文件名: asd_factory.py
+描述:
+    ASD 检测器工厂，通过配置信息分别创建 FaceDetector、TurnDetector、SpeakerDetector，
+    然后组装为 ASD 实例。
+"""
+from .deeptalk_logger import DeepTalkLogger
+from .asd_interface import ASDInterface
+import traceback
+
+logger = DeepTalkLogger(__name__)
+
+
+class ASDDetectorFactory:
+    def __init__(self, **kwargs):
+        """
+        使用配置信息创建 ASD 实例。
+
+        参数:
+            kwargs: 包含三个子组件的配置信息:
+                face_detector: dict, 人脸检测器配置, 如 {"type": "inspireface", ...}
+                turn_detector: dict, 轮次检测器配置, 如 {"type": "silero-vad", "model_path": "...", ...}
+                speaker_detector: dict, 说话者检测器配置, 如 {"type": "LR-ASD-ONNX", "onnx_dir": "...", ...}
+        """
+        self.kwargs = kwargs
+
+    def create(self) -> ASDInterface:
+        """
+        根据配置创建 ASD 实例。
+
+        :return: ASDInterface 实例，创建失败时返回 None
+        """
+        try:
+            # 1. 创建 FaceDetector
+            face_config = dict(self.kwargs.get('face_detector', {}))
+            face_type = face_config.pop('type', None)
+            if face_type is None:
+                logger.error("face_detector 配置缺少 'type' 字段")
+                return None
+
+            from .face_detector.factory import FaceDetectorFactory
+            face_detector = FaceDetectorFactory(face_type, **face_config).face_detector()
+            if face_detector is None:
+                logger.error(f"创建 FaceDetector(type={face_type}) 失败")
+                return None
+            logger.info(f"FaceDetector 创建成功: type={face_type}")
+
+            # 2. 创建 TurnDetector
+            turn_config = dict(self.kwargs.get('turn_detector', {}))
+            turn_type = turn_config.pop('type', None)
+            if turn_type is None:
+                logger.error("turn_detector 配置缺少 'type' 字段")
+                return None
+
+            from .turn_detector.factory import TurnDetectorFactory
+            turn_detector = TurnDetectorFactory(turn_type, **turn_config).turn_detector()
+            if turn_detector is None:
+                logger.error(f"创建 TurnDetector(type={turn_type}) 失败")
+                return None
+            logger.info(f"TurnDetector 创建成功: type={turn_type}")
+
+            # 3. 创建 SpeakerDetector
+            speaker_config = dict(self.kwargs.get('speaker_detector', {}))
+            speaker_type = speaker_config.pop('type', None)
+            if speaker_type is None:
+                logger.error("speaker_detector 配置缺少 'type' 字段")
+                return None
+
+            from .speaker_detector.factory import SpeakerDetectorFactory
+            speaker_detector = SpeakerDetectorFactory(speaker_type, **speaker_config).speaker_detector()
+            if speaker_detector is None:
+                logger.error(f"创建 SpeakerDetector(type={speaker_type}) 失败")
+                return None
+            logger.info(f"SpeakerDetector 创建成功: type={speaker_type}")
+
+            # 4. 组装 ASD 实例
+            from .asd import ASD
+            asd = ASD(
+                face_detector=face_detector,
+                turn_detector=turn_detector,
+                speaker_detector=speaker_detector,
+            )
+            logger.info("ASD 实例创建成功")
+            return asd
+
+        except Exception as e:
+            logger.error(f"创建 ASD 实例失败: {e}")
+            traceback.print_exc()
+            return None

deeptalk_asd-0.2.0/src/asd_interface.py

@@ -0,0 +1,57 @@
+from .video_frame import VideoFrame
+from .audio_frame import AudioFrame
+from .turn_detector.utterance import Utterance
+from .face_detector.interface import FaceProfile
+from typing import List
+
+
+class ASDInterface:
+    """活动说话者检测核心类"""
+    def __init__(self, **kwargs):
+        """
+        初始化活动说话者检测系统
+        
+        参数:
+            kwargs: 可选参数，当前未使用，保留用于未来扩展。
+        """
+        pass
+    
+    def append_video(self, video_frame: VideoFrame, create_time: float = None) -> List[FaceProfile]:
+        """
+        添加视频帧，视频帧率缺省为25
+
+        参数:
+            video_frame: 当前视频帧
+            create_time: 此块视频帧的创建时间，调用方一般使用time.perf_counter()获得，用于和音频块时间对齐
+        """
+        pass
+    
+    def append_audio(self, audio_frame: AudioFrame, create_time: float = None) -> Utterance:
+        """
+        添加音频块到处理队列
+        
+        参数:
+            audio_frame: 当前音频帧
+            create_time: 此块音频帧的创建时间，调用方一般使用time.perf_counter()获得，用于和视频帧时间对齐
+        """
+        pass
+    
+    def evaluate(self, start_time: float = None, end_time: float = None):
+        """
+        评估当前活动说话者，一般是前置VAD检测发现结束说话时调用
+        
+        参数:
+            start_time: 评估起始时间（time.perf_counter 时间戳），可选
+            end_time: 评估结束时间（time.perf_counter 时间戳），可选
+            若均提供，则只对该时间范围内的音视频做推理；
+            若时间范围超过缓冲区长度，则回退为使用整个缓冲区。
+            若不传，则使用整个缓冲区。
+        
+        返回:
+            最新时间点的活动说话者的tracker_id和置信度得分
+        """
+        pass
+    
+    def reset(self):
+        """重置系统状态"""
+        pass