dv-pipecat-ai 0.0.85.dev856__tar.gz → 0.0.85.dev858__tar.gz

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/CLAUDE.md

@@ -110,6 +110,69 @@ pip install "pipecat-ai[openai,deepgram,cartesia]"
 
 Pipecat is a **pipeline-based real-time AI conversation framework** for building voice-enabled, multimodal AI applications. Built around these key architectural concepts:
 
+### Flow Cheat‑Sheet
+
+```
+(Input Transport) ──DOWNSTREAM──► [Audio/VAD/Filters] ──► [STT Service] ──►
+    |                                 |                       |
+    |                                 |                       └─ emits TranscriptionFrame / InterimTranscriptionFrame (DataFrame)
+    |                                 └─ may send STTMuteFrame (SystemFrame) UPSTREAM
+    | 
+    └─ sends StartFrame (SystemFrame) at boot
+
+[Transcript Processor + Aggregators]
+    └─ combine TranscriptionFrame → produce LLMTextFrame (DataFrame)
+
+[LLM Service]
+    ├─ streams LLMTextFrame responses (DataFrame)
+    └─ may emit FunctionCallFrame / FunctionCallResultFrame
+
+[TTS Service]
+    └─ converts LLMTextFrame → TTSAudioRawFrame (DataFrame)
+
+(Output Transport)
+    └─ serializes TTSAudioRawFrame → client protocol (WebRTC/Telephony/etc.)
+
+Control/interrupt frames (UserStartedSpeakingFrame, SpeechControlParamsFrame, CancelFrame…)
+can travel UPSTREAM at any point so earlier processors/transports react immediately.
+```
+
+**Key points:**
+- `FrameDirection.DOWNSTREAM` = toward later processors/services; `FrameDirection.UPSTREAM` = back toward source (e.g., mute STT, cancel playback).
+- `SystemFrame` (StartFrame, STTMuteFrame, SpeechControlParamsFrame, ErrorFrame…) always gets priority in the processor queue so control signals are never starved.
+- `DataFrame` carries payloads (audio/text/image): `InputAudioRawFrame`, `TTSAudioRawFrame`, `TranscriptionFrame`, `LLMTextFrame`, `FunctionCallFrame`, etc.
+- `ControlFrame` manages flow (`EndFrame`, `UserStartedSpeakingFrame`, `InterruptionFrame`).
+
+### Frame Life Cycle (detailed)
+
+1. **Ingress / Transports**
+   - `BaseInputTransport` (FastAPI WebSocket, Daily WebRTC, Twilio/Plivo serializers) receives media, emits `StartFrame`, then chunks of `InputAudioRawFrame` downstream.
+   - Transports also consume upstream frames (mute/unmute, cancel) and translate them into protocol-specific control messages.
+
+2. **Audio Front-End**
+   - Mixers, VAD analyzers (`SileroVADAnalyzer`), and noise filters act on audio frames before ASR.
+   - VAD produces `SpeechControlParamsFrame` to tune STT sensitivity; these propagate upstream/downstream as needed.
+
+3. **STT Stage**
+   - `STTService.process_frame` buffers audio and streams `InterimTranscriptionFrame` and final `TranscriptionFrame` downstream.
+   - When speech begins/ends, STT emits `UserStartedSpeakingFrame`/`UserStoppedSpeakingFrame` upstream so transports can pause/resume playback or update UI.
+
+4. **Context / Aggregation**
+   - `TranscriptProcessor`, `OpenAILLMContext`, and flow-specific processors collect transcription frames, manage conversation state, and finally emit `LLMTextFrame` requests.
+
+5. **LLM Stage**
+   - `LLMService` (OpenAI/Azure/Anthropic/etc.) streams `LLMTextFrame` responses, optionally `FunctionCallFrame` + `FunctionCallResultFrame` for tool calls.
+   - `FrameProcessorMetrics` (observer) listens to this flow and emits `MetricsFrame` (TTFB, TTFT, token counts) that downstream observers (e.g., `CallMetricsCollector`) consume.
+
+6. **TTS + Output**
+   - `TTSService` converts `LLMTextFrame` to `TTSAudioRawFrame`.
+   - `BaseOutputTransport` serializes audio into the channel’s format (raw PCM for WebRTC, base64 for telephony, etc.), until it receives `EndFrame`/`CancelFrame`.
+
+### Diagnostics & Metrics
+- `CallMetricsCollector` (observer) watches every `MetricsFrame`, tallying TTFT/TTFB per service, first TTS time, etc.
+- `ServiceConnectionMonitor` wraps service `_connect()`/`_connect_websocket()` to capture real connection latencies and retry counts per call. Its data feeds the `service_connections` summary stored alongside transcripts.
+- Both collector and monitor are stored per call id so concurrent calls never mix metrics.
+
 ### Framework Philosophy
 - **Real-time Processing**: Conversational interactions without noticeable delays
 - **Multimodal Support**: Combines audio, video, images, and text in interactions
@@ -371,4 +434,4 @@ def process_audio_bad(audio_data):
 - Pipeline lifecycle: StartFrame → processing → EndFrame/StopFrame
 - **Telephony**: WebSocket connections require proper call_id tracking and cleanup
 - **Async**: Never block the event loop - use async/await for all I/O operations
-- **Latency**: Voice AI requires sub-second response times - optimize every pipeline stage
+- **Latency**: Voice AI requires sub-second response times - optimize every pipeline stage

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dv-pipecat-ai
-Version: 0.0.85.dev856
+Version: 0.0.85.dev858
 Summary: An open source framework for voice (and multimodal) assistants
 License-Expression: BSD-2-Clause
 Project-URL: Source, https://github.com/pipecat-ai/pipecat

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/dv_pipecat_ai.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dv-pipecat-ai
-Version: 0.0.85.dev856
+Version: 0.0.85.dev858
 Summary: An open source framework for voice (and multimodal) assistants
 License-Expression: BSD-2-Clause
 Project-URL: Source, https://github.com/pipecat-ai/pipecat

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/dv_pipecat_ai.egg-info/SOURCES.txt

@@ -165,6 +165,7 @@ src/pipecat/frames/frames.proto
 src/pipecat/frames/frames.py
 src/pipecat/frames/protobufs/frames_pb2.py
 src/pipecat/metrics/__init__.py
+src/pipecat/metrics/connection_metrics.py
 src/pipecat/metrics/metrics.py
 src/pipecat/observers/__init__.py
 src/pipecat/observers/base_observer.py

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/audio/turn/smart_turn/local_smart_turn_v3.py

@@ -35,12 +35,15 @@ class LocalSmartTurnAnalyzerV3(BaseSmartTurn):
     enabling offline operation without network dependencies.
     """
 
-    def __init__(self, *, smart_turn_model_path: Optional[str] = None, **kwargs):
+    def __init__(
+        self, *, smart_turn_model_path: Optional[str] = None, cpu_count: int = 1, **kwargs
+    ):
         """Initialize the local ONNX smart-turn-v3 analyzer.
 
         Args:
             smart_turn_model_path: Path to the ONNX model file. If this is not
                 set, the bundled smart-turn-v3.0 model will be used.
+            cpu_count: The number of CPUs to use for inference. Defaults to 1.
             **kwargs: Additional arguments passed to BaseSmartTurn.
         """
         super().__init__(**kwargs)
@@ -70,6 +73,7 @@ class LocalSmartTurnAnalyzerV3(BaseSmartTurn):
         so = ort.SessionOptions()
         so.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
         so.inter_op_num_threads = 1
+        so.intra_op_num_threads = cpu_count
         so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
 
         self._feature_extractor = WhisperFeatureExtractor(chunk_length=8)

dv_pipecat_ai-0.0.85.dev858/src/pipecat/metrics/connection_metrics.py

@@ -0,0 +1,45 @@
+"""Connection metrics data models for Pipecat framework.
+
+This module extends the core metrics system with connection-specific metrics
+including connection establishment times, retry attempts, and network latencies.
+"""
+
+from typing import Optional
+
+from pydantic import BaseModel
+
+from pipecat.metrics.metrics import MetricsData
+
+
+class ConnectionMetricsData(MetricsData):
+    """Unified connection and reconnection metrics data.
+
+    Handles both initial connection establishment and reconnection scenarios.
+    For initial connections, use connect_time, success, connection_attempts.
+    For reconnections, use reconnect_count, downtime, reconnect_success, reason.
+
+    Parameters:
+        connect_time: Time taken to establish connection in seconds.
+        success: Whether the connection attempt was successful.
+        connection_attempts: Number of connection attempts made.
+        error_message: Error message if connection failed.
+        connection_type: Type of connection (websocket, http, etc.).
+        reconnect_count: Number of reconnection attempts (for reconnection scenarios).
+        downtime: Time connection was down in seconds (for reconnection scenarios).
+        reconnect_success: Whether reconnection was successful (for reconnection scenarios).
+        reason: Reason for reconnection (for reconnection scenarios).
+    """
+
+    connect_time: Optional[float] = None
+    success: bool = True
+    connection_attempts: int = 1
+    error_message: Optional[str] = None
+    connection_type: Optional[str] = None
+    
+    # Reconnection-specific fields
+    reconnect_count: Optional[int] = None
+    downtime: Optional[float] = None
+    reconnect_success: Optional[bool] = None
+    reason: Optional[str] = None
+
+

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/processors/frame_processor.py

@@ -436,10 +436,53 @@ class FrameProcessor(BaseObject):
             if frame:
                 await self.push_frame(frame)
 
+    async def start_connection_metrics(self):
+        """Start connection establishment metrics collection."""
+        if self.can_generate_metrics() and self.metrics_enabled:
+            await self._metrics.start_connection_metrics()
+
+    async def stop_connection_metrics(
+        self, 
+        success: bool = True, 
+        error: str = None,
+        connection_type: str = None
+    ):
+        """Stop connection metrics collection and emit metrics frame.
+        
+        Args:
+            success: Whether the connection was successful.
+            error: Error message if connection failed.
+            connection_type: Type of connection (websocket, http, etc.).
+        """
+        if self.can_generate_metrics() and self.metrics_enabled:
+            frame = await self._metrics.stop_connection_metrics(success, error, connection_type)
+            if frame:
+                await self.push_frame(frame)
+
+
+    async def start_reconnection_metrics(self):
+        """Start reconnection metrics collection."""
+        if self.can_generate_metrics() and self.metrics_enabled:
+            await self._metrics.start_reconnection_metrics()
+
+    async def stop_reconnection_metrics(self, success: bool = True, reason: str = None):
+        """Stop reconnection metrics collection and emit metrics frame.
+        
+        Args:
+            success: Whether the reconnection was successful.
+            reason: Reason for reconnection.
+        """
+        if self.can_generate_metrics() and self.metrics_enabled:
+            frame = await self._metrics.stop_reconnection_metrics(success, reason)
+            if frame:
+                await self.push_frame(frame)
+
+
     async def stop_all_metrics(self):
         """Stop all active metrics collection."""
         await self.stop_ttfb_metrics()
         await self.stop_processing_metrics()
+        await self.stop_connection_metrics()
 
     def create_task(self, coroutine: Coroutine, name: Optional[str] = None) -> asyncio.Task:
         """Create a new task managed by this processor.
@@ -811,7 +854,7 @@ class FrameProcessor(BaseObject):
         Returns:
             True if the processor has been started.
         """
-        if not self.__started:
+        if not self.__started and not isinstance(frame, SystemFrame):
             self.logger.error(f"{self} Trying to process {frame} but StartFrame not received yet")
         return self.__started
 

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/processors/metrics/frame_processor_metrics.py

@@ -20,6 +20,9 @@ from pipecat.metrics.metrics import (
     TTFBMetricsData,
     TTSUsageMetricsData,
 )
+from pipecat.metrics.connection_metrics import (
+    ConnectionMetricsData,
+)
 from pipecat.utils.asyncio.task_manager import BaseTaskManager
 from pipecat.utils.base_object import BaseObject
 
@@ -46,6 +49,13 @@ class FrameProcessorMetrics(BaseObject):
         self._last_ttfb_time = 0
         self._should_report_ttfb = True
         self._logger = logger
+        
+        # Connection metrics state
+        self._start_connection_time = 0
+        self._connection_attempts = 0
+        self._last_connection_error = None
+        self._reconnection_start_time = 0
+        self._reconnect_count = 0
 
     async def setup(self, task_manager: BaseTaskManager):
         """Set up the metrics collector with a task manager.
@@ -195,3 +205,101 @@ class FrameProcessorMetrics(BaseObject):
         )
         self._logger.debug(f"{self._processor_name()} usage characters: {characters.value}")
         return MetricsFrame(data=[characters])
+
+    async def start_connection_metrics(self):
+        """Start measuring connection establishment time."""
+        self._start_connection_time = time.time()
+        self._connection_attempts += 1
+        self._last_connection_error = None
+
+    async def stop_connection_metrics(
+        self, 
+        success: bool = True, 
+        error: str = None,
+        connection_type: str = None
+    ):
+        """Stop connection measurement and generate metrics frame.
+
+        Args:
+            success: Whether the connection was successful.
+            error: Error message if connection failed.
+            connection_type: Type of connection (websocket, http, etc.).
+
+        Returns:
+            MetricsFrame containing connection data, or None if not measuring.
+        """
+        if self._start_connection_time == 0:
+            return None
+
+        connect_time = time.time() - self._start_connection_time
+        
+        if not success:
+            self._last_connection_error = error
+        
+        logstr = f"{self._processor_name()} connection "
+        logstr += "successful" if success else f"failed: {error}"
+        logstr += f" (attempt #{self._connection_attempts}, {connect_time:.3f}s)"
+        
+        if success:
+            self._logger.debug(logstr)
+        else:
+            self._logger.warning(logstr)
+        
+        connection_data = ConnectionMetricsData(
+            processor=self._processor_name(),
+            model=self._model_name(),
+            connect_time=round(connect_time, 3),
+            success=success,
+            connection_attempts=self._connection_attempts,
+            error_message=error,
+            connection_type=connection_type
+        )
+        
+        self._start_connection_time = 0
+        return MetricsFrame(data=[connection_data])
+
+
+    async def start_reconnection_metrics(self):
+        """Start measuring reconnection downtime."""
+        self._reconnection_start_time = time.time()
+        self._reconnect_count += 1
+
+    async def stop_reconnection_metrics(
+        self, 
+        success: bool = True, 
+        reason: str = None
+    ):
+        """Stop reconnection measurement and generate metrics frame.
+
+        Args:
+            success: Whether the reconnection was successful.
+            reason: Reason for reconnection.
+
+        Returns:
+            MetricsFrame containing reconnection data, or None if not measuring.
+        """
+        if self._reconnection_start_time == 0:
+            return None
+
+        downtime = time.time() - self._reconnection_start_time
+        
+        logstr = f"{self._processor_name()} reconnection #{self._reconnect_count} "
+        logstr += "successful" if success else "failed"
+        logstr += f" (downtime: {downtime:.3f}s)"
+        if reason:
+            logstr += f" - {reason}"
+        
+        self._logger.debug(logstr)
+        
+        reconnection_data = ConnectionMetricsData(
+            processor=self._processor_name(),
+            model=self._model_name(),
+            reconnect_count=self._reconnect_count,
+            downtime=round(downtime, 3),
+            reconnect_success=success,
+            reason=reason
+        )
+        
+        self._reconnection_start_time = 0
+        return MetricsFrame(data=[reconnection_data])
+

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/services/deepgram/stt.py

@@ -388,6 +388,7 @@ class DeepgramSTTService(STTService):
 
     async def _connect(self):
         self.logger.debug("Attempting to connect to Deepgram...")
+        await self.start_connection_metrics()
 
         loop = asyncio.get_running_loop()
         for attempt in range(self._max_connect_retries):
@@ -489,6 +490,8 @@ class DeepgramSTTService(STTService):
                             elapsed_ms,
                             diagnostics,
                         )
+                        await self.stop_connection_metrics(success=True, connection_type="websocket")
+                        await self.stop_reconnection_metrics(success=True, reason="successful_reconnection")
                         return  # Exit the method on success
 
                 self.logger.warning(
@@ -524,6 +527,12 @@ class DeepgramSTTService(STTService):
         error_msg = (
             f"{self}: unable to connect to Deepgram after {self._max_connect_retries} attempts."
         )
+        await self.stop_connection_metrics(
+            success=False, 
+            error=f"Failed after {self._max_connect_retries} attempts", 
+            connection_type="websocket"
+        )
+        await self.stop_reconnection_metrics(success=False, reason="max_retries_exceeded")
         self.logger.error(error_msg)
         await self.push_error(ErrorFrame(error_msg, fatal=True))
 
@@ -589,6 +598,7 @@ class DeepgramSTTService(STTService):
         # NOTE(aleix): we don't disconnect (i.e. call finish on the connection)
         # because this triggers more errors internally in the Deepgram SDK. So,
         # we just forget about the previous connection and create a new one.
+        await self.start_reconnection_metrics()
         await self._connect()
 
     async def _on_speech_started(self, *args, **kwargs):

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/services/elevenlabs/tts.py

@@ -519,6 +519,7 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
                 return
 
             self.logger.debug("Connecting to ElevenLabs")
+            await self.start_connection_metrics()
 
             voice_id = self._voice_id
             model = self.model_name
@@ -551,14 +552,19 @@ class ElevenLabsTTSService(AudioContextWordTTSService):
                 additional_headers={"xi-api-key": self._api_key},
             )
 
+            await self.stop_connection_metrics(success=True, connection_type="websocket")
+            await self.stop_reconnection_metrics(success=True, reason="successful_reconnection")
             await self._call_event_handler("on_connected")
         except Exception as e:
             self.logger.error(f"{self} initialization error: {e}")
+            await self.stop_connection_metrics(success=False, error=str(e), connection_type="websocket")
+            await self.stop_reconnection_metrics(success=False, reason="connection_failed")
             self._websocket = None
             await self._call_event_handler("on_connection_error", f"{e}")
 
     async def _disconnect_websocket(self):
         try:
+            await self.start_reconnection_metrics()
             await self.stop_all_metrics()
 
             if self._websocket:

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/services/google/llm.py

@@ -760,12 +760,19 @@ class GoogleLLMService(LLMService):
 
         generation_config = GenerateContentConfig(system_instruction=system)
 
-        # Use the new google-genai client's async method
-        response = await self._client.aio.models.generate_content(
-            model=self._model_name,
-            contents=messages,
-            config=generation_config,
-        )
+        await self.start_connection_metrics()
+        
+        try:
+            # Use the new google-genai client's async method
+            response = await self._client.aio.models.generate_content(
+                model=self._model_name,
+                contents=messages,
+                config=generation_config,
+            )
+            await self.stop_connection_metrics(success=True, connection_type="grpc")
+        except Exception as e:
+            await self.stop_connection_metrics(success=False, error=str(e), connection_type="grpc")
+            raise
 
         # Extract text from response
         if response.candidates and response.candidates[0].content:
@@ -849,11 +856,19 @@ class GoogleLLMService(LLMService):
         )
 
         await self.start_ttfb_metrics()
-        return await self._client.aio.models.generate_content_stream(
-            model=self._model_name,
-            contents=messages,
-            config=generation_config,
-        )
+        await self.start_connection_metrics()
+        
+        try:
+            result = await self._client.aio.models.generate_content_stream(
+                model=self._model_name,
+                contents=messages,
+                config=generation_config,
+            )
+            await self.stop_connection_metrics(success=True, connection_type="grpc")
+            return result
+        except Exception as e:
+            await self.stop_connection_metrics(success=False, error=str(e), connection_type="grpc")
+            raise
 
     async def _stream_content_specific_context(
         self, context: OpenAILLMContext

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/services/openai/base_llm.py

@@ -205,20 +205,29 @@ class BaseOpenAILLMService(LLMService):
         """
         params = self.build_chat_completion_params(params_from_context)
 
-        if self._retry_on_timeout:
-            try:
-                chunks = await asyncio.wait_for(
-                    self._client.chat.completions.create(**params), timeout=self._retry_timeout_secs
-                )
-                return chunks
-            except (APITimeoutError, asyncio.TimeoutError):
-                # Retry, this time without a timeout so we get a response
-                logger.debug(f"{self}: Retrying chat completion due to timeout")
+        await self.start_connection_metrics()
+        
+        try:
+            if self._retry_on_timeout:
+                try:
+                    chunks = await asyncio.wait_for(
+                        self._client.chat.completions.create(**params), timeout=self._retry_timeout_secs
+                    )
+                    await self.stop_connection_metrics(success=True, connection_type="http")
+                    return chunks
+                except (APITimeoutError, asyncio.TimeoutError):
+                    # Retry, this time without a timeout so we get a response
+                    logger.debug(f"{self}: Retrying chat completion due to timeout")
+                    chunks = await self._client.chat.completions.create(**params)
+                    await self.stop_connection_metrics(success=True, connection_type="http")
+                    return chunks
+            else:
                 chunks = await self._client.chat.completions.create(**params)
+                await self.stop_connection_metrics(success=True, connection_type="http")
                 return chunks
-        else:
-            chunks = await self._client.chat.completions.create(**params)
-            return chunks
+        except Exception as e:
+            await self.stop_connection_metrics(success=False, error=str(e), connection_type="http")
+            raise
 
     def build_chat_completion_params(self, params_from_context: OpenAILLMInvocationParams) -> dict:
         """Build parameters for chat completion request.

{dv_pipecat_ai-0.0.85.dev856 → dv_pipecat_ai-0.0.85.dev858}/src/pipecat/services/vistaar/llm.py

@@ -344,9 +344,12 @@ class VistaarLLMService(LLMService):
             logger.error(f"Failed to generate JWT token: {e}")
             raise
 
+        await self.start_connection_metrics()
+        
         try:
             # Use httpx to handle SSE streaming
             async with self._client.stream("GET", url, headers=headers) as response:
+                await self.stop_connection_metrics(success=True, connection_type="http")
                 self._current_response = response  # Store for potential cancellation
                 response.raise_for_status()
 
@@ -364,14 +367,17 @@ class VistaarLLMService(LLMService):
                     yield line
 
         except httpx.HTTPStatusError as e:
+            await self.stop_connection_metrics(success=False, error=f"HTTP {e.response.status_code}", connection_type="http")
             logger.error(
                 f"Vistaar HTTP error - Status: {e.response.status_code}, URL: {url}, Response: {e.response.text if hasattr(e.response, 'text') else 'N/A'}"
             )
             raise
         except httpx.TimeoutException as e:
+            await self.stop_connection_metrics(success=False, error="Timeout", connection_type="http")
             logger.error(f"Vistaar timeout error - URL: {url}, Timeout: {self._timeout}s")
             raise
         except Exception as e:
+            await self.stop_connection_metrics(success=False, error=str(e), connection_type="http")
             logger.error(
                 f"Vistaar unexpected error - Type: {type(e).__name__}, Message: {str(e)}, URL: {url}"
             )