sinapsis-speech 0.3.5__tar.gz → 0.4.1__tar.gz

{sinapsis_speech-0.3.5/packages/sinapsis_speech.egg-info → sinapsis_speech-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sinapsis-speech
-Version: 0.3.5
+Version: 0.4.1
 Summary: Generate speech using various libraries.
 Author-email: SinapsisAI <dev@sinapsis.tech>
 Project-URL: Homepage, https://sinapsis.tech
@@ -18,6 +18,8 @@ Requires-Dist: sinapsis-f5-tts[all]; extra == "all"
 Requires-Dist: sinapsis-kokoro[all]; extra == "all"
 Requires-Dist: sinapsis-speech[gradio-app]; extra == "all"
 Requires-Dist: sinapsis-zonos[all]; extra == "all"
+Requires-Dist: sinapsis-parakeet-tdt[all]; extra == "all"
+Requires-Dist: sinapsis-orpheus-cpp[all]; extra == "all"
 Provides-Extra: gradio-app
 Requires-Dist: sinapsis[webapp]>=0.2.3; extra == "gradio-app"
 Dynamic: license-file
@@ -33,7 +35,7 @@ Sinapsis Speech
 <br>
 </h1>
 
-<h4 align="center"> Templates for a wide range of voice generation tasks.</h4>
+<h4 align="center"> A monorepo housing multiple packages and templates for versatile voice generation, text-to-speech, speech-to-text, and beyond.</h4>
 
 <p align="center">
 <a href="#installation">🐍 Installation</a> •
@@ -55,8 +57,10 @@ This repo includes packages for performing speech synthesis using different tool
 
 * <code>sinapsis-elevenlabs</code>
 * <code>sinapsis-f5-tts</code>
-* * <code>sinapsis-kokoro</code>
+* <code>sinapsis-kokoro</code>
 * <code>sinapsis-zonos</code>
+* <code>sinapsis-orpheus-cpp</code>
+* <code>sinapsis-parakeet</code>
 
 Install using your preferred package manager. We strongly recommend using <code>uv</code>. To install <code>uv</code>, refer to the [official documentation](https://docs.astral.sh/uv/getting-started/installation/#installation-methods).
 
@@ -104,10 +108,14 @@ This repository is organized into modular packages, each designed for integratio
 <details>
 <summary id="elevenlabs"><strong><span style="font-size: 1.4em;"> Sinapsis ElevenLabs </span></strong></summary>
 
-This package offers a suite of templates and utilities designed for effortless integrating, configuration, and execution of **text-to-speech (TTS)** and **voice generation** functionalities powered by [ElevenLabs](https://elevenlabs.io/).
+This package offers a suite of templates and utilities designed for effortless integrating, configuration, and execution of **text-to-speech (TTS)**, **speech-to-speech (STS)**, **voice cloning**, and **voice generation** functionalities powered by [ElevenLabs](https://elevenlabs.io/).
+
+- **ElevenLabsSTS**: Template for transforming a voice into a different character or style using the ElevenLabs Speech-to-Speech API.
 
 - **ElevenLabsTTS**: Template for converting text into speech using ElevenLabs' voice models.
 
+- **ElevenLabsVoiceClone**: Template for creating a synthetic copy of an existing voice using the ElevenLabs API.
+
 - **ElevenLabsVoiceGeneration**: Template for generating custom synthetic voices based on user-provided descriptions.
 
 For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_elevenlabs/README.md).
@@ -144,6 +152,30 @@ For specific instructions and further details, see the [README.md](https://githu
 
 </details>
 
+
+<details>
+<summary id="orpheus-cpp"><strong><span style="font-size: 1.4em;"> Sinapsis Orppheus-CPP</span></strong></summary>
+
+This package provides a template for seamlessly integrating, configuring, and running **text-to-speech (TTS)** functionalities powered by [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS).
+
+- **OrpheusTTS**: Converts text to speech using the Orpheus TTS model with advanced neural voice synthesis. The template processes text packets from the input container, generates corresponding audio using Orpheus TTS, and adds the resulting audio packets to the container. Features graceful error handling for out-of-memory conditions
+
+For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_orpheus_cpp/README.md).
+
+</details>
+
+<details>
+<summary id="parakeet-tdt"><strong><span style="font-size: 1.4em;"> Sinapsis Parakeet-TDT</span></strong></summary>
+
+This package provides a template for seamlessly integrating, configuring, and running **speech-to-text (STT)** functionalities powered by [NVIDIA's Parakeet TDT model](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2).
+
+
+- **ParakeetTDTInference**: Converts speech to text using NVIDIA's Parakeet TDT 0.6B model. This template processes audio packets from the input container or specified file paths, performs transcription with optional timestamp prediction, and adds the resulting text packets to the container.
+
+For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_parakeet_tdt/README.md).
+
+</details>
+
 <h2 id="webapp">🌐 Webapps</h2>
 The webapps included in this project showcase the modularity of the templates, in this case for speech generation tasks.
 
@@ -182,7 +214,6 @@ cd sinapsis-speech
 docker compose -f docker/compose.yaml build
 ```
 
-
 2. **Start the app container**:
 
 - For ElevenLabs:
@@ -205,6 +236,16 @@ docker compose -f docker/compose_apps.yaml up -d sinapsis-kokoro
 docker compose -f docker/compose_apps.yaml up -d sinapsis-zonos
 ```
 
+- For Orpheus-CPP:
+```bash
+docker compose -f docker/compose_apps.yaml up -d sinapsis-orpheus-tts
+```
+
+- For Parakeet:
+```bash
+docker compose -f docker/compose_apps.yaml up -d sinapsis-parakeet
+```
+
 3. **Check the logs**
 
 - For ElevenLabs:
@@ -224,6 +265,17 @@ docker logs -f sinapsis-kokoro
 ```bash
 docker logs -f sinapsis-zonos
 ```
+
+- For Orpheus-CPP:
+```bash
+docker logs -f sinapsis-orpheus-tts
+```
+
+- For Parakeet:
+```bash
+docker logs -f sinapsis-parakeet
+```
+
 4. **The logs will display the URL to access the webapp, e.g.,:**:
 ```bash
 Running on local URL:  http://127.0.0.1:7860
@@ -240,6 +292,17 @@ docker compose -f docker/compose_apps.yaml down
 
 To run the webapp using the <code>uv</code> package manager, follow these steps:
 
+
+> [!IMPORTANT]
+> If you're using sinapsis-orpheus-cpp, you need to export cuda environment variables:
+
+
+```bash
+export CMAKE_ARGS="-DGGML_CUDA=on"
+export FORCE_CMAKE="1"
+export CUDACXX=$(command -v nvcc)
+```
+
 1. **Sync the virtual environment**:
 
 ```bash

{sinapsis_speech-0.3.5 → sinapsis_speech-0.4.1}/README.md

@@ -9,7 +9,7 @@ Sinapsis Speech
 <br>
 </h1>
 
-<h4 align="center"> Templates for a wide range of voice generation tasks.</h4>
+<h4 align="center"> A monorepo housing multiple packages and templates for versatile voice generation, text-to-speech, speech-to-text, and beyond.</h4>
 
 <p align="center">
 <a href="#installation">🐍 Installation</a> •
@@ -31,8 +31,10 @@ This repo includes packages for performing speech synthesis using different tool
 
 * <code>sinapsis-elevenlabs</code>
 * <code>sinapsis-f5-tts</code>
-* * <code>sinapsis-kokoro</code>
+* <code>sinapsis-kokoro</code>
 * <code>sinapsis-zonos</code>
+* <code>sinapsis-orpheus-cpp</code>
+* <code>sinapsis-parakeet</code>
 
 Install using your preferred package manager. We strongly recommend using <code>uv</code>. To install <code>uv</code>, refer to the [official documentation](https://docs.astral.sh/uv/getting-started/installation/#installation-methods).
 
@@ -80,10 +82,14 @@ This repository is organized into modular packages, each designed for integratio
 <details>
 <summary id="elevenlabs"><strong><span style="font-size: 1.4em;"> Sinapsis ElevenLabs </span></strong></summary>
 
-This package offers a suite of templates and utilities designed for effortless integrating, configuration, and execution of **text-to-speech (TTS)** and **voice generation** functionalities powered by [ElevenLabs](https://elevenlabs.io/).
+This package offers a suite of templates and utilities designed for effortless integrating, configuration, and execution of **text-to-speech (TTS)**, **speech-to-speech (STS)**, **voice cloning**, and **voice generation** functionalities powered by [ElevenLabs](https://elevenlabs.io/).
+
+- **ElevenLabsSTS**: Template for transforming a voice into a different character or style using the ElevenLabs Speech-to-Speech API.
 
 - **ElevenLabsTTS**: Template for converting text into speech using ElevenLabs' voice models.
 
+- **ElevenLabsVoiceClone**: Template for creating a synthetic copy of an existing voice using the ElevenLabs API.
+
 - **ElevenLabsVoiceGeneration**: Template for generating custom synthetic voices based on user-provided descriptions.
 
 For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_elevenlabs/README.md).
@@ -120,6 +126,30 @@ For specific instructions and further details, see the [README.md](https://githu
 
 </details>
 
+
+<details>
+<summary id="orpheus-cpp"><strong><span style="font-size: 1.4em;"> Sinapsis Orppheus-CPP</span></strong></summary>
+
+This package provides a template for seamlessly integrating, configuring, and running **text-to-speech (TTS)** functionalities powered by [Orpheus-TTS](https://github.com/canopyai/Orpheus-TTS).
+
+- **OrpheusTTS**: Converts text to speech using the Orpheus TTS model with advanced neural voice synthesis. The template processes text packets from the input container, generates corresponding audio using Orpheus TTS, and adds the resulting audio packets to the container. Features graceful error handling for out-of-memory conditions
+
+For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_orpheus_cpp/README.md).
+
+</details>
+
+<details>
+<summary id="parakeet-tdt"><strong><span style="font-size: 1.4em;"> Sinapsis Parakeet-TDT</span></strong></summary>
+
+This package provides a template for seamlessly integrating, configuring, and running **speech-to-text (STT)** functionalities powered by [NVIDIA's Parakeet TDT model](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2).
+
+
+- **ParakeetTDTInference**: Converts speech to text using NVIDIA's Parakeet TDT 0.6B model. This template processes audio packets from the input container or specified file paths, performs transcription with optional timestamp prediction, and adds the resulting text packets to the container.
+
+For specific instructions and further details, see the [README.md](https://github.com/Sinapsis-AI/sinapsis-speech/blob/main/packages/sinapsis_parakeet_tdt/README.md).
+
+</details>
+
 <h2 id="webapp">🌐 Webapps</h2>
 The webapps included in this project showcase the modularity of the templates, in this case for speech generation tasks.
 
@@ -158,7 +188,6 @@ cd sinapsis-speech
 docker compose -f docker/compose.yaml build
 ```
 
-
 2. **Start the app container**:
 
 - For ElevenLabs:
@@ -181,6 +210,16 @@ docker compose -f docker/compose_apps.yaml up -d sinapsis-kokoro
 docker compose -f docker/compose_apps.yaml up -d sinapsis-zonos
 ```
 
+- For Orpheus-CPP:
+```bash
+docker compose -f docker/compose_apps.yaml up -d sinapsis-orpheus-tts
+```
+
+- For Parakeet:
+```bash
+docker compose -f docker/compose_apps.yaml up -d sinapsis-parakeet
+```
+
 3. **Check the logs**
 
 - For ElevenLabs:
@@ -200,6 +239,17 @@ docker logs -f sinapsis-kokoro
 ```bash
 docker logs -f sinapsis-zonos
 ```
+
+- For Orpheus-CPP:
+```bash
+docker logs -f sinapsis-orpheus-tts
+```
+
+- For Parakeet:
+```bash
+docker logs -f sinapsis-parakeet
+```
+
 4. **The logs will display the URL to access the webapp, e.g.,:**:
 ```bash
 Running on local URL:  http://127.0.0.1:7860
@@ -216,6 +266,17 @@ docker compose -f docker/compose_apps.yaml down
 
 To run the webapp using the <code>uv</code> package manager, follow these steps:
 
+
+> [!IMPORTANT]
+> If you're using sinapsis-orpheus-cpp, you need to export cuda environment variables:
+
+
+```bash
+export CMAKE_ARGS="-DGGML_CUDA=on"
+export FORCE_CMAKE="1"
+export CUDACXX=$(command -v nvcc)
+```
+
 1. **Sync the virtual environment**:
 
 ```bash

sinapsis_speech-0.4.1/packages/sinapsis_elevenlabs/src/sinapsis_elevenlabs/helpers/tags.py

@@ -0,0 +1,15 @@
+# -*- coding: utf-8 -*-
+from enum import Enum
+
+
+class Tags(Enum):
+    AUDIO = "audio"
+    AUDIO_GENERATION = "audio_generation"
+    ELEVENLABS = "elevenlabs"
+    PROMPT = "prompt"
+    SPEECH = "speech"
+    SPEECH_TO_SPEECH = "speech_to_speech"
+    TEXT_TO_SPEECH = "text_to_speech"
+    VOICE_CONVERSION = "voice_conversion"
+    VOICE_CLONING = "voice_cloning"
+    VOICE_GENERATION = "voice_generation"

sinapsis_speech-0.4.1/packages/sinapsis_elevenlabs/src/sinapsis_elevenlabs/helpers/voice_utils.py

@@ -0,0 +1,100 @@
+# -*- coding: utf-8 -*-
+import json
+
+from elevenlabs import Voice, VoiceSettings
+from elevenlabs.client import ElevenLabs
+from sinapsis_core.data_containers.data_packet import TextPacket
+from sinapsis_core.utils.logging_utils import sinapsis_logger
+
+
+def create_voice_settings(settings: VoiceSettings, as_json: bool = False) -> VoiceSettings | None | str:
+    """
+    Creates or updates a `VoiceSettings` object based on the provided settings.
+
+    Args:
+        settings (VoiceSettings | None): An instance of `VoiceSettings` containing the settings to be applied.
+            If `None`, the function returns the default settings.
+        as_json (bool): Whether to return the settings as JSON string.
+
+    Returns:
+        VoiceSettings | None | str: The provided `VoiceSettings` object if `settings` is not `None`. Otherwise,
+            `None` is returned for default settings.
+    """
+    if not settings:
+        return None
+
+    if as_json:
+        return json.dumps(settings.model_dump(exclude_none=True))
+
+    return settings
+
+
+def get_voice_id(client: ElevenLabs, voice: str | Voice | None) -> str:
+    """
+    Resolves the voice ID for a given voice name or ID.
+
+    This function searches through available voices from the ElevenLabs API
+    to match the provided voice name or ID. If the specified voice is not found,
+    it logs the error and returns the first available voice ID as a fallback.
+
+    Args:
+        client (ElevenLabs): The ElevenLabs API client instance.
+        voice (str | Voice | None): The name or ID of the desired voice.
+
+    Returns:
+        str: The resolved voice ID.
+
+    Raises:
+        ValueError: If no voices are available to resolve.
+    """
+    if not voice:
+        return get_default_voice(client).voice_id
+
+    if isinstance(voice, Voice):
+        sinapsis_logger.debug(f"Voice object provided, using voice_id: {voice.voice_id}")
+        return voice.voice_id
+
+    try:
+        voices_response = client.voices.get_all()
+        voices = voices_response.voices
+
+        for v in voices:
+            if voice == v.name or voice == v.voice_id:
+                sinapsis_logger.debug(f"Voice {voice} resolved to ID: {v.voice_id}")
+                return v.voice_id
+
+        sinapsis_logger.error(f"Voice {voice} is not available.")
+        if voices:
+            sinapsis_logger.info(f"Returning default voice ID: {voices[0].voice_id}")
+            return voices[0].voice_id
+
+        raise ValueError("No available voices to resolve. Ensure the client is configured correctly.")
+    except Exception as e:
+        sinapsis_logger.error(f"Error resolving voice ID: {e}")
+        raise
+
+
+def get_default_voice(client: ElevenLabs) -> Voice:
+    """
+    Gets the first available voice as default.
+
+    Args:
+        client (ElevenLabs): The ElevenLabs API client instance.
+
+    Returns:
+        Voice: The default voice object.
+    """
+    try:
+        voices_response = client.voices.get_all()
+        voices = voices_response.voices
+        if voices:
+            return voices[0]
+        raise ValueError("No voices available")
+    except Exception as e:
+        sinapsis_logger.error(f"Error getting default voice: {e}")
+        raise
+
+
+def load_input_text(input_data: list[TextPacket]) -> str:
+    """Loads and concatenates the text content from a list of TextPacket objects."""
+    return "".join([item.content for item in input_data])

{sinapsis_speech-0.3.5 → sinapsis_speech-0.4.1}/packages/sinapsis_elevenlabs/src/sinapsis_elevenlabs/templates/__init__.py

@@ -7,6 +7,8 @@ _root_lib_path = "sinapsis_elevenlabs.templates"
 _template_lookup = {
     "ElevenLabsTTS": f"{_root_lib_path}.elevenlabs_tts",
     "ElevenLabsVoiceGeneration": f"{_root_lib_path}.elevenlabs_voice_generation",
+    "ElevenLabsVoiceClone": f"{_root_lib_path}.elevenlabs_voice_clone",
+    "ElevenLabsSTS": f"{_root_lib_path}.elevenlabs_sts",
 }
 
 

{sinapsis_speech-0.3.5 → sinapsis_speech-0.4.1}/packages/sinapsis_elevenlabs/src/sinapsis_elevenlabs/templates/elevenlabs_base.py

@@ -3,12 +3,11 @@
 
 import abc
 import os
-import uuid
-from io import BytesIO
-from typing import IO, Iterator, Literal
+from typing import Generator, Iterable, Iterator, Literal
 
-from elevenlabs import Voice, VoiceSettings, save
-from elevenlabs.client import ElevenLabs, VoiceId, VoiceName
+import numpy as np
+from elevenlabs import Voice, VoiceSettings
+from elevenlabs.client import ElevenLabs
 from elevenlabs.types import OutputFormat
 from pydantic import Field
 from sinapsis_core.data_containers.data_packet import AudioPacket, DataContainer, Packet
@@ -19,9 +18,11 @@ from sinapsis_core.template_base.base_models import (
     UIPropertiesMetadata,
 )
 from sinapsis_core.template_base.template import Template
-from sinapsis_core.utils.env_var_keys import SINAPSIS_CACHE_DIR
+from sinapsis_core.utils.env_var_keys import WORKING_DIR
+from sinapsis_generic_data_tools.helpers.audio_encoder import audio_bytes_to_numpy
 
 from sinapsis_elevenlabs.helpers.env_var_keys import ELEVENLABS_API_KEY
+from sinapsis_elevenlabs.helpers.tags import Tags
 
 RESPONSE_TYPE = Iterator[bytes] | list[bytes] | list[Iterator[bytes]] | None
 
@@ -51,9 +52,7 @@ class ElevenLabsBase(Template, abc.ABC):
             output_format (OutputFormat): The output audio format and quality. Options include:
                 ["mp3_22050_32", "mp3_44100_32", "mp3_44100_64", "mp3_44100_96", "mp3_44100_128",
                 "mp3_44100_192", "pcm_16000", "pcm_22050", "pcm_24000", "pcm_44100", "ulaw_8000"]
-            output_folder (str): The folder where generated audio files will be saved.
-            stream (bool): If True, the audio is returned as a stream; otherwise, saved to a file.
-            voice (VoiceId | VoiceName | Voice): The voice to use for speech synthesis. This can be a voice ID (str),
+            voice (str | Voice | None): The voice to use for speech synthesis. This can be a voice ID (str),
                 a voice name (str) or an elevenlabs voice object (Voice).
             voice_settings (VoiceSettings): A dictionary of settings that control the behavior of the voice.
                 - stability (float)
@@ -74,17 +73,20 @@ class ElevenLabsBase(Template, abc.ABC):
         ] = "eleven_turbo_v2_5"
         output_file_name: str | None = None
         output_format: OutputFormat = "mp3_44100_128"
-        output_folder: str = os.path.join(SINAPSIS_CACHE_DIR, "elevenlabs", "audios")
+        output_folder: str = os.path.join(WORKING_DIR, "elevenlabs", "audios")
         stream: bool = False
-        voice: VoiceId | VoiceName | Voice = None
+        voice: str | Voice | None = None
         voice_settings: VoiceSettings = Field(default_factory=dict)  # type: ignore[arg-type]
 
-    UIProperties = UIPropertiesMetadata(category="Elevenlabs", output_type=OutputTypes.AUDIO)
+    UIProperties = UIPropertiesMetadata(
+        category="Elevenlabs",
+        output_type=OutputTypes.AUDIO,
+        tags=[Tags.AUDIO, Tags.ELEVENLABS, Tags.SPEECH],
+    )
 
     def __init__(self, attributes: TemplateAttributeType) -> None:
         """Initializes the ElevenLabs API client with the given attributes."""
         super().__init__(attributes)
-        os.makedirs(self.attributes.output_folder, exist_ok=True)
         self.client = self.init_elevenlabs_client()
 
     def init_elevenlabs_client(self) -> ElevenLabs:
@@ -92,44 +94,27 @@ class ElevenLabsBase(Template, abc.ABC):
         key = self.attributes.api_key if self.attributes.api_key else ELEVENLABS_API_KEY
         return ElevenLabs(api_key=key)
 
-    def reset_state(self) -> None:
+    def reset_state(self, template_name: str | None = None) -> None:
         """Resets state of model"""
+        _ = template_name
         self.client = self.init_elevenlabs_client()
 
     @abc.abstractmethod
     def synthesize_speech(self, input_data: list[Packet]) -> RESPONSE_TYPE:
         """Abstract method for ElevenLabs speech synthesis."""
 
-    def _save_audio(self, response: Iterator[bytes] | bytes, file_format: str, idx: int) -> str:
-        """Saves the audio to a file and returns the file path."""
-        if self.attributes.output_file_name:
-            file_name = self.attributes.output_file_name + "_" + str(idx)
-        else:
-            file_name = uuid.uuid4()
-
-        output_file = os.path.join(self.attributes.output_folder, f"{file_name}.{file_format}")
-        try:
-            save(response, output_file)
-            self.logger.info(f"Audio saved to: {output_file}")
-            return output_file
-        except OSError as e:
-            self.logger.error(f"File system error while saving speech to file: {e}")
-            raise
-
-    def _generate_audio_stream(self, response: Iterator[bytes] | bytes) -> IO[bytes]:
+    def _generate_audio_stream(self, response: Iterable | bytes) -> bytes:
         """Generates and returns the audio stream."""
-        audio_stream = BytesIO()
+
         try:
             if isinstance(response, Iterator):
-                for chunk in response:
-                    if chunk:
-                        audio_stream.write(chunk)
+                audio_stream = b"".join(chunk for chunk in response)
             elif isinstance(response, bytes):
-                audio_stream.write(response)
+                audio_stream = response
+
             else:
                 raise TypeError(f"Unsupported response type: {type(response)}")
 
-            audio_stream.seek(0)
             self.logger.info("Returning audio stream")
             return audio_stream
         except IOError as e:
@@ -139,14 +124,15 @@ class ElevenLabsBase(Template, abc.ABC):
             self.logger.error(f"Value error while processing audio chunks: {e}")
             raise
 
-    def _process_audio_output(self, idx: int, response: Iterator[bytes] | bytes) -> str | IO[bytes]:
+    def _process_audio_output(self, response: Iterable | bytes) -> tuple[np.ndarray, int]:
         """Processes a single audio output (either stream or file)."""
-        if self.attributes.stream:
-            return self._generate_audio_stream(response)
-        file_format = "mp3" if "mp3" in self.attributes.output_format else "wav"
-        return self._save_audio(response, file_format, idx)
 
-    def generate_speech(self, input_data: list[Packet]) -> list[str | IO[bytes]] | None:
+        result = self._generate_audio_stream(response)
+        audio_np, sample_rate = audio_bytes_to_numpy(result)
+
+        return audio_np, sample_rate
+
+    def generate_speech(self, input_data: list[Packet]) -> list[tuple] | None:
         """Generates speech and saves it to a file."""
         responses: RESPONSE_TYPE = self.synthesize_speech(input_data)
         if not responses:
@@ -154,29 +140,29 @@ class ElevenLabsBase(Template, abc.ABC):
 
         if isinstance(responses, Iterator):
             responses = [responses]
-
-        audio_outputs = [self._process_audio_output(idx, response) for idx, response in enumerate(responses)]
+        elif isinstance(responses, Generator):
+            responses = list(responses)
+        audio_outputs = [self._process_audio_output(response) for response in responses]
         return audio_outputs
 
-    def _handle_streaming_output(self, audio_outputs: list[str | IO[bytes]]) -> list[AudioPacket]:
+    def _handle_streaming_output(self, audio_outputs: list[tuple]) -> list[AudioPacket]:
         """Handles audio stream output by adding it to the container as AudioPackets."""
         generated_audios: list[AudioPacket] = []
-        sample_rate = int(self.attributes.output_format.split("_")[1])
+        # sample_rate = int(self.attributes.output_format.split("_")[1])
         for audio_output in audio_outputs:
+            audio = audio_output[0]
+            sample_rate = audio_output[1]
             audio_packet = AudioPacket(
-                content=audio_output,
+                content=audio,
                 sample_rate=sample_rate,
             )
             generated_audios.append(audio_packet)
         return generated_audios
 
-    def _handle_audio_outputs(self, audio_outputs: list[str | IO[bytes]], container: DataContainer) -> None:
+    def _handle_audio_outputs(self, audio_outputs: list[tuple], container: DataContainer) -> None:
         """Handles the audio outputs by appending to the container based on the output type (stream or file)."""
-        if self.attributes.stream:
-            container.audios = container.audios or []
-            container.audios.extend(self._handle_streaming_output(audio_outputs))
-        else:
-            self._set_generic_data(container, audio_outputs)
+        container.audios = container.audios or []
+        container.audios = self._handle_streaming_output(audio_outputs)
 
     def execute(self, container: DataContainer) -> DataContainer:
         """

sinapsis_speech-0.4.1/packages/sinapsis_elevenlabs/src/sinapsis_elevenlabs/templates/elevenlabs_sts.py

@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+"""Speech-To-Speech template for ElevenLabs."""
+
+from typing import Callable, Iterator, Literal
+
+from sinapsis_core.data_containers.data_packet import AudioPacket
+
+from sinapsis_elevenlabs.helpers.tags import Tags
+from sinapsis_elevenlabs.helpers.voice_utils import create_voice_settings, get_voice_id
+from sinapsis_elevenlabs.templates.elevenlabs_base import ElevenLabsBase
+
+ElevenLabsSTSUIProperties = ElevenLabsBase.UIProperties
+ElevenLabsSTSUIProperties.tags.extend([Tags.SPEECH_TO_SPEECH, Tags.VOICE_CONVERSION])
+
+
+class ElevenLabsSTS(ElevenLabsBase):
+    """Template to interact with the ElevenLabs Speech-to-Speech API.
+
+    This template takes an input audio and converts it to a new voice using
+    the ElevenLabs Speech-to-Speech (STS) API.
+
+    Usage example:
+
+    agent:
+      name: my_test_agent
+    templates:
+    - template_name: InputTemplate
+      class_name: InputTemplate
+      attributes: {}
+    - template_name: ElevenLabsSTS
+      class_name: ElevenLabsSTS
+      template_input: InputTemplate
+      attributes:
+        api_key: null
+        model: eleven_multilingual_sts_v2
+        output_file_name: null
+        output_format: mp3_44100_128
+        output_folder: <WORKING_DIR>/elevenlabs/audios
+        stream: false
+        voice: null
+        voice_settings:
+            stability: null
+            similarity_boost: null
+            style: null
+            use_speaker_boost: null
+            speed: null
+        streaming_latency: null
+
+    """
+
+    PACKET_TYPE_NAME: str = "audios"
+    UIProperties = ElevenLabsSTSUIProperties
+
+    class AttributesBaseModel(ElevenLabsBase.AttributesBaseModel):
+        """Attributes specific to ElevenLabs STS API interaction.
+
+        Attributes:
+            model (Literal): The STS model to use. Options are "eleven_english_sts_v2" or "eleven_multilingual_sts_v2".
+            streaming_latency (int | None): Optional latency optimization for streaming. Defaults to None.
+        """
+
+        model: Literal["eleven_english_sts_v2", "eleven_multilingual_sts_v2"] = "eleven_multilingual_sts_v2"
+        streaming_latency: int | None = None
+
+    def synthesize_speech(self, input_data: list[AudioPacket]) -> Iterator[bytes]:
+        """Sends an audio input to the ElevenLabs API for speech-to-speech synthesis.
+
+        Args:
+            input_data (list[AudioPacket]): List of AudioPacket objects containing the audio to be converted.
+                Only the first AudioPacket in the list is used.
+
+        Returns:
+            Iterator[bytes]: An iterator yielding audio data chunks in the output format specified.
+
+        Raises:
+            ValueError: If there is a problem with the input data or parameters.
+            TypeError: If the input data or files are of incorrect type.
+            KeyError: If the expected key is missing in the API response.
+        """
+        try:
+            method: Callable[..., Iterator[bytes]] = self.client.speech_to_speech.stream  # (
+
+            return method(
+                voice_id=get_voice_id(self.client, voice=self.attributes.voice),
+                audio=input_data[0].content,
+                model_id=self.attributes.model,
+                voice_settings=create_voice_settings(self.attributes.voice_settings, as_json=True),
+                output_format=self.attributes.output_format,
+                optimize_streaming_latency=self.attributes.streaming_latency,
+            )
+        except ValueError as e:
+            self.logger.error(f"Value error synthesizing speech: {e}")
+            raise
+        except TypeError as e:
+            self.logger.error(f"Type error in input data or parameters: {e}")
+            raise
+        except KeyError as e:
+            self.logger.error(f"Missing key in input data or settings: {e}")
+            raise