voxy 0.0.2__tar.gz

voxy-0.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

voxy-0.0.2/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.2
+Name: voxy
+Version: 0.0.2
+Summary: Facade for voice cloning and speech synthesis
+Home-page: https://github.com/thorwhalen/voxy
+Author: Thor Whalen
+License: mit
+Platform: any
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# voxy
+
+Facade for voice cloning and speech synthesis
+
+To install:	```pip install voxy```
+
+Voxy is a flexible Python module for speech synthesis and voice cloning, with initial support for the Sesame CSM-1B model. It provides a plugin architecture that can be extended to support other models in the future.
+
+## Features
+
+- Voice cloning from audio samples
+- High-quality speech synthesis
+- Flexible input formats (file paths, bytes, streams, tensors)
+- Audio cleanup utilities
+- Automatic audio transcription (using Whisper)
+- Plugin architecture for different speech models
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10+
+- PyTorch and TorchAudio
+- CUDA-compatible GPU (recommended)
+- FFmpeg for audio processing
+
+### Install the CSM Model
+
+The intention is to make `voxy` into a plugin-enabled facade, where you can chose your 
+own engine (for voice cloning, voice synthesis, etc.). 
+But for now, we just support, what seems to be the best open-source model out there
+(at the time of writing this): 
+[Sesame AI Lab's](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice) 
+CSM model. It's just that, well, they did an amazing job at the model, but a terrible one
+(so far) for the python interface -- which is what inspired me to develop `voxy` 
+in the first place.
+
+Follow the instructions in the [CSM repository](https://github.com/SesameAILabs/csm) 
+to install the CSM model and its dependencies.
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Generate speech with default voice
+audio = model.generate_speech(
+    text="Hello, this is a test of the CSM speech model.", 
+    output_path="output.wav"
+)
+```
+
+### Voice Cloning
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice from an audio file
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice for cloning purposes."
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This is my cloned voice speaking. Isn't it amazing?",
+    voice_profile=voice_profile,
+    output_path="cloned_voice.wav"
+)
+```
+
+### Automatic Transcription
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice with automatic transcription
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    # No transcript provided, will use automatic transcription
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This voice was cloned using automatic transcription.",
+    voice_profile=voice_profile,
+    output_path="auto_transcribed_voice.wav"
+)
+```
+
+### Flexible Input Formats
+
+The module supports various input formats:
+
+```python
+# From file path
+voice_profile1 = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="Text transcript."
+)
+
+# From bytes
+with open("sample_voice.wav", "rb") as f:
+    audio_bytes = f.read()
+voice_profile2 = model.clone_voice(
+    audio_input=audio_bytes,
+    transcript="Text transcript."
+)
+
+# From file object
+with open("sample_voice.wav", "rb") as f:
+    voice_profile3 = model.clone_voice(
+        audio_input=f,
+        transcript="Text transcript."
+    )
+
+# From tensor
+import torch
+import torchaudio
+audio_tensor, sample_rate = torchaudio.load("sample_voice.wav")
+voice_profile4 = model.clone_voice(
+    audio_input=audio_tensor,
+    transcript="Text transcript."
+)
+```
+
+## Configuration
+
+You can configure the default device by setting the `DFLT_VOXY_DEVICE` environment variable:
+
+```bash
+# Use CUDA
+export DFLT_VOXY_DEVICE=cuda
+
+# Use CPU
+export DFLT_VOXY_DEVICE=cpu
+
+# Use MPS (Apple Silicon)
+export DFLT_VOXY_DEVICE=mps
+```
+
+## Advanced Usage
+
+### Audio Cleanup
+
+The module includes an audio cleanup function that normalizes volume and removes silence:
+
+```python
+from voxy import cleanup_audio
+import torchaudio
+
+# Load audio
+audio, sample_rate = torchaudio.load("noisy_audio.wav")
+
+# Clean up audio
+cleaned_audio = cleanup_audio(
+    audio=audio,
+    sample_rate=sample_rate,
+    normalize=True,
+    remove_silence=True,
+    silence_threshold=0.02,
+    min_silence_duration=0.2
+)
+
+# Save cleaned audio
+torchaudio.save("cleaned_audio.wav", cleaned_audio, sample_rate)
+```
+
+### Disabling Audio Cleanup
+
+You can disable audio cleanup when cloning a voice:
+
+```python
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice.",
+    cleanup_audio_fn=None  # Disable audio cleanup
+)
+```
+
+### Custom Audio Cleanup
+
+You can also provide your own audio cleanup function:
+
+```python
+def my_custom_cleanup(audio, sample_rate, **kwargs):
+    # Custom cleanup logic
+    return processed_audio
+
+voice_profile = model.

voxy-0.0.2/README.md

@@ -0,0 +1,201 @@
+# voxy
+
+Facade for voice cloning and speech synthesis
+
+To install:	```pip install voxy```
+
+Voxy is a flexible Python module for speech synthesis and voice cloning, with initial support for the Sesame CSM-1B model. It provides a plugin architecture that can be extended to support other models in the future.
+
+## Features
+
+- Voice cloning from audio samples
+- High-quality speech synthesis
+- Flexible input formats (file paths, bytes, streams, tensors)
+- Audio cleanup utilities
+- Automatic audio transcription (using Whisper)
+- Plugin architecture for different speech models
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10+
+- PyTorch and TorchAudio
+- CUDA-compatible GPU (recommended)
+- FFmpeg for audio processing
+
+### Install the CSM Model
+
+The intention is to make `voxy` into a plugin-enabled facade, where you can chose your 
+own engine (for voice cloning, voice synthesis, etc.). 
+But for now, we just support, what seems to be the best open-source model out there
+(at the time of writing this): 
+[Sesame AI Lab's](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice) 
+CSM model. It's just that, well, they did an amazing job at the model, but a terrible one
+(so far) for the python interface -- which is what inspired me to develop `voxy` 
+in the first place.
+
+Follow the instructions in the [CSM repository](https://github.com/SesameAILabs/csm) 
+to install the CSM model and its dependencies.
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Generate speech with default voice
+audio = model.generate_speech(
+    text="Hello, this is a test of the CSM speech model.", 
+    output_path="output.wav"
+)
+```
+
+### Voice Cloning
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice from an audio file
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice for cloning purposes."
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This is my cloned voice speaking. Isn't it amazing?",
+    voice_profile=voice_profile,
+    output_path="cloned_voice.wav"
+)
+```
+
+### Automatic Transcription
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice with automatic transcription
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    # No transcript provided, will use automatic transcription
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This voice was cloned using automatic transcription.",
+    voice_profile=voice_profile,
+    output_path="auto_transcribed_voice.wav"
+)
+```
+
+### Flexible Input Formats
+
+The module supports various input formats:
+
+```python
+# From file path
+voice_profile1 = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="Text transcript."
+)
+
+# From bytes
+with open("sample_voice.wav", "rb") as f:
+    audio_bytes = f.read()
+voice_profile2 = model.clone_voice(
+    audio_input=audio_bytes,
+    transcript="Text transcript."
+)
+
+# From file object
+with open("sample_voice.wav", "rb") as f:
+    voice_profile3 = model.clone_voice(
+        audio_input=f,
+        transcript="Text transcript."
+    )
+
+# From tensor
+import torch
+import torchaudio
+audio_tensor, sample_rate = torchaudio.load("sample_voice.wav")
+voice_profile4 = model.clone_voice(
+    audio_input=audio_tensor,
+    transcript="Text transcript."
+)
+```
+
+## Configuration
+
+You can configure the default device by setting the `DFLT_VOXY_DEVICE` environment variable:
+
+```bash
+# Use CUDA
+export DFLT_VOXY_DEVICE=cuda
+
+# Use CPU
+export DFLT_VOXY_DEVICE=cpu
+
+# Use MPS (Apple Silicon)
+export DFLT_VOXY_DEVICE=mps
+```
+
+## Advanced Usage
+
+### Audio Cleanup
+
+The module includes an audio cleanup function that normalizes volume and removes silence:
+
+```python
+from voxy import cleanup_audio
+import torchaudio
+
+# Load audio
+audio, sample_rate = torchaudio.load("noisy_audio.wav")
+
+# Clean up audio
+cleaned_audio = cleanup_audio(
+    audio=audio,
+    sample_rate=sample_rate,
+    normalize=True,
+    remove_silence=True,
+    silence_threshold=0.02,
+    min_silence_duration=0.2
+)
+
+# Save cleaned audio
+torchaudio.save("cleaned_audio.wav", cleaned_audio, sample_rate)
+```
+
+### Disabling Audio Cleanup
+
+You can disable audio cleanup when cloning a voice:
+
+```python
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice.",
+    cleanup_audio_fn=None  # Disable audio cleanup
+)
+```
+
+### Custom Audio Cleanup
+
+You can also provide your own audio cleanup function:
+
+```python
+def my_custom_cleanup(audio, sample_rate, **kwargs):
+    # Custom cleanup logic
+    return processed_audio
+
+voice_profile = model.

voxy-0.0.2/setup.cfg

@@ -0,0 +1,25 @@
+[metadata]
+name = voxy
+version = 0.0.2
+url = https://github.com/thorwhalen/voxy
+platforms = any
+description_file = README.md
+root_url = https://github.com/thorwhalen/
+license = mit
+author = Thor Whalen
+description = Facade for voice cloning and speech synthesis
+long_description = file:README.md
+long_description_content_type = text/markdown
+keywords = 
+display_name = voxy
+
+[options]
+packages = find:
+include_package_data = True
+zip_safe = False
+install_requires = 
+
+[egg_info]
+tag_build = 
+tag_date = 0
+

voxy-0.0.2/setup.py

@@ -0,0 +1,3 @@
+from setuptools import setup
+
+setup()  # Note: Everything should be in the local setup.cfg

voxy-0.0.2/voxy/__init__.py

@@ -0,0 +1,5 @@
+"""
+Facade for voice cloning and speech synthesis
+"""
+
+from voxy.base import create_speech_model, audio_to_text, cleanup_audio

voxy-0.0.2/voxy/base.py

@@ -0,0 +1,576 @@
+"""
+Voxy: A flexible speech synthesis and voice cloning module.
+
+This module provides a plugin architecture for working with different speech synthesis
+models, with initial support for the CSM-1B model.
+"""
+
+import os
+import io
+import pathlib
+from typing import Union, Optional, List, Dict, Any, Callable, BinaryIO, Tuple
+from dataclasses import dataclass
+import tempfile
+
+import torch
+import torchaudio
+import numpy as np
+from huggingface_hub import hf_hub_download
+
+# Try to import whisper for transcription, but don't fail if it's not available
+try:
+    import whisper
+
+    _HAS_WHISPER = True
+except ImportError:
+    _HAS_WHISPER = False
+
+
+# Determine the default device for model inference
+DFLT_VOXY_DEVICE = os.environ.get("DFLT_VOXY_DEVICE", None)
+
+if DFLT_VOXY_DEVICE is None:
+    if torch.backends.mps.is_available():
+        DFLT_VOXY_DEVICE = "mps"
+    elif torch.cuda.is_available():
+        DFLT_VOXY_DEVICE = "cuda"
+    else:
+        DFLT_VOXY_DEVICE = "cpu"
+
+
+# -----------------------------------------------------------------------------
+# Helper functions for input normalization
+# -----------------------------------------------------------------------------
+
+
+def _resolve_audio_input(
+    audio_input: Union[str, bytes, BinaryIO, torch.Tensor, np.ndarray],
+) -> Tuple[torch.Tensor, int]:
+    """
+    Resolves various audio input formats to a torch.Tensor and sample rate.
+
+    Args:
+        audio_input: Audio in various formats:
+            - str: Path to an audio file
+            - bytes: Raw audio data
+            - BinaryIO: File-like object containing audio data
+            - torch.Tensor: Direct audio tensor
+            - np.ndarray: Numpy array of audio samples
+
+    Returns:
+        Tuple of (audio_tensor, sample_rate)
+    """
+    if isinstance(audio_input, str):
+        # Check if it's a file path
+        if os.path.isfile(audio_input):
+            return torchaudio.load(audio_input)
+        else:
+            raise ValueError(f"Audio path does not exist: {audio_input}")
+
+    elif isinstance(audio_input, bytes):
+        # Convert bytes to file-like object
+        byte_stream = io.BytesIO(audio_input)
+        return torchaudio.load(byte_stream)
+
+    elif isinstance(audio_input, (io.IOBase, BinaryIO)):
+        # File-like object
+        return torchaudio.load(audio_input)
+
+    elif isinstance(audio_input, torch.Tensor):
+        # Assume default sample rate of 16000 if directly passed tensor
+        # and the tensor shape is [channels, samples] or [samples]
+        if len(audio_input.shape) > 2:
+            raise ValueError(f"Invalid audio tensor shape: {audio_input.shape}")
+        return audio_input, 16000
+
+    elif isinstance(audio_input, np.ndarray):
+        # Convert numpy array to tensor
+        # Assume default sample rate of 16000
+        audio_tensor = torch.from_numpy(audio_input)
+        if len(audio_tensor.shape) == 1:
+            # Add channel dimension if not present
+            audio_tensor = audio_tensor.unsqueeze(0)
+        return audio_tensor, 16000
+
+    else:
+        raise TypeError(f"Unsupported audio input type: {type(audio_input)}")
+
+
+def _resolve_text_input(text_input: Union[str, bytes, io.TextIOBase]) -> str:
+    """
+    Resolves various text input formats to a string.
+
+    Args:
+        text_input: Text in various formats:
+            - str: Direct text or path to a text file
+            - bytes: UTF-8 encoded text
+            - TextIOBase: File-like object containing text
+
+    Returns:
+        String containing the text
+    """
+    if isinstance(text_input, str):
+        # If it starts with / and is a file, read the content
+        if text_input.startswith('/') and os.path.isfile(text_input):
+            with open(text_input, 'r') as f:
+                return f.read()
+        # Otherwise, use the string directly
+        return text_input
+
+    elif isinstance(text_input, bytes):
+        # Decode bytes to string
+        return text_input.decode('utf-8')
+
+    elif isinstance(text_input, io.TextIOBase):
+        # Read from file-like object
+        return text_input.read()
+
+    else:
+        raise TypeError(f"Unsupported text input type: {type(text_input)}")
+
+
+# -----------------------------------------------------------------------------
+# Audio processing functions
+# -----------------------------------------------------------------------------
+
+
+def cleanup_audio(
+    audio: torch.Tensor,
+    sample_rate: int,
+    normalize: bool = True,
+    remove_silence: bool = True,
+    silence_threshold: float = 0.02,
+    min_silence_duration: float = 0.2,
+) -> torch.Tensor:
+    """
+    Clean up audio by normalizing volume and removing silence.
+
+    Args:
+        audio: Audio tensor [channels, samples] or [samples]
+        sample_rate: Sample rate of the audio
+        normalize: Whether to normalize the audio volume
+        remove_silence: Whether to remove silence
+        silence_threshold: Threshold for silence detection (0.0-1.0)
+        min_silence_duration: Minimum silence duration in seconds
+
+    Returns:
+        Processed audio tensor
+    """
+    # Ensure input is 2D with shape [channels, samples]
+    if len(audio.shape) == 1:
+        audio = audio.unsqueeze(0)
+
+    # Convert to mono if not already
+    if audio.shape[0] > 1:
+        audio = torch.mean(audio, dim=0, keepdim=True)
+
+    # Move to CPU for processing
+    device = audio.device
+    audio = audio.cpu()
+
+    # Normalize volume
+    if normalize:
+        max_val = torch.max(torch.abs(audio))
+        if max_val > 0:
+            audio = audio / (max_val + 1e-8)
+
+    # Remove silence
+    if remove_silence:
+        # Convert to numpy for easier processing
+        audio_np = audio.squeeze(0).numpy()
+
+        # Calculate energy
+        energy = np.abs(audio_np)
+
+        # Find regions above threshold (speech)
+        is_speech = energy > silence_threshold
+
+        # Convert min_silence_duration to samples
+        min_silence_samples = int(min_silence_duration * sample_rate)
+
+        # Find speech segments
+        speech_segments = []
+        in_speech = False
+        speech_start = 0
+
+        for i in range(len(is_speech)):
+            if is_speech[i] and not in_speech:
+                # Start of speech segment
+                in_speech = True
+                speech_start = i
+            elif not is_speech[i] and in_speech:
+                # Potential end of speech segment
+                # Only end if silence is long enough
+                silence_count = 0
+                for j in range(i, min(len(is_speech), i + min_silence_samples)):
+                    if not is_speech[j]:
+                        silence_count += 1
+                    else:
+                        break
+
+                if silence_count >= min_silence_samples:
+                    # End of speech segment
+                    in_speech = False
+                    speech_segments.append((speech_start, i))
+
+        # Handle case where audio ends during speech
+        if in_speech:
+            speech_segments.append((speech_start, len(is_speech)))
+
+        # Concatenate speech segments
+        if not speech_segments:
+            # If no speech found, return original audio
+            processed_audio = audio
+        else:
+            # Add small buffer around segments
+            buffer_samples = int(0.05 * sample_rate)  # 50ms buffer
+            processed_segments = []
+
+            for start, end in speech_segments:
+                buffered_start = max(0, start - buffer_samples)
+                buffered_end = min(len(audio_np), end + buffer_samples)
+                processed_segments.append(audio_np[buffered_start:buffered_end])
+
+            # Concatenate all segments
+            processed_audio_np = np.concatenate(processed_segments)
+            processed_audio = torch.tensor(processed_audio_np, device='cpu').unsqueeze(
+                0
+            )
+    else:
+        processed_audio = audio
+
+    # Return to original device
+    return processed_audio.to(device)
+
+
+def audio_to_text(
+    audio_input: Union[str, bytes, BinaryIO, torch.Tensor, np.ndarray],
+    model_size: str = "base",
+) -> str:
+    """
+    Transcribe audio to text using Whisper.
+
+    Args:
+        audio_input: Audio in various formats
+        model_size: Whisper model size ('tiny', 'base', 'small', 'medium', 'large')
+
+    Returns:
+        Transcribed text
+
+    Raises:
+        ImportError: If whisper is not installed
+    """
+    if not _HAS_WHISPER:
+        raise ImportError(
+            "whisper is required for transcription. Install with 'pip install whisper'"
+        )
+
+    # Resolve audio input
+    audio, sample_rate = _resolve_audio_input(audio_input)
+
+    # Load whisper model
+    model = whisper.load_model(model_size)
+
+    # If audio is a torch tensor, convert to numpy array
+    if isinstance(audio, torch.Tensor):
+        # Ensure mono
+        if len(audio.shape) > 1 and audio.shape[0] > 1:
+            audio = torch.mean(audio, dim=0)
+        else:
+            audio = audio.squeeze(0)
+
+        # Convert to numpy
+        audio_np = audio.cpu().numpy()
+    else:
+        audio_np = audio
+
+    # Resample if needed
+    if sample_rate != 16000:
+        # Whisper expects 16kHz
+        # Use torchaudio for resampling
+        audio_tensor = torch.tensor(audio_np).unsqueeze(0)
+        audio_tensor = torchaudio.functional.resample(
+            audio_tensor, orig_freq=sample_rate, new_freq=16000
+        )
+        audio_np = audio_tensor.squeeze(0).numpy()
+
+    # Transcribe
+    result = model.transcribe(audio_np)
+
+    return result["text"].strip()
+
+
+# -----------------------------------------------------------------------------
+# Main SpeechModel classes
+# -----------------------------------------------------------------------------
+
+
+@dataclass
+class VoiceProfile:
+    """Data class to store voice cloning information."""
+
+    segment: Any  # Model-specific voice segment
+    speaker_id: int
+    model_type: str
+    sample_rate: int
+    metadata: Dict[str, Any] = None
+
+
+class SpeechModel:
+    """Base class for speech synthesis models."""
+
+    def __init__(self, device: str = DFLT_VOXY_DEVICE):
+        """
+        Initialize the speech model.
+
+        Args:
+            device: Device for model inference ('cuda', 'cpu', 'mps')
+        """
+        self.device = device
+
+    def clone_voice(
+        self,
+        audio_input: Union[str, bytes, BinaryIO, torch.Tensor, np.ndarray],
+        transcript: Optional[str] = None,
+        speaker_id: int = 999,
+        *,
+        cleanup_audio_fn: Optional[Callable] = cleanup_audio,
+    ) -> VoiceProfile:
+        """
+        Create a voice profile from an audio sample and its transcript.
+
+        Args:
+            audio_input: Audio in various formats
+            transcript: Text transcription of the audio (if None, auto-transcribed)
+            speaker_id: Unique ID for this voice
+            cleanup_audio_fn: Function to clean up audio (None to skip)
+
+        Returns:
+            VoiceProfile: A packaged voice profile
+        """
+        raise NotImplementedError("Subclasses must implement this method")
+
+    def generate_speech(
+        self,
+        text: Union[str, bytes, io.TextIOBase],
+        voice_profile: Optional[VoiceProfile] = None,
+        output_path: Optional[str] = None,
+        max_length_ms: int = 10000,
+        **kwargs,
+    ) -> torch.Tensor:
+        """
+        Generate speech using a voice profile.
+
+        Args:
+            text: Text to synthesize
+            voice_profile: Voice profile from clone_voice()
+            output_path: Path to save the audio (optional)
+            max_length_ms: Maximum audio length in milliseconds
+            **kwargs: Additional model-specific parameters
+
+        Returns:
+            Generated audio tensor
+        """
+        raise NotImplementedError("Subclasses must implement this method")
+
+
+class CSMSpeechModel(SpeechModel):
+    """Speech model implementation using Sesame's CSM-1B model."""
+
+    def __init__(
+        self, model_path: Optional[str] = None, device: str = DFLT_VOXY_DEVICE
+    ):
+        """
+        Initialize the CSM speech model.
+
+        Args:
+            model_path: Path to the model checkpoint (None to download from HF)
+            device: Device for model inference ('cuda', 'cpu', 'mps')
+        """
+        super().__init__(device)
+        self.model_path = model_path
+        self._generator = None  # Lazy initialization
+
+    def _ensure_generator_loaded(self):
+        """Ensure the generator is loaded."""
+        if self._generator is None:
+            # Import here to avoid dependencies if not using CSM
+            from generator import load_csm_1b, Segment
+
+            if self.model_path is None:
+                # Download the model if not provided
+                try:
+                    self.model_path = hf_hub_download(
+                        repo_id="sesame/csm-1b", filename="ckpt.pt"
+                    )
+                except Exception as e:
+                    raise RuntimeError(
+                        "Failed to download CSM-1B model. Ensure you have huggingface-cli "
+                        f"installed and are logged in with appropriate permissions: {e}"
+                    )
+
+            # Load the generator
+            self._generator = load_csm_1b(self.model_path, self.device)
+
+            # Save a reference to the Segment class
+            self.Segment = Segment
+
+    def clone_voice(
+        self,
+        audio_input: Union[str, bytes, BinaryIO, torch.Tensor, np.ndarray],
+        transcript: Optional[str] = None,
+        speaker_id: int = 999,
+        *,
+        cleanup_audio_fn: Optional[Callable] = cleanup_audio,
+    ) -> VoiceProfile:
+        """
+        Create a voice profile from an audio sample and its transcript.
+
+        Args:
+            audio_input: Audio in various formats
+            transcript: Text transcription of the audio (if None, auto-transcribed)
+            speaker_id: Unique ID for this voice
+            cleanup_audio_fn: Function to clean up audio (None to skip)
+
+        Returns:
+            VoiceProfile: A packaged voice profile
+        """
+        # Load model if not already loaded
+        self._ensure_generator_loaded()
+
+        # Resolve audio input
+        audio_tensor, sample_rate = _resolve_audio_input(audio_input)
+
+        # Clean up audio if requested
+        if cleanup_audio_fn is not None:
+            audio_tensor = cleanup_audio_fn(audio_tensor, sample_rate)
+
+        # Convert to mono if stereo
+        if audio_tensor.shape[0] > 1:
+            audio_tensor = torch.mean(audio_tensor, dim=0, keepdim=True)
+
+        # Squeeze out channel dimension if present
+        audio_tensor = audio_tensor.squeeze(0)
+
+        # Resample if needed
+        if sample_rate != self._generator.sample_rate:
+            audio_tensor = torchaudio.functional.resample(
+                audio_tensor,
+                orig_freq=sample_rate,
+                new_freq=self._generator.sample_rate,
+            )
+
+        # Auto-transcribe if no transcript provided
+        if transcript is None:
+            transcript = audio_to_text(audio_tensor)
+        else:
+            # Resolve transcript if not a string
+            transcript = _resolve_text_input(transcript)
+
+        # Create segment for voice profile
+        segment = self.Segment(
+            text=transcript, speaker=speaker_id, audio=audio_tensor.to(self.device)
+        )
+
+        # Create and return voice profile
+        return VoiceProfile(
+            segment=segment,
+            speaker_id=speaker_id,
+            model_type="csm",
+            sample_rate=self._generator.sample_rate,
+            metadata={
+                "transcript_length": len(transcript),
+                "audio_length_seconds": len(audio_tensor) / self._generator.sample_rate,
+            },
+        )
+
+    def generate_speech(
+        self,
+        text: Union[str, bytes, io.TextIOBase],
+        voice_profile: Optional[VoiceProfile] = None,
+        output_path: Optional[str] = None,
+        max_length_ms: int = 10000,
+        temperature: float = 0.7,
+        topk: int = 30,
+    ) -> torch.Tensor:
+        """
+        Generate speech using a voice profile.
+
+        Args:
+            text: Text to synthesize
+            voice_profile: Voice profile from clone_voice()
+            output_path: Path to save the audio (optional)
+            max_length_ms: Maximum audio length in milliseconds
+            temperature: Sampling temperature (lower = more deterministic)
+            topk: Top-k sampling parameter
+
+        Returns:
+            Generated audio tensor
+        """
+        # Load model if not already loaded
+        self._ensure_generator_loaded()
+
+        # Resolve text input
+        text = _resolve_text_input(text)
+
+        # Set up context and speaker ID
+        if voice_profile is not None:
+            if voice_profile.model_type != "csm":
+                raise ValueError(
+                    f"Incompatible voice profile type: {voice_profile.model_type}"
+                )
+
+            context = [voice_profile.segment]
+            speaker_id = voice_profile.speaker_id
+        else:
+            # No voice profile, use default speaker
+            context = []
+            speaker_id = 0
+
+        # Add punctuation if missing to help with phrasing
+        if not any(p in text for p in ['.', ',', '!', '?']):
+            text = text + '.'
+
+        # Generate audio
+        audio = self._generator.generate(
+            text=text,
+            speaker=speaker_id,
+            context=context,
+            max_audio_length_ms=max_length_ms,
+            temperature=temperature,
+            topk=topk,
+        )
+
+        # Save if path provided
+        if output_path:
+            os.makedirs(os.path.dirname(os.path.abspath(output_path)), exist_ok=True)
+            torchaudio.save(
+                output_path, audio.unsqueeze(0).cpu(), self._generator.sample_rate
+            )
+
+        return audio
+
+
+# -----------------------------------------------------------------------------
+# Factory function for creating speech models
+# -----------------------------------------------------------------------------
+
+
+def create_speech_model(model_type: str = "csm", **kwargs) -> SpeechModel:
+    """
+    Create a speech model of the specified type.
+
+    Args:
+        model_type: Type of speech model ('csm' currently supported)
+        **kwargs: Additional model-specific parameters
+
+    Returns:
+        SpeechModel instance
+
+    Raises:
+        ValueError: If the model type is not supported
+    """
+    if model_type.lower() == "csm":
+        return CSMSpeechModel(**kwargs)
+    else:
+        raise ValueError(f"Unsupported model type: {model_type}")

voxy-0.0.2/voxy.egg-info/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.2
+Name: voxy
+Version: 0.0.2
+Summary: Facade for voice cloning and speech synthesis
+Home-page: https://github.com/thorwhalen/voxy
+Author: Thor Whalen
+License: mit
+Platform: any
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# voxy
+
+Facade for voice cloning and speech synthesis
+
+To install:	```pip install voxy```
+
+Voxy is a flexible Python module for speech synthesis and voice cloning, with initial support for the Sesame CSM-1B model. It provides a plugin architecture that can be extended to support other models in the future.
+
+## Features
+
+- Voice cloning from audio samples
+- High-quality speech synthesis
+- Flexible input formats (file paths, bytes, streams, tensors)
+- Audio cleanup utilities
+- Automatic audio transcription (using Whisper)
+- Plugin architecture for different speech models
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10+
+- PyTorch and TorchAudio
+- CUDA-compatible GPU (recommended)
+- FFmpeg for audio processing
+
+### Install the CSM Model
+
+The intention is to make `voxy` into a plugin-enabled facade, where you can chose your 
+own engine (for voice cloning, voice synthesis, etc.). 
+But for now, we just support, what seems to be the best open-source model out there
+(at the time of writing this): 
+[Sesame AI Lab's](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice) 
+CSM model. It's just that, well, they did an amazing job at the model, but a terrible one
+(so far) for the python interface -- which is what inspired me to develop `voxy` 
+in the first place.
+
+Follow the instructions in the [CSM repository](https://github.com/SesameAILabs/csm) 
+to install the CSM model and its dependencies.
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Generate speech with default voice
+audio = model.generate_speech(
+    text="Hello, this is a test of the CSM speech model.", 
+    output_path="output.wav"
+)
+```
+
+### Voice Cloning
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice from an audio file
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice for cloning purposes."
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This is my cloned voice speaking. Isn't it amazing?",
+    voice_profile=voice_profile,
+    output_path="cloned_voice.wav"
+)
+```
+
+### Automatic Transcription
+
+```python
+from voxy import create_speech_model
+
+# Create a speech model
+model = create_speech_model(model_type="csm")
+
+# Clone a voice with automatic transcription
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    # No transcript provided, will use automatic transcription
+)
+
+# Generate speech with the cloned voice
+audio = model.generate_speech(
+    text="This voice was cloned using automatic transcription.",
+    voice_profile=voice_profile,
+    output_path="auto_transcribed_voice.wav"
+)
+```
+
+### Flexible Input Formats
+
+The module supports various input formats:
+
+```python
+# From file path
+voice_profile1 = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="Text transcript."
+)
+
+# From bytes
+with open("sample_voice.wav", "rb") as f:
+    audio_bytes = f.read()
+voice_profile2 = model.clone_voice(
+    audio_input=audio_bytes,
+    transcript="Text transcript."
+)
+
+# From file object
+with open("sample_voice.wav", "rb") as f:
+    voice_profile3 = model.clone_voice(
+        audio_input=f,
+        transcript="Text transcript."
+    )
+
+# From tensor
+import torch
+import torchaudio
+audio_tensor, sample_rate = torchaudio.load("sample_voice.wav")
+voice_profile4 = model.clone_voice(
+    audio_input=audio_tensor,
+    transcript="Text transcript."
+)
+```
+
+## Configuration
+
+You can configure the default device by setting the `DFLT_VOXY_DEVICE` environment variable:
+
+```bash
+# Use CUDA
+export DFLT_VOXY_DEVICE=cuda
+
+# Use CPU
+export DFLT_VOXY_DEVICE=cpu
+
+# Use MPS (Apple Silicon)
+export DFLT_VOXY_DEVICE=mps
+```
+
+## Advanced Usage
+
+### Audio Cleanup
+
+The module includes an audio cleanup function that normalizes volume and removes silence:
+
+```python
+from voxy import cleanup_audio
+import torchaudio
+
+# Load audio
+audio, sample_rate = torchaudio.load("noisy_audio.wav")
+
+# Clean up audio
+cleaned_audio = cleanup_audio(
+    audio=audio,
+    sample_rate=sample_rate,
+    normalize=True,
+    remove_silence=True,
+    silence_threshold=0.02,
+    min_silence_duration=0.2
+)
+
+# Save cleaned audio
+torchaudio.save("cleaned_audio.wav", cleaned_audio, sample_rate)
+```
+
+### Disabling Audio Cleanup
+
+You can disable audio cleanup when cloning a voice:
+
+```python
+voice_profile = model.clone_voice(
+    audio_input="sample_voice.wav",
+    transcript="This is a sample of my voice.",
+    cleanup_audio_fn=None  # Disable audio cleanup
+)
+```
+
+### Custom Audio Cleanup
+
+You can also provide your own audio cleanup function:
+
+```python
+def my_custom_cleanup(audio, sample_rate, **kwargs):
+    # Custom cleanup logic
+    return processed_audio
+
+voice_profile = model.

voxy-0.0.2/voxy.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+setup.cfg
+setup.py
+voxy/__init__.py
+voxy/base.py
+voxy.egg-info/PKG-INFO
+voxy.egg-info/SOURCES.txt
+voxy.egg-info/dependency_links.txt
+voxy.egg-info/not-zip-safe
+voxy.egg-info/top_level.txt

voxy-0.0.2/voxy.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

voxy-0.0.2/voxy.egg-info/not-zip-safe

@@ -0,0 +1 @@
+

voxy-0.0.2/voxy.egg-info/top_level.txt

@@ -0,0 +1 @@
+voxy