mlx-speech 0.1.0__tar.gz

mlx_speech-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 AppAutomaton swarm of agents
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mlx_speech-0.1.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.3
+Name: mlx-speech
+Version: 0.1.0
+Summary: MLX-native speech library for Apple Silicon.
+Author: AppAutomaton
+License: MIT License
+         
+         Copyright (c) 2026 AppAutomaton swarm of agents
+         
+         Permission is hereby granted, free of charge, to any person obtaining a copy
+         of this software and associated documentation files (the "Software"), to deal
+         in the Software without restriction, including without limitation the rights
+         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+         copies of the Software, and to permit persons to whom the Software is
+         furnished to do so, subject to the following conditions:
+         
+         The above copyright notice and this permission notice shall be included in all
+         copies or substantial portions of the Software.
+         
+         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+         SOFTWARE.
+Requires-Dist: mlx>=0.31.1
+Requires-Dist: numpy
+Requires-Dist: safetensors
+Requires-Dist: soundfile>=0.13.1
+Requires-Dist: tokenizers>=0.22.2
+Requires-Python: >=3.13
+Project-URL: Repository, https://github.com/appautomaton/mlx-speech
+Description-Content-Type: text/markdown
+
+# mlx-speech
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
+[![Platform](https://img.shields.io/badge/platform-Apple%20Silicon-black?logo=apple)](https://developer.apple.com/documentation/apple-silicon)
+
+Local speech synthesis on Apple Silicon, running pure MLX. No cloud, no PyTorch.
+
+| Model | Best for |
+| --- | --- |
+| MossTTSLocal | shorter TTS, voice cloning, continuation |
+| MOSS-TTSD | multi-speaker dialogue |
+| MOSS-SoundEffect | text-to-sound-effect |
+| VibeVoice | long-form speech, voice-conditioned generation |
+
+## Requirements
+
+- Apple Silicon Mac (M1 or later)
+- Python 3.13+
+- [uv](https://docs.astral.sh/uv/)
+
+## Installation
+
+```bash
+git clone https://github.com/appautomaton/mlx-speech.git
+cd mlx-speech
+uv sync
+```
+
+> PyPI package (`pip install mlx-speech`) coming soon.
+
+Convert the checkpoints you want to use — each model family has a `scripts/convert_*.py` entry point:
+
+```bash
+python scripts/convert_moss_local.py
+python scripts/convert_moss_audio_tokenizer.py
+python scripts/convert_moss_ttsd.py
+python scripts/convert_moss_sound_effect.py
+python scripts/convert_vibevoice.py
+```
+
+## Usage
+
+**Generate speech:**
+
+```bash
+python scripts/generate_moss_local.py \
+  --text "Hello, this is a test." \
+  --output outputs/moss_local.wav
+```
+
+**Clone a voice:**
+
+```bash
+python scripts/generate_moss_local.py \
+  --mode clone \
+  --text "Hello, this is a cloned sample." \
+  --reference-audio reference.wav \
+  --output outputs/moss_local_clone.wav
+```
+
+**Multi-speaker dialogue:**
+
+```bash
+python scripts/generate_moss_ttsd.py \
+  --text "[S1] Watson, we should go now." \
+  --output outputs/ttsd.wav
+```
+
+**Sound effect:**
+
+```bash
+python scripts/generate_moss_sound_effect.py \
+  --ambient-sound "rolling thunder with steady rainfall on a metal roof" \
+  --duration-seconds 8 \
+  --output outputs/thunder.wav
+```
+
+**VibeVoice:**
+
+```bash
+python scripts/generate_vibevoice.py \
+  --text "Hello from VibeVoice." \
+  --output outputs/vibevoice.wav
+```
+
+## Exploring the Codebase
+
+The PyPI package is still in progress. The best way to explore right now is to drop the repo into an agentic coding tool like [Claude Code](https://claude.ai/code) or Cursor — the codebase is structured and self-describing, and an agent can walk you through it quickly.
+
+## Model Guides
+
+Each family has a doc covering behavior, flags, and known limitations:
+
+- [MossTTSLocal](./docs/moss-local.md)
+- [MOSS-TTSD](./docs/moss-ttsd.md)
+- [MOSS-SoundEffect](./docs/moss-sound-effect.md)
+- [VibeVoice](./docs/vibevoice.md)
+
+## Development
+
+```bash
+uv run pytest
+uv run ruff check .
+uv build --no-sources
+```
+
+```text
+mlx-speech/
+  src/mlx_speech/    library code
+  scripts/          conversion and generation entry points
+  models/           local checkpoints (not in git)
+  tests/            unit and integration tests
+  docs/             model-family behavior guides
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE)

mlx_speech-0.1.0/README.md

@@ -0,0 +1,119 @@
+# mlx-speech
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
+[![Platform](https://img.shields.io/badge/platform-Apple%20Silicon-black?logo=apple)](https://developer.apple.com/documentation/apple-silicon)
+
+Local speech synthesis on Apple Silicon, running pure MLX. No cloud, no PyTorch.
+
+| Model | Best for |
+| --- | --- |
+| MossTTSLocal | shorter TTS, voice cloning, continuation |
+| MOSS-TTSD | multi-speaker dialogue |
+| MOSS-SoundEffect | text-to-sound-effect |
+| VibeVoice | long-form speech, voice-conditioned generation |
+
+## Requirements
+
+- Apple Silicon Mac (M1 or later)
+- Python 3.13+
+- [uv](https://docs.astral.sh/uv/)
+
+## Installation
+
+```bash
+git clone https://github.com/appautomaton/mlx-speech.git
+cd mlx-speech
+uv sync
+```
+
+> PyPI package (`pip install mlx-speech`) coming soon.
+
+Convert the checkpoints you want to use — each model family has a `scripts/convert_*.py` entry point:
+
+```bash
+python scripts/convert_moss_local.py
+python scripts/convert_moss_audio_tokenizer.py
+python scripts/convert_moss_ttsd.py
+python scripts/convert_moss_sound_effect.py
+python scripts/convert_vibevoice.py
+```
+
+## Usage
+
+**Generate speech:**
+
+```bash
+python scripts/generate_moss_local.py \
+  --text "Hello, this is a test." \
+  --output outputs/moss_local.wav
+```
+
+**Clone a voice:**
+
+```bash
+python scripts/generate_moss_local.py \
+  --mode clone \
+  --text "Hello, this is a cloned sample." \
+  --reference-audio reference.wav \
+  --output outputs/moss_local_clone.wav
+```
+
+**Multi-speaker dialogue:**
+
+```bash
+python scripts/generate_moss_ttsd.py \
+  --text "[S1] Watson, we should go now." \
+  --output outputs/ttsd.wav
+```
+
+**Sound effect:**
+
+```bash
+python scripts/generate_moss_sound_effect.py \
+  --ambient-sound "rolling thunder with steady rainfall on a metal roof" \
+  --duration-seconds 8 \
+  --output outputs/thunder.wav
+```
+
+**VibeVoice:**
+
+```bash
+python scripts/generate_vibevoice.py \
+  --text "Hello from VibeVoice." \
+  --output outputs/vibevoice.wav
+```
+
+## Exploring the Codebase
+
+The PyPI package is still in progress. The best way to explore right now is to drop the repo into an agentic coding tool like [Claude Code](https://claude.ai/code) or Cursor — the codebase is structured and self-describing, and an agent can walk you through it quickly.
+
+## Model Guides
+
+Each family has a doc covering behavior, flags, and known limitations:
+
+- [MossTTSLocal](./docs/moss-local.md)
+- [MOSS-TTSD](./docs/moss-ttsd.md)
+- [MOSS-SoundEffect](./docs/moss-sound-effect.md)
+- [VibeVoice](./docs/vibevoice.md)
+
+## Development
+
+```bash
+uv run pytest
+uv run ruff check .
+uv build --no-sources
+```
+
+```text
+mlx-speech/
+  src/mlx_speech/    library code
+  scripts/          conversion and generation entry points
+  models/           local checkpoints (not in git)
+  tests/            unit and integration tests
+  docs/             model-family behavior guides
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE)

mlx_speech-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[project]
+name = "mlx-speech"
+version = "0.1.0"
+description = "MLX-native speech library for Apple Silicon."
+license = { file = "LICENSE" }
+authors = [{ name = "AppAutomaton" }]
+urls = { Repository = "https://github.com/appautomaton/mlx-speech" }
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "mlx>=0.31.1",
+    "numpy",
+    "safetensors",
+    "soundfile>=0.13.1",
+    "tokenizers>=0.22.2",
+]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.3,<9",
+    "ruff>=0.11,<0.12",
+]
+
+[build-system]
+requires = ["uv_build>=0.11.2,<0.12"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "local_integration: test requires local model artifacts or repo-specific runtime assets",
+]
+
+[tool.ruff]
+target-version = "py313"
+exclude = [".references", ".venv"]

mlx_speech-0.1.0/src/mlx_speech/__init__.py

@@ -0,0 +1,5 @@
+"""Top-level package for mlx-voice."""
+
+__all__ = ["__version__"]
+
+__version__ = "0.1.0"

mlx_speech-0.1.0/src/mlx_speech/audio/__init__.py

@@ -0,0 +1,21 @@
+"""Audio utilities for mlx-voice."""
+
+from .io import (
+    load_audio,
+    loudness_normalize,
+    mix_down_mono,
+    normalize_peak,
+    resample_audio,
+    trim_leading_silence,
+    write_wav,
+)
+
+__all__ = [
+    "load_audio",
+    "loudness_normalize",
+    "mix_down_mono",
+    "normalize_peak",
+    "resample_audio",
+    "trim_leading_silence",
+    "write_wav",
+]

mlx_speech-0.1.0/src/mlx_speech/audio/io.py

@@ -0,0 +1,179 @@
+"""Small audio I/O helpers for local v0 validation."""
+
+from __future__ import annotations
+
+import wave
+from pathlib import Path
+
+import numpy as np
+import mlx.core as mx
+
+try:
+    import soundfile as sf
+except ModuleNotFoundError:  # pragma: no cover - exercised only in lean envs
+    sf = None
+
+
+def mix_down_mono(samples: mx.array) -> mx.array:
+    """Convert audio shaped like (samples,) or (samples, channels) to mono."""
+
+    waveform = np.asarray(samples, dtype=np.float32)
+    if waveform.ndim == 1:
+        return mx.array(waveform, dtype=mx.float32)
+    if waveform.ndim != 2:
+        raise ValueError(
+            f"Expected waveform with shape (samples,) or (samples, channels), got {waveform.shape}."
+        )
+    return mx.array(np.mean(waveform, axis=1, dtype=np.float32), dtype=mx.float32)
+
+
+def resample_audio(
+    samples: mx.array,
+    *,
+    orig_sample_rate: int,
+    target_sample_rate: int,
+) -> mx.array:
+    """Resample mono audio with linear interpolation."""
+
+    if orig_sample_rate <= 0 or target_sample_rate <= 0:
+        raise ValueError("Sample rates must be positive.")
+    waveform = np.asarray(samples, dtype=np.float32)
+    if waveform.ndim != 1:
+        raise ValueError(f"Expected mono waveform with shape (samples,), got {waveform.shape}.")
+    if waveform.size == 0 or orig_sample_rate == target_sample_rate:
+        return mx.array(waveform, dtype=mx.float32)
+
+    duration = waveform.shape[0] / float(orig_sample_rate)
+    target_samples = max(1, int(round(duration * target_sample_rate)))
+    source_positions = np.linspace(0.0, 1.0, num=waveform.shape[0], endpoint=False, dtype=np.float32)
+    target_positions = np.linspace(0.0, 1.0, num=target_samples, endpoint=False, dtype=np.float32)
+    resampled = np.interp(target_positions, source_positions, waveform).astype(np.float32)
+    return mx.array(resampled, dtype=mx.float32)
+
+
+def loudness_normalize(
+    samples: mx.array,
+    *,
+    target_dbfs: float = -20.0,
+    gain_range: tuple[float, float] = (-3.0, 3.0),
+) -> mx.array:
+    """Apply a small loudness correction in dBFS."""
+
+    waveform = samples.astype(mx.float32)
+    if waveform.size == 0:
+        return waveform
+
+    power = float(mx.mean(waveform * waveform).item())
+    current_dbfs = 10.0 * np.log10(power + 1e-9)
+    gain = max(gain_range[0], min(target_dbfs - current_dbfs, gain_range[1]))
+    factor = 10.0 ** (gain / 20.0)
+    return waveform * factor
+
+
+def load_audio(
+    path: str | Path,
+    *,
+    sample_rate: int | None = None,
+    mono: bool = True,
+) -> tuple[mx.array, int]:
+    """Load local audio from disk."""
+
+    if sf is not None:
+        waveform, loaded_sample_rate = sf.read(str(path), always_2d=False, dtype="float32")
+        samples = mx.array(waveform, dtype=mx.float32)
+    else:
+        with wave.open(str(path), "rb") as wav_file:
+            loaded_sample_rate = wav_file.getframerate()
+            channels = wav_file.getnchannels()
+            sample_width = wav_file.getsampwidth()
+            frames = wav_file.readframes(wav_file.getnframes())
+        if sample_width != 2:
+            raise RuntimeError("WAV fallback supports 16-bit PCM only.")
+        waveform = np.frombuffer(frames, dtype=np.int16).astype(np.float32) / 32767.0
+        if channels > 1:
+            waveform = waveform.reshape(-1, channels)
+        samples = mx.array(waveform, dtype=mx.float32)
+    if mono:
+        samples = mix_down_mono(samples)
+    if sample_rate is not None and int(loaded_sample_rate) != int(sample_rate):
+        samples = resample_audio(
+            samples,
+            orig_sample_rate=int(loaded_sample_rate),
+            target_sample_rate=int(sample_rate),
+        )
+        loaded_sample_rate = int(sample_rate)
+    return samples, int(loaded_sample_rate)
+
+
+def trim_leading_silence(
+    samples: mx.array,
+    *,
+    sample_rate: int,
+    threshold: float = 0.003,
+    frame_ms: float = 20.0,
+    keep_ms: float = 80.0,
+) -> mx.array:
+    """Trim leading low-energy audio using a small RMS window."""
+
+    waveform = np.asarray(samples, dtype=np.float32)
+    if waveform.ndim != 1:
+        raise ValueError(f"Expected mono waveform with shape (samples,), got {waveform.shape}.")
+    if waveform.size == 0:
+        return samples
+
+    frame_size = max(1, int(sample_rate * frame_ms / 1000.0))
+    keep_samples = max(0, int(sample_rate * keep_ms / 1000.0))
+
+    start_index = 0
+    for idx in range(0, waveform.size, frame_size):
+        frame = waveform[idx : idx + frame_size]
+        if frame.size == 0:
+            break
+        rms = float(np.sqrt(np.mean(frame * frame)))
+        if rms >= threshold:
+            start_index = max(0, idx - keep_samples)
+            break
+    else:
+        return samples
+
+    return mx.array(waveform[start_index:], dtype=samples.dtype)
+
+
+def normalize_peak(
+    samples: mx.array,
+    *,
+    target_peak: float = 0.95,
+    max_gain: float = 4.0,
+) -> mx.array:
+    """Scale waveform so peak amplitude approaches `target_peak`."""
+
+    waveform = samples.astype(mx.float32)
+    peak = float(mx.max(mx.abs(waveform)).item()) if waveform.size > 0 else 0.0
+    if peak <= 0.0:
+        return waveform
+
+    gain = min(max_gain, target_peak / peak)
+    if gain <= 1.0:
+        return waveform
+    return waveform * gain
+
+
+def write_wav(path: str | Path, samples: mx.array, *, sample_rate: int) -> Path:
+    """Write a mono float waveform to 16-bit PCM WAV."""
+
+    output_path = Path(path)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    waveform = np.asarray(samples, dtype=np.float32)
+    if waveform.ndim != 1:
+        raise ValueError(f"Expected mono waveform with shape (samples,), got {waveform.shape}.")
+    waveform = np.clip(waveform, -1.0, 1.0)
+    pcm16 = (waveform * 32767.0).astype(np.int16)
+
+    with wave.open(str(output_path), "wb") as wav_file:
+        wav_file.setnchannels(1)
+        wav_file.setsampwidth(2)
+        wav_file.setframerate(sample_rate)
+        wav_file.writeframes(pcm16.tobytes())
+
+    return output_path

mlx_speech-0.1.0/src/mlx_speech/checkpoints/__init__.py

@@ -0,0 +1,21 @@
+"""Checkpoint loading and remapping helpers for mlx-voice."""
+
+from .layout import ModelArtifactLayout, OpenMossV0Layouts, get_openmoss_v0_layouts
+from .sharded import (
+    INDEX_FILENAME,
+    LoadedStateDict,
+    ShardedCheckpointIndex,
+    load_state_dict,
+    summarize_prefixes,
+)
+
+__all__ = [
+    "INDEX_FILENAME",
+    "LoadedStateDict",
+    "ModelArtifactLayout",
+    "OpenMossV0Layouts",
+    "ShardedCheckpointIndex",
+    "get_openmoss_v0_layouts",
+    "load_state_dict",
+    "summarize_prefixes",
+]

mlx_speech-0.1.0/src/mlx_speech/checkpoints/layout.py

@@ -0,0 +1,85 @@
+"""Local checkpoint layout helpers."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MODELS_ROOT = REPO_ROOT / "models"
+
+
+@dataclass(frozen=True)
+class ModelArtifactLayout:
+    """Filesystem layout for one model family."""
+
+    family: str
+    model_name: str
+    repo_id: str
+    root_dir: Path
+    original_dir: Path
+    mlx_int8_dir: Path
+
+    def ensure(self) -> "ModelArtifactLayout":
+        self.original_dir.mkdir(parents=True, exist_ok=True)
+        self.mlx_int8_dir.mkdir(parents=True, exist_ok=True)
+        return self
+
+
+@dataclass(frozen=True)
+class OpenMossV0Layouts:
+    """Grouped layout for the v0 OpenMOSS assets."""
+
+    moss_tts_local: ModelArtifactLayout
+    audio_tokenizer: ModelArtifactLayout
+    moss_sound_effect: ModelArtifactLayout
+
+    def ensure(self) -> "OpenMossV0Layouts":
+        self.moss_tts_local.ensure()
+        self.audio_tokenizer.ensure()
+        self.moss_sound_effect.ensure()
+        return self
+
+
+def _build_model_layout(
+    models_root: Path,
+    family: str,
+    model_name: str,
+    repo_id: str,
+) -> ModelArtifactLayout:
+    root_dir = models_root / family / model_name
+    return ModelArtifactLayout(
+        family=family,
+        model_name=model_name,
+        repo_id=repo_id,
+        root_dir=root_dir,
+        original_dir=root_dir / "original",
+        mlx_int8_dir=root_dir / "mlx-int8",
+    )
+
+
+def get_openmoss_v0_layouts(models_root: Path | None = None) -> OpenMossV0Layouts:
+    """Return the local model layout used by the current v0 plan."""
+
+    resolved_root = MODELS_ROOT if models_root is None else Path(models_root)
+    return OpenMossV0Layouts(
+        moss_tts_local=_build_model_layout(
+            models_root=resolved_root,
+            family="openmoss",
+            model_name="moss_tts_local",
+            repo_id="OpenMOSS-Team/MOSS-TTS-Local-Transformer",
+        ),
+        audio_tokenizer=_build_model_layout(
+            models_root=resolved_root,
+            family="openmoss",
+            model_name="moss_audio_tokenizer",
+            repo_id="OpenMOSS-Team/MOSS-Audio-Tokenizer",
+        ),
+        moss_sound_effect=_build_model_layout(
+            models_root=resolved_root,
+            family="openmoss",
+            model_name="moss_sound_effect",
+            repo_id="OpenMOSS-Team/MOSS-SoundEffect",
+        ),
+    )