gbx-codecs 0.1.0__tar.gz

gbx_codecs-0.1.0/.github/workflows/test.yml

@@ -0,0 +1,16 @@
+name: test
+on:
+  push:
+    branches: [main]
+  pull_request:
+jobs:
+  test:
+    runs-on: macos-14
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install ".[test]" "ruff==0.17.3"
+      - run: ruff check . && ruff format --check .
+      - run: pytest -q tests

gbx_codecs-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.egg-info/
+dist/
+build/
+.venv/
+.pytest_cache/

gbx_codecs-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 GREENBITAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gbx_codecs-0.1.0/PKG-INFO

@@ -0,0 +1,73 @@
+Metadata-Version: 2.5
+Name: gbx-codecs
+Version: 0.1.0
+Summary: Audio codecs for speech models served by gbx-lm, run as a companion process on Apple silicon
+Project-URL: Homepage, https://github.com/GreenBitAI/gbx-codecs
+Author: GreenBitAI
+License-Expression: MIT
+License-File: LICENSE
+License-File: licenses/LICENSE.mlx-audio.txt
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Requires-Python: >=3.10
+Requires-Dist: miniaudio>=1.59
+Requires-Dist: mlx>=0.29
+Requires-Dist: numpy>=1.24
+Provides-Extra: test
+Requires-Dist: pytest; extra == 'test'
+Description-Content-Type: text/markdown
+
+# gbx-codecs
+
+Audio codecs for the speech models that [gbx-lm](https://github.com/GreenBitAI/gbx-lm) serves, on Apple silicon.
+
+A speech model generated by gbx-lm produces **codes**, not audio. This package turns codes into audio, and reference audio into codes. It runs as a **separate process** that gbx-lm starts and talks to over a pipe, so the server itself needs none of these dependencies, and the codec can be installed, upgraded or removed without touching gbx-lm.
+
+| codec | used by | rate | codebooks |
+|---|---|---|---|
+| `fish_s1` (Fish Audio S1 DAC) | Fish Audio Drama 3 | 44.1 kHz mono, 21.5 frames/s | 10 |
+
+The codec weights come with the model repository; this package holds code only.
+
+## Install
+
+Python 3.10 or newer on an Apple silicon Mac. A virtual environment of its own is recommended:
+
+```bash
+python3 -m venv ~/.gbx-codecs
+~/.gbx-codecs/bin/pip install gbx-codecs
+```
+
+Dependencies: `mlx`, `numpy`, and `miniaudio` (to read reference audio in wav, flac or mp3). No PyTorch and no ffmpeg. The process needs about 3 GB of memory while decoding (Fish S1: 1.75 GB of weights, decoded 8 frames at a time).
+
+Check it against a downloaded model:
+
+```bash
+~/.gbx-codecs/bin/gbx-codecs check /path/to/the/model
+# codec OK: 10 codebooks, 44 frames for 2 s, 44100 Hz, round-trip correlation 0.9..
+```
+
+Then tell gbx-lm which Python to start it with (`--codec_python ~/.gbx-codecs/bin/python`, or `GBX__SERVER__CODEC_PYTHON`). See the model card of the speech model for the full setup.
+
+## Command line
+
+```bash
+gbx-codecs encode MODEL_DIR reference.wav codes.npy   # audio -> int32 codes [codebooks, frames]
+gbx-codecs decode MODEL_DIR codes.npy out.wav          # codes -> 16-bit wav
+gbx-codecs serve                                       # the companion process gbx-lm starts
+```
+
+## Protocol
+
+`gbx-codecs serve` reads requests on stdin and writes replies on stdout; logs go to stderr. Each message is `GBXC`, a 4-byte header length and an 8-byte payload length (big-endian), a UTF-8 JSON header, then the payload. Operations: `hello`, `load` (codec, model directory), `encode` (audio file to int32 codes), `begin` (output sample rate), `decode` (int32 codes to 16-bit PCM, streamed), `end`, `quit`. See `src/gbx_codecs/protocol.py` and `src/gbx_codecs/worker.py`.
+
+Decoding is streaming: frames sent a piece at a time decode to the same audio whatever the size of the pieces, to within one 16-bit step, and resampling (for example to 24 kHz for OpenAI-style `pcm` output) carries its state across pieces.
+
+## License
+
+MIT, see [LICENSE](LICENSE).
+
+The Fish Audio S1 DAC implementation in `src/gbx_codecs/fish_s1/dac.py` is ported from [mlx-audio](https://github.com/Blaizzy/mlx-audio) (MIT License, Copyright (c) 2024 Prince Canuma; see [licenses/LICENSE.mlx-audio.txt](licenses/LICENSE.mlx-audio.txt)). The codec weights are distributed with the models under the model's own license; this package does not include them.
+
+Not affiliated with or endorsed by Fish Audio.

gbx_codecs-0.1.0/README.md

@@ -0,0 +1,53 @@
+# gbx-codecs
+
+Audio codecs for the speech models that [gbx-lm](https://github.com/GreenBitAI/gbx-lm) serves, on Apple silicon.
+
+A speech model generated by gbx-lm produces **codes**, not audio. This package turns codes into audio, and reference audio into codes. It runs as a **separate process** that gbx-lm starts and talks to over a pipe, so the server itself needs none of these dependencies, and the codec can be installed, upgraded or removed without touching gbx-lm.
+
+| codec | used by | rate | codebooks |
+|---|---|---|---|
+| `fish_s1` (Fish Audio S1 DAC) | Fish Audio Drama 3 | 44.1 kHz mono, 21.5 frames/s | 10 |
+
+The codec weights come with the model repository; this package holds code only.
+
+## Install
+
+Python 3.10 or newer on an Apple silicon Mac. A virtual environment of its own is recommended:
+
+```bash
+python3 -m venv ~/.gbx-codecs
+~/.gbx-codecs/bin/pip install gbx-codecs
+```
+
+Dependencies: `mlx`, `numpy`, and `miniaudio` (to read reference audio in wav, flac or mp3). No PyTorch and no ffmpeg. The process needs about 3 GB of memory while decoding (Fish S1: 1.75 GB of weights, decoded 8 frames at a time).
+
+Check it against a downloaded model:
+
+```bash
+~/.gbx-codecs/bin/gbx-codecs check /path/to/the/model
+# codec OK: 10 codebooks, 44 frames for 2 s, 44100 Hz, round-trip correlation 0.9..
+```
+
+Then tell gbx-lm which Python to start it with (`--codec_python ~/.gbx-codecs/bin/python`, or `GBX__SERVER__CODEC_PYTHON`). See the model card of the speech model for the full setup.
+
+## Command line
+
+```bash
+gbx-codecs encode MODEL_DIR reference.wav codes.npy   # audio -> int32 codes [codebooks, frames]
+gbx-codecs decode MODEL_DIR codes.npy out.wav          # codes -> 16-bit wav
+gbx-codecs serve                                       # the companion process gbx-lm starts
+```
+
+## Protocol
+
+`gbx-codecs serve` reads requests on stdin and writes replies on stdout; logs go to stderr. Each message is `GBXC`, a 4-byte header length and an 8-byte payload length (big-endian), a UTF-8 JSON header, then the payload. Operations: `hello`, `load` (codec, model directory), `encode` (audio file to int32 codes), `begin` (output sample rate), `decode` (int32 codes to 16-bit PCM, streamed), `end`, `quit`. See `src/gbx_codecs/protocol.py` and `src/gbx_codecs/worker.py`.
+
+Decoding is streaming: frames sent a piece at a time decode to the same audio whatever the size of the pieces, to within one 16-bit step, and resampling (for example to 24 kHz for OpenAI-style `pcm` output) carries its state across pieces.
+
+## License
+
+MIT, see [LICENSE](LICENSE).
+
+The Fish Audio S1 DAC implementation in `src/gbx_codecs/fish_s1/dac.py` is ported from [mlx-audio](https://github.com/Blaizzy/mlx-audio) (MIT License, Copyright (c) 2024 Prince Canuma; see [licenses/LICENSE.mlx-audio.txt](licenses/LICENSE.mlx-audio.txt)). The codec weights are distributed with the models under the model's own license; this package does not include them.
+
+Not affiliated with or endorsed by Fish Audio.

gbx_codecs-0.1.0/licenses/LICENSE.mlx-audio.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Prince Canuma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gbx_codecs-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "gbx-codecs"
+version = "0.1.0"
+description = "Audio codecs for speech models served by gbx-lm, run as a companion process on Apple silicon"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE", "licenses/*"]
+requires-python = ">=3.10"
+authors = [{ name = "GreenBitAI" }]
+dependencies = ["mlx>=0.29", "numpy>=1.24", "miniaudio>=1.59"]
+classifiers = [
+    "Operating System :: MacOS",
+    "Programming Language :: Python :: 3",
+    "Topic :: Multimedia :: Sound/Audio :: Speech",
+]
+
+[project.urls]
+Homepage = "https://github.com/GreenBitAI/gbx-codecs"
+
+[project.scripts]
+gbx-codecs = "gbx_codecs.__main__:main"
+
+[project.optional-dependencies]
+test = ["pytest"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/gbx_codecs"]
+
+[tool.ruff]
+line-length = 110
+# Ported as upstream wrote it, so the two can be diffed.
+extend-exclude = ["src/gbx_codecs/fish_s1/dac.py"]
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "I", "B", "UP"]

gbx_codecs-0.1.0/src/gbx_codecs/__init__.py

@@ -0,0 +1,8 @@
+"""Audio codecs for models served by gbx-lm, run beside the server.
+
+gbx-lm generates codes; this package turns codes into audio and audio into
+codes, in a process of its own (``python -m gbx_codecs serve``) so the server
+needs none of its dependencies.
+"""
+
+__version__ = "0.1.0"

gbx_codecs-0.1.0/src/gbx_codecs/__main__.py

@@ -0,0 +1,98 @@
+"""gbx-codecs command line.
+
+python -m gbx_codecs serve                       # the process gbx-lm starts
+python -m gbx_codecs check MODEL_DIR             # load the codec, round-trip a tone
+python -m gbx_codecs encode MODEL_DIR IN.wav OUT.npy
+python -m gbx_codecs decode MODEL_DIR IN.npy OUT.wav
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+import wave
+
+import numpy as np
+
+from .worker import CACHE_LIMIT, MAX_DECODE_FRAMES
+
+
+def _codec(model_dir: str):
+    from .fish_s1 import DAC
+
+    return DAC.from_pretrained(model_dir)
+
+
+def _decode(codec, codes: np.ndarray) -> np.ndarray:
+    import mlx.core as mx
+
+    codec.reset_streaming_state()
+    parts = [
+        np.array(
+            codec.streaming_step(mx.array(codes[:, b : b + MAX_DECODE_FRAMES])[None])[0, 0].astype(mx.float32)
+        )
+        for b in range(0, codes.shape[1], MAX_DECODE_FRAMES)
+    ]
+    return np.concatenate(parts) if parts else np.zeros(0, np.float32)
+
+
+def _write_wav(path: str, samples: np.ndarray, rate: int) -> None:
+    from .audio import to_pcm16
+
+    with wave.open(path, "wb") as w:
+        w.setnchannels(1)
+        w.setsampwidth(2)
+        w.setframerate(rate)
+        w.writeframes(to_pcm16(samples))
+
+
+def main(argv: list[str] | None = None) -> None:
+    ap = argparse.ArgumentParser(prog="python -m gbx_codecs")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    sub.add_parser("serve", help="speak the codec protocol on stdin/stdout")
+    c = sub.add_parser("check", help="load the codec and round-trip a test tone")
+    c.add_argument("model_dir")
+    e = sub.add_parser("encode", help="audio file to codes")
+    e.add_argument("model_dir")
+    e.add_argument("audio")
+    e.add_argument("out")
+    d = sub.add_parser("decode", help="codes to a wav file")
+    d.add_argument("model_dir")
+    d.add_argument("codes")
+    d.add_argument("out")
+    a = ap.parse_args(argv)
+
+    if a.cmd == "serve":
+        from .worker import main as serve
+
+        serve()
+        return
+    import mlx.core as mx
+
+    mx.set_cache_limit(CACHE_LIMIT)
+    codec = _codec(a.model_dir)
+    if a.cmd == "check":
+        rate = codec.sample_rate
+        t = np.arange(rate * 2) / rate
+        tone = (0.3 * np.sin(2 * np.pi * 220 * t)).astype(np.float32)
+        idx, lengths = codec.encode(mx.array(tone)[None, None, :])
+        codes = np.array(idx[0, :, : int(lengths[0].item())]).astype(np.int32)
+        back = _decode(codec, codes)[: len(tone)]
+        corr = float(np.corrcoef(back, tone[: len(back)])[0, 1])
+        print(
+            f"codec OK: {codes.shape[0]} codebooks, {codes.shape[1]} frames for 2 s, "
+            f"{rate} Hz, round-trip correlation {corr:.3f}"
+        )
+        sys.exit(0 if corr > 0.5 else 1)
+    if a.cmd == "encode":
+        from .audio import read_audio
+
+        audio = read_audio(a.audio, codec.sample_rate)
+        idx, lengths = codec.encode(mx.array(audio)[None, None, :])
+        np.save(a.out, np.array(idx[0, :, : int(lengths[0].item())]).astype(np.int32))
+    elif a.cmd == "decode":
+        _write_wav(a.out, _decode(codec, np.load(a.codes).astype(np.int32)), codec.sample_rate)
+
+
+if __name__ == "__main__":
+    main()

gbx_codecs-0.1.0/src/gbx_codecs/audio.py

@@ -0,0 +1,88 @@
+"""Reading reference audio, resampling, and 16-bit PCM."""
+
+from __future__ import annotations
+
+import math
+from pathlib import Path
+
+import numpy as np
+
+
+def read_audio(path: str | Path, sample_rate: int) -> np.ndarray:
+    """Mono float32 samples of an audio file (wav, flac, mp3, ...) at ``sample_rate``."""
+    import miniaudio
+
+    decoded = miniaudio.decode_file(
+        str(path),
+        output_format=miniaudio.SampleFormat.FLOAT32,
+        nchannels=1,
+        sample_rate=sample_rate,
+    )
+    return np.frombuffer(decoded.samples, dtype=np.float32).copy()
+
+
+def to_pcm16(samples: np.ndarray) -> bytes:
+    """Little-endian signed 16-bit PCM."""
+    return (np.clip(samples, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()
+
+
+class Resampler:
+    """Streaming band-limited resampling by a rational factor.
+
+    Windowed-sinc interpolation with a Kaiser window, computed for every output
+    phase once. Feeding a signal in pieces gives exactly what feeding it whole
+    does: the input the next piece's outputs still need is carried over.
+    """
+
+    def __init__(self, source_rate: int, target_rate: int, half_width: int = 16, beta: float = 8.6):
+        g = math.gcd(source_rate, target_rate)
+        self.up, self.down = target_rate // g, source_rate // g
+        self.identity = self.up == self.down
+        self.half_width = half_width
+        cutoff = min(1.0, self.up / self.down) * 0.95
+        # taps[p, j]: weight of input (base + j - half_width + 1) for an output at
+        # fractional input position base + p / up.
+        offsets = (
+            np.arange(-half_width + 1, half_width + 1)[None, :] - (np.arange(self.up) / self.up)[:, None]
+        )
+        window = np.kaiser(2 * half_width + 1, beta)
+        x = offsets / half_width
+        w = np.interp(x, np.linspace(-1, 1, 2 * half_width + 1), window)
+        self.taps = (cutoff * np.sinc(cutoff * offsets) * w).astype(np.float32)
+        self.reset()
+
+    def reset(self) -> None:
+        # Leading zeros stand for the signal before the first sample.
+        self._buffer = np.zeros(self.half_width - 1, np.float32)
+        self._consumed = 0  # input samples dropped from the buffer's front
+        self._next = 0  # index of the next output sample
+
+    def __call__(self, samples: np.ndarray, final: bool = False) -> np.ndarray:
+        if self.identity:
+            return np.asarray(samples, np.float32)
+        buf = np.concatenate([self._buffer, np.asarray(samples, np.float32)])
+        if final:
+            buf = np.concatenate([buf, np.zeros(self.half_width, np.float32)])
+        lead = self.half_width - 1
+        available = self._consumed + len(buf) - lead - self.half_width  # last usable centre
+        # The last output whose centre, floor(n * down / up), is below `available`.
+        last = (available * self.up - 1) // self.down if available > 0 else -1
+        if final:
+            total_in = self._consumed + len(buf) - lead - self.half_width
+            last = min(last, math.ceil(total_in * self.up / self.down) - 1)
+        n = np.arange(self._next, last + 1)
+        if len(n) == 0:
+            out = np.zeros(0, np.float32)
+        else:
+            pos = n * self.down
+            base = pos // self.up - self._consumed + lead  # index into buf of the centre
+            phase = pos % self.up
+            idx = base[:, None] + np.arange(-self.half_width + 1, self.half_width + 1)[None, :]
+            out = (buf[idx] * self.taps[phase]).sum(axis=1).astype(np.float32)
+            self._next = int(last) + 1
+        # Keep what the next output still reads.
+        keep_from = (self._next * self.down) // self.up - self._consumed + lead - (self.half_width - 1)
+        keep_from = max(0, min(int(keep_from), len(buf)))
+        self._buffer = buf[keep_from:]
+        self._consumed += keep_from
+        return out

gbx_codecs-0.1.0/src/gbx_codecs/fish_s1/__init__.py

@@ -0,0 +1,5 @@
+"""Fish Audio's S1 DAC: 10 codebooks at 21.5 frames/s, 44.1 kHz mono."""
+
+from .dac import DAC
+
+__all__ = ["DAC"]