pytest-audioeval 0.1.0__tar.gz

pytest_audioeval-0.1.0/.gitignore

@@ -0,0 +1,207 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/

pytest_audioeval-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

pytest_audioeval-0.1.0/.vscode/settings.json

@@ -0,0 +1,9 @@
+{
+    "python.testing.pytestArgs": [
+        "tests",
+        "--stt-url=ws://localhost:45120",
+        "--tts-url=http://localhost:45130/v1/audio/speech"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true
+}

pytest_audioeval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Damien Volkov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytest_audioeval-0.1.0/Makefile

@@ -0,0 +1,68 @@
+.PHONY: install sync lint test test-unit test-integration coverage generate-samples docs docs-serve docs-deploy build publish publish-test infra-up infra-down infra-logs infra-status
+
+install:
+	@uv sync --dev
+
+sync:
+	@uv sync --dev
+
+lint:
+	@uv run ruff check --fix src/ tests/
+	@uv run ruff format src/ tests/
+
+test:
+	@uv run pytest
+
+test-unit:
+	@uv run pytest tests/unit/
+
+test-integration:
+	@uv run pytest tests/integration/ --stt-url=ws://localhost:45120 --tts-url=http://localhost:45130/v1/audio/speech
+
+coverage:
+	@uv run coverage run -m pytest tests/unit/ -q --no-header
+	@uv run coverage report
+
+generate-samples:
+	@uv run python scripts/generate_samples.py
+
+##### DOCUMENTATION #####
+
+docs:
+	@uv run mkdocs build --strict
+
+docs-serve:
+	@uv run mkdocs serve
+
+docs-deploy:
+	@uv run mkdocs gh-deploy --force
+
+##### PUBLISHING #####
+
+build:
+	@rm -rf dist/
+	@uv build
+
+publish-test: lint test-unit build
+	@test -n "$(PYPI_API_TOKEN)" || (echo "ERROR: PYPI_API_TOKEN not set" && exit 1)
+	@uv publish --publish-url https://test.pypi.org/legacy/ --token $(PYPI_API_TOKEN)
+
+publish: lint test-unit build
+	@test -n "$(PYPI_API_TOKEN)" || (echo "ERROR: PYPI_API_TOKEN not set" && exit 1)
+	@uv publish --token $(PYPI_API_TOKEN)
+
+##### DOCKER — INTEGRATION SERVICES #####
+
+infra-up:
+	@docker compose -f compose.integration.yml up -d
+	@echo "Waiting for services to become healthy..."
+	@docker compose -f compose.integration.yml ps
+
+infra-down:
+	@docker compose -f compose.integration.yml down
+
+infra-logs:
+	@docker compose -f compose.integration.yml logs -f
+
+infra-status:
+	@docker compose -f compose.integration.yml ps

pytest_audioeval-0.1.0/PKG-INFO

@@ -0,0 +1,282 @@
+Metadata-Version: 2.4
+Name: pytest-audioeval
+Version: 0.1.0
+Summary: Pytest plugin for STT/TTS integration testing with httpx, metrics, and embedded audio samples.
+Project-URL: Homepage, https://damvolkov.github.io/pytest-audioeval
+Project-URL: Documentation, https://damvolkov.github.io/pytest-audioeval
+Project-URL: Repository, https://github.com/damvolkov/pytest-audioeval
+Project-URL: Changelog, https://damvolkov.github.io/pytest-audioeval/changelog/
+Author-email: damvolkov <damvolkovv@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: audio,evaluation,metrics,pesq,pytest,speech,stt,testing,tts,wer
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.13
+Requires-Dist: httpx-sse>=0.4.3
+Requires-Dist: httpx-ws>=0.8.2
+Requires-Dist: httpx>=0.27
+Requires-Dist: jiwer>=3.0
+Requires-Dist: numpy>=2.0
+Requires-Dist: pesq>=0.0.4
+Requires-Dist: pytest>=8.0
+Requires-Dist: soundfile>=0.12
+Description-Content-Type: text/markdown
+
+# pytest-audioeval
+
+Pytest plugin for STT/TTS integration testing. Built on the httpx ecosystem (`httpx`, `httpx-ws`, `httpx-sse`) with built-in metrics, embedded ground-truth audio samples, and chainable assertions.
+
+## Features
+
+- **STT via WebSocket** — `audioeval.stt.ws()` streams audio, collects transcription
+- **TTS via HTTP** — `audioeval.tts.post()` batch, `.stream()` chunked, `.sse()` Server-Sent Events
+- **Text metrics** — WER, CER, substitutions, insertions, deletions (via `jiwer`)
+- **Audio metrics** — PESQ MOS 1–5 scale (via `pesq`)
+- **Embedded samples** — ground-truth audio + reference text pairs, multi-language ready
+- **Chainable assertions** — `result.compute_metrics(ref).assert_quality(max_wer=0.2)`
+- **CLI thresholds** — `--audioeval-wer`, `--audioeval-cer`, `--audioeval-mos`
+
+## Install
+
+```bash
+uv add pytest-audioeval
+```
+
+## Quick Start
+
+### STT — WebSocket
+
+```python
+import asyncio
+import uuid
+import orjson as json
+from pytest_audioeval.client import AudioEval
+
+
+async def test_user_stt_ws(audioeval: AudioEval) -> None:
+    sample = audioeval.samples.en_hello_world
+
+    async with audioeval.stt.ws(sample=sample) as session:
+        config = json.dumps(
+            {"uid": str(uuid.uuid4()), "language": "en", "task": "transcribe",
+             "model": "large-v3-turbo", "use_vad": True}
+        ).decode()
+        await session.send_text(config)
+
+        ready = await session.receive_text()
+        assert "SERVER_READY" in ready
+
+        await session.send_sample(sample, chunk_ms=200)
+        await asyncio.sleep(2)
+        await session.send_text("END_OF_AUDIO")
+
+        # Collect transcription segments...
+```
+
+### TTS — Batch POST
+
+```python
+import io
+import soundfile as sf
+from pytest_audioeval.client import AudioEval
+
+
+async def test_user_tts_batch(audioeval: AudioEval) -> None:
+    response = await audioeval.tts.post(
+        json={"input": "Hello world.", "model": "kokoro",
+              "voice": "af_heart", "response_format": "wav", "stream": False},
+    )
+    data, rate = sf.read(io.BytesIO(response.content), dtype="float32")
+    assert rate == 24_000
+    assert len(data) > 0
+```
+
+### TTS — Chunked Streaming
+
+```python
+async def test_user_tts_streaming(audioeval: AudioEval) -> None:
+    chunks = []
+    async with audioeval.tts.stream(json={"input": "Hello.", ...}) as response:
+        async for chunk in response.aiter_bytes():
+            chunks.append(chunk)
+    assert len(chunks) > 0
+```
+
+### TTS — Server-Sent Events
+
+```python
+async def test_user_tts_sse(audioeval: AudioEval) -> None:
+    async with audioeval.tts.sse(json={"input": "Hello.", ...}) as event_source:
+        async for sse in event_source.aiter_sse():
+            print(sse.data)
+```
+
+### Text Metrics
+
+```python
+from pytest_audioeval.metrics.text import TextMetrics
+
+
+async def test_user_metrics_text() -> None:
+    metrics = TextMetrics.compute(
+        reference="the quick brown fox jumps over the lazy dog",
+        hypothesis="the quick brown fox jumps over the lazy dock",
+    )
+    assert metrics.wer < 0.15
+    assert metrics.substitutions == 1
+```
+
+### STT Result — Chainable Assertions
+
+```python
+from pytest_audioeval.stt import STTResult
+
+
+async def test_user_stt_result() -> None:
+    result = STTResult(hypothesis_text="Hello world.")
+    result.compute_metrics("Hello world.")
+    result.assert_quality(max_wer=0.2, max_cer=0.15)
+```
+
+### Sample Registry
+
+```python
+from pytest_audioeval.samples.registry import SampleLang
+
+
+async def test_user_samples_browse(audioeval: AudioEval) -> None:
+    # All samples
+    assert len(audioeval.samples) >= 3
+
+    # Filter by language
+    en_samples = audioeval.samples.by_lang(SampleLang.EN)
+
+    # Attribute access: {lang}_{name}
+    sample = audioeval.samples.en_hello_world
+    assert sample.reference_text == "Hello world."
+
+    # Audio access
+    audio_f32 = sample.audio_numpy()        # numpy float32 array
+    audio_raw = sample.audio_bytes()         # raw bytes
+    chunks = sample.chunks(chunk_ms=200)     # chunked for streaming
+```
+
+### CLI Thresholds
+
+```python
+async def test_user_thresholds(audioeval_thresholds: dict[str, float]) -> None:
+    assert audioeval_thresholds["max_wer"] == 0.2
+    assert audioeval_thresholds["max_cer"] == 0.15
+    assert audioeval_thresholds["min_mos"] == 3.0
+```
+
+## CLI Options
+
+```bash
+pytest --stt-url=ws://localhost:45120 --tts-url=http://localhost:45130/v1/audio/speech
+pytest --audioeval-wer=0.15 --audioeval-cer=0.10 --audioeval-mos=3.5
+```
+
+| Option | Default | Description |
+|---|---|---|
+| `--stt-url` | `None` | STT service WebSocket URL |
+| `--tts-url` | `None` | TTS service HTTP URL |
+| `--audioeval-wer` | `0.2` | Max WER threshold |
+| `--audioeval-cer` | `0.15` | Max CER threshold |
+| `--audioeval-mos` | `3.0` | Min PESQ MOS threshold |
+
+## Fixtures
+
+| Fixture | Scope | Type | Description |
+|---|---|---|---|
+| `audioeval` | session | `AudioEval` | Main facade — `audioeval.stt`, `audioeval.tts`, `audioeval.samples` |
+| `audioeval_thresholds` | function | `dict[str, float]` | CLI-driven threshold dict |
+
+## Architecture
+
+```
+src/pytest_audioeval/
+├── plugin.py              # pytest entry point (fixtures, CLI options)
+├── client.py              # AudioEval facade
+├── stt.py                 # STTClient (httpx-ws), STTSession, STTResult
+├── tts.py                 # TTSClient (httpx + httpx-sse)
+├── metrics/
+│   ├── text.py            # TextMetrics — WER, CER via jiwer
+│   └── audio.py           # AudioMetrics — PESQ MOS via pesq
+└── samples/
+    ├── registry.py        # SampleRegistry + AudioSample + SampleLang
+    └── audio/en/          # Embedded ground-truth WAV + TXT pairs
+```
+
+### Clients
+
+| Client | Transport | Methods |
+|---|---|---|
+| `STTClient` | `httpx-ws` | `.ws()` — WebSocket context manager yielding `STTSession` |
+| `TTSClient` | `httpx` + `httpx-sse` | `.post()` batch, `.stream()` chunked, `.sse()` SSE |
+
+### Metrics
+
+| Metric | Class | Source | Range |
+|---|---|---|---|
+| Word Error Rate (WER) | `TextMetrics` | `jiwer` | 0.0 – 1.0+ |
+| Character Error Rate (CER) | `TextMetrics` | `jiwer` | 0.0 – 1.0+ |
+| Substitutions / Insertions / Deletions | `TextMetrics` | `jiwer` | 0 – N |
+| PESQ MOS | `AudioMetrics` | `pesq` | 1.0 – 5.0 |
+
+### Samples
+
+Embedded ground-truth audio with reference transcriptions:
+
+```
+samples/audio/
+└── en/                    # English (16kHz, float32)
+    ├── hello_world.wav    # "Hello world."
+    ├── quick_brown_fox.wav
+    └── counting.wav       # "One, two, three, four, five."
+```
+
+Access: `audioeval.samples.en_hello_world`, `audioeval.samples.en_counting`, etc.
+
+## Infrastructure
+
+Integration tests require GPU-accelerated TTS/STT services:
+
+```bash
+make infra-up       # Start TTS (Kokoro) + STT (WhisperLive)
+make infra-status   # Check health
+make infra-logs     # View logs
+make infra-down     # Stop services
+```
+
+| Service | Image | Port | Protocol |
+|---|---|---|---|
+| TTS (Kokoro) | `ghcr.io/remsky/kokoro-fastapi-gpu` | `45130` | HTTP |
+| STT (WhisperLive) | `ghcr.io/collabora/whisperlive-gpu` | `45120` | WebSocket |
+
+## Development
+
+```bash
+make install            # uv sync --dev
+make lint               # ruff check + format
+make test-unit          # unit tests (no services)
+make test-integration   # integration tests (requires services)
+make coverage           # coverage report (>90%)
+```
+
+## Requirements
+
+- Python >= 3.13
+- NVIDIA GPU + Docker with nvidia-container-toolkit (for integration tests)
+
+## License
+
+MIT