OpenSTBench 0.3.3__tar.gz → 1.1.0__tar.gz

openstbench-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yanjie An
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

openstbench-1.1.0/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.4
+Name: OpenSTBench
+Version: 1.1.0
+Summary: Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings
+Author-email: Yanjie An <691476922@qq.com>
+License: MIT
+Project-URL: Homepage, https://github.com/sjtuayj/OpenSTBench
+Project-URL: Repository, https://github.com/sjtuayj/OpenSTBench
+Project-URL: Documentation, https://github.com/sjtuayj/OpenSTBench#readme
+Project-URL: Bug Tracker, https://github.com/sjtuayj/OpenSTBench/issues
+Keywords: translation,evaluation,BLEU,COMET,BLEURT,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.11,>=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch<2.7,>=2.6
+Requires-Dist: torchaudio<2.7,>=2.6
+Requires-Dist: numpy
+Requires-Dist: sacrebleu>=2.0.0
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Requires-Dist: scipy
+Requires-Dist: soundfile
+Requires-Dist: jiwer
+Requires-Dist: textgrid
+Requires-Dist: transformers
+Requires-Dist: matplotlib
+Requires-Dist: resemblyzer
+Provides-Extra: comet
+Requires-Dist: unbabel-comet>=2.2.2; extra == "comet"
+Provides-Extra: whisper
+Requires-Dist: openai-whisper; extra == "whisper"
+Provides-Extra: speech-quality
+Requires-Dist: jiwer; extra == "speech-quality"
+Provides-Extra: emotion
+Requires-Dist: funasr; extra == "emotion"
+Requires-Dist: modelscope; extra == "emotion"
+Provides-Extra: paralinguistics
+Requires-Dist: transformers; extra == "paralinguistics"
+Provides-Extra: all
+Requires-Dist: unbabel-comet>=2.2.2; extra == "all"
+Requires-Dist: openai-whisper; extra == "all"
+Requires-Dist: jiwer; extra == "all"
+Requires-Dist: funasr; extra == "all"
+Requires-Dist: modelscope; extra == "all"
+Requires-Dist: transformers; extra == "all"
+Dynamic: license-file
+
+# OpenSTBench
+
+English | [中文](./README_zh.md)
+
+[![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
+[![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
+[![GitHub](https://img.shields.io/badge/GitHub-OpenSTBench-181717?style=for-the-badge&logo=github)](https://github.com/sjtuayj/OpenSTBench)
+[![X-LANCE](https://img.shields.io/badge/X--LANCE-grey?labelColor=lightgrey&logo=leanpub&style=for-the-badge)](https://x-lance.sjtu.edu.cn/)
+
+OpenSTBench is a multidimensional evaluation toolkit for speech translation. It is designed for heterogeneous systems, including speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline systems, and streaming systems.
+
+The toolkit organizes evaluation into three dimensions:
+
+- **Translation Quality**: whether the translated text preserves the source meaning.
+- **Speech Quality**: whether generated speech is natural, text-consistent, speaker-preserving, emotion-preserving, and faithful to non-verbal or paralinguistic events.
+- **Temporal Quality**: whether generated speech preserves duration structure and, for streaming systems, whether output is responsive.
+
+## Installation
+
+```bash
+pip install OpenSTBench
+```
+
+For local development:
+
+```bash
+pip install -e .
+```
+
+Optional extras:
+
+```bash
+pip install "OpenSTBench[comet]"
+pip install "OpenSTBench[whisper]"
+pip install "OpenSTBench[speech_quality]"
+pip install "OpenSTBench[emotion]"
+pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[all]"
+```
+
+BLEURT is installed separately:
+
+```bash
+pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
+```
+
+## Package Names
+
+- PyPI package: `OpenSTBench`
+- Python import: `openstbench`
+
+## Evaluation Dimensions
+
+| Dimension | Evaluator | System type | Main outputs |
+| :--- | :--- | :--- | :--- |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
+| Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
+| Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
+| Speech Quality | `ParalinguisticEvaluator` | S2ST | `Acoustic_Event_Count_F1`, `Acoustic_Event_Localization_F1`, `Acoustic_Event_Onset_Error` |
+| Temporal Quality | `TemporalConsistencyEvaluator` | S2ST | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
+| Temporal Quality | `LatencyEvaluator` | Streaming S2TT/S2ST | `First_Audio_Delay_(StartOffset_ms)`, `Overall_Translation_Delay_(ATD_ms)`, `End_Action_Delay_(CustomATD_ms)`, `Real_Time_Factor_(RTF)` |
+
+Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+
+## Datasets
+
+The paper uses the following datasets. Please follow the license and access terms of each original dataset.
+
+| Dataset | Used for | Link |
+| :--- | :--- | :--- |
+| MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
+| MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
+| NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
+| SynParaSpeech | Paralinguistic fidelity | [shawnpi/SynParaSpeech](https://huggingface.co/datasets/shawnpi/SynParaSpeech) |
+
+## Quick Start
+
+```python
+from openstbench import TranslationEvaluator
+
+evaluator = TranslationEvaluator(
+    use_bleu=True,
+    use_chrf=True,
+    use_comet=False,
+    use_bleurt=False,
+    device="cuda",
+)
+
+scores = evaluator.evaluate_all(
+    reference=["我喜欢看电影。", "今天天气很好。"],
+    target_text=["我喜欢看电影。", "今天天气很好。"],
+    source=["I like watching movies.", "The weather is nice today."],
+    target_lang="zh",
+)
+
+print(scores)
+```
+
+## Examples
+
+Complete parameter templates are kept in `examples/`. The README intentionally stays compact; use these files for configurable parameters, input formats, and output fields.
+
+- `examples/python/translation_eval.py`
+- `examples/python/speech_quality_eval.py`
+- `examples/python/speaker_similarity_eval.py`
+- `examples/python/emotion_eval.py`
+- `examples/python/paralinguistic_eval.py`
+- `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
+- `examples/python/latency_eval.py`
+- `examples/bash/install_extras.sh`
+- `examples/bash/run_latency_cli.sh`
+
+Latency can also be run from the module CLI:
+
+```bash
+python -m openstbench.latency.cli --help
+```
+
+## Conventions
+
+- Text inputs generally accept `list[str]`, one-sample-per-line `.txt` files, and `.json` files where supported by the evaluator.
+- Audio inputs generally accept folders, `list[str]`, `.txt` path lists, and `.json` path lists where supported by the evaluator.
+- For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
+- Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
+- Optional dependencies are loaded only when the corresponding evaluator needs them.
+
+
+## License
+
+OpenSTBench's original code is released under the MIT License. See [LICENSE](LICENSE).
+
+Some latency evaluation components include code adapted from [SimulEval](https://github.com/facebookresearch/SimulEval), which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for details.
+
+The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.

openstbench-1.1.0/README.md

@@ -0,0 +1,139 @@
+# OpenSTBench
+
+English | [中文](./README_zh.md)
+
+[![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
+[![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
+[![GitHub](https://img.shields.io/badge/GitHub-OpenSTBench-181717?style=for-the-badge&logo=github)](https://github.com/sjtuayj/OpenSTBench)
+[![X-LANCE](https://img.shields.io/badge/X--LANCE-grey?labelColor=lightgrey&logo=leanpub&style=for-the-badge)](https://x-lance.sjtu.edu.cn/)
+
+OpenSTBench is a multidimensional evaluation toolkit for speech translation. It is designed for heterogeneous systems, including speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline systems, and streaming systems.
+
+The toolkit organizes evaluation into three dimensions:
+
+- **Translation Quality**: whether the translated text preserves the source meaning.
+- **Speech Quality**: whether generated speech is natural, text-consistent, speaker-preserving, emotion-preserving, and faithful to non-verbal or paralinguistic events.
+- **Temporal Quality**: whether generated speech preserves duration structure and, for streaming systems, whether output is responsive.
+
+## Installation
+
+```bash
+pip install OpenSTBench
+```
+
+For local development:
+
+```bash
+pip install -e .
+```
+
+Optional extras:
+
+```bash
+pip install "OpenSTBench[comet]"
+pip install "OpenSTBench[whisper]"
+pip install "OpenSTBench[speech_quality]"
+pip install "OpenSTBench[emotion]"
+pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[all]"
+```
+
+BLEURT is installed separately:
+
+```bash
+pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
+```
+
+## Package Names
+
+- PyPI package: `OpenSTBench`
+- Python import: `openstbench`
+
+## Evaluation Dimensions
+
+| Dimension | Evaluator | System type | Main outputs |
+| :--- | :--- | :--- | :--- |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
+| Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
+| Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
+| Speech Quality | `ParalinguisticEvaluator` | S2ST | `Acoustic_Event_Count_F1`, `Acoustic_Event_Localization_F1`, `Acoustic_Event_Onset_Error` |
+| Temporal Quality | `TemporalConsistencyEvaluator` | S2ST | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
+| Temporal Quality | `LatencyEvaluator` | Streaming S2TT/S2ST | `First_Audio_Delay_(StartOffset_ms)`, `Overall_Translation_Delay_(ATD_ms)`, `End_Action_Delay_(CustomATD_ms)`, `Real_Time_Factor_(RTF)` |
+
+Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+
+## Datasets
+
+The paper uses the following datasets. Please follow the license and access terms of each original dataset.
+
+| Dataset | Used for | Link |
+| :--- | :--- | :--- |
+| MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
+| MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
+| NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
+| SynParaSpeech | Paralinguistic fidelity | [shawnpi/SynParaSpeech](https://huggingface.co/datasets/shawnpi/SynParaSpeech) |
+
+## Quick Start
+
+```python
+from openstbench import TranslationEvaluator
+
+evaluator = TranslationEvaluator(
+    use_bleu=True,
+    use_chrf=True,
+    use_comet=False,
+    use_bleurt=False,
+    device="cuda",
+)
+
+scores = evaluator.evaluate_all(
+    reference=["我喜欢看电影。", "今天天气很好。"],
+    target_text=["我喜欢看电影。", "今天天气很好。"],
+    source=["I like watching movies.", "The weather is nice today."],
+    target_lang="zh",
+)
+
+print(scores)
+```
+
+## Examples
+
+Complete parameter templates are kept in `examples/`. The README intentionally stays compact; use these files for configurable parameters, input formats, and output fields.
+
+- `examples/python/translation_eval.py`
+- `examples/python/speech_quality_eval.py`
+- `examples/python/speaker_similarity_eval.py`
+- `examples/python/emotion_eval.py`
+- `examples/python/paralinguistic_eval.py`
+- `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
+- `examples/python/latency_eval.py`
+- `examples/bash/install_extras.sh`
+- `examples/bash/run_latency_cli.sh`
+
+Latency can also be run from the module CLI:
+
+```bash
+python -m openstbench.latency.cli --help
+```
+
+## Conventions
+
+- Text inputs generally accept `list[str]`, one-sample-per-line `.txt` files, and `.json` files where supported by the evaluator.
+- Audio inputs generally accept folders, `list[str]`, `.txt` path lists, and `.json` path lists where supported by the evaluator.
+- For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
+- Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
+- Optional dependencies are loaded only when the corresponding evaluator needs them.
+
+
+## License
+
+OpenSTBench's original code is released under the MIT License. See [LICENSE](LICENSE).
+
+Some latency evaluation components include code adapted from [SimulEval](https://github.com/facebookresearch/SimulEval), which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for details.
+
+The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.

{openstbench-0.3.3 → openstbench-1.1.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "OpenSTBench"
-version = "0.3.3"  
-description = "Open speech and translation benchmarking toolkit supporting MT, ASR, TTS, SimulST, VC, and paralinguistics with optimized CJK language support"
+version = "1.1.0"  
+description = "Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings"
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.9,<3.11"
 license = {text = "MIT"}
@@ -15,9 +15,9 @@ authors = [
 
 keywords = [
     "translation", "evaluation", "BLEU", "COMET", "BLEURT",
-    "ASR", "speech-synthesis", "voice-conversion", "UTMOS", "WER",
-    "machine-translation", "multilingual", "CJK", "emotion",
-    "paralinguistics", "CLAP", "audio-events"
+    "speech-translation", "S2TT", "S2ST", "streaming", "latency",
+    "temporal-consistency", "UTMOS", "WER", "speaker-preservation",
+    "emotion", "paralinguistics", "CLAP", "audio-events"
 ]
 
 classifiers = [
@@ -37,6 +37,7 @@ dependencies = [
     "tqdm",
     "scipy",
     "soundfile",
+    "jiwer",
     "textgrid",
     "transformers",    
     "matplotlib",

openstbench-1.1.0/src/OpenSTBench.egg-info/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.4
+Name: OpenSTBench
+Version: 1.1.0
+Summary: Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings
+Author-email: Yanjie An <691476922@qq.com>
+License: MIT
+Project-URL: Homepage, https://github.com/sjtuayj/OpenSTBench
+Project-URL: Repository, https://github.com/sjtuayj/OpenSTBench
+Project-URL: Documentation, https://github.com/sjtuayj/OpenSTBench#readme
+Project-URL: Bug Tracker, https://github.com/sjtuayj/OpenSTBench/issues
+Keywords: translation,evaluation,BLEU,COMET,BLEURT,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: <3.11,>=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch<2.7,>=2.6
+Requires-Dist: torchaudio<2.7,>=2.6
+Requires-Dist: numpy
+Requires-Dist: sacrebleu>=2.0.0
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Requires-Dist: scipy
+Requires-Dist: soundfile
+Requires-Dist: jiwer
+Requires-Dist: textgrid
+Requires-Dist: transformers
+Requires-Dist: matplotlib
+Requires-Dist: resemblyzer
+Provides-Extra: comet
+Requires-Dist: unbabel-comet>=2.2.2; extra == "comet"
+Provides-Extra: whisper
+Requires-Dist: openai-whisper; extra == "whisper"
+Provides-Extra: speech-quality
+Requires-Dist: jiwer; extra == "speech-quality"
+Provides-Extra: emotion
+Requires-Dist: funasr; extra == "emotion"
+Requires-Dist: modelscope; extra == "emotion"
+Provides-Extra: paralinguistics
+Requires-Dist: transformers; extra == "paralinguistics"
+Provides-Extra: all
+Requires-Dist: unbabel-comet>=2.2.2; extra == "all"
+Requires-Dist: openai-whisper; extra == "all"
+Requires-Dist: jiwer; extra == "all"
+Requires-Dist: funasr; extra == "all"
+Requires-Dist: modelscope; extra == "all"
+Requires-Dist: transformers; extra == "all"
+Dynamic: license-file
+
+# OpenSTBench
+
+English | [中文](./README_zh.md)
+
+[![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
+[![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
+[![GitHub](https://img.shields.io/badge/GitHub-OpenSTBench-181717?style=for-the-badge&logo=github)](https://github.com/sjtuayj/OpenSTBench)
+[![X-LANCE](https://img.shields.io/badge/X--LANCE-grey?labelColor=lightgrey&logo=leanpub&style=for-the-badge)](https://x-lance.sjtu.edu.cn/)
+
+OpenSTBench is a multidimensional evaluation toolkit for speech translation. It is designed for heterogeneous systems, including speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline systems, and streaming systems.
+
+The toolkit organizes evaluation into three dimensions:
+
+- **Translation Quality**: whether the translated text preserves the source meaning.
+- **Speech Quality**: whether generated speech is natural, text-consistent, speaker-preserving, emotion-preserving, and faithful to non-verbal or paralinguistic events.
+- **Temporal Quality**: whether generated speech preserves duration structure and, for streaming systems, whether output is responsive.
+
+## Installation
+
+```bash
+pip install OpenSTBench
+```
+
+For local development:
+
+```bash
+pip install -e .
+```
+
+Optional extras:
+
+```bash
+pip install "OpenSTBench[comet]"
+pip install "OpenSTBench[whisper]"
+pip install "OpenSTBench[speech_quality]"
+pip install "OpenSTBench[emotion]"
+pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[all]"
+```
+
+BLEURT is installed separately:
+
+```bash
+pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
+```
+
+## Package Names
+
+- PyPI package: `OpenSTBench`
+- Python import: `openstbench`
+
+## Evaluation Dimensions
+
+| Dimension | Evaluator | System type | Main outputs |
+| :--- | :--- | :--- | :--- |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
+| Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
+| Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
+| Speech Quality | `ParalinguisticEvaluator` | S2ST | `Acoustic_Event_Count_F1`, `Acoustic_Event_Localization_F1`, `Acoustic_Event_Onset_Error` |
+| Temporal Quality | `TemporalConsistencyEvaluator` | S2ST | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
+| Temporal Quality | `LatencyEvaluator` | Streaming S2TT/S2ST | `First_Audio_Delay_(StartOffset_ms)`, `Overall_Translation_Delay_(ATD_ms)`, `End_Action_Delay_(CustomATD_ms)`, `Real_Time_Factor_(RTF)` |
+
+Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+
+## Datasets
+
+The paper uses the following datasets. Please follow the license and access terms of each original dataset.
+
+| Dataset | Used for | Link |
+| :--- | :--- | :--- |
+| MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
+| MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
+| NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
+| SynParaSpeech | Paralinguistic fidelity | [shawnpi/SynParaSpeech](https://huggingface.co/datasets/shawnpi/SynParaSpeech) |
+
+## Quick Start
+
+```python
+from openstbench import TranslationEvaluator
+
+evaluator = TranslationEvaluator(
+    use_bleu=True,
+    use_chrf=True,
+    use_comet=False,
+    use_bleurt=False,
+    device="cuda",
+)
+
+scores = evaluator.evaluate_all(
+    reference=["我喜欢看电影。", "今天天气很好。"],
+    target_text=["我喜欢看电影。", "今天天气很好。"],
+    source=["I like watching movies.", "The weather is nice today."],
+    target_lang="zh",
+)
+
+print(scores)
+```
+
+## Examples
+
+Complete parameter templates are kept in `examples/`. The README intentionally stays compact; use these files for configurable parameters, input formats, and output fields.
+
+- `examples/python/translation_eval.py`
+- `examples/python/speech_quality_eval.py`
+- `examples/python/speaker_similarity_eval.py`
+- `examples/python/emotion_eval.py`
+- `examples/python/paralinguistic_eval.py`
+- `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
+- `examples/python/latency_eval.py`
+- `examples/bash/install_extras.sh`
+- `examples/bash/run_latency_cli.sh`
+
+Latency can also be run from the module CLI:
+
+```bash
+python -m openstbench.latency.cli --help
+```
+
+## Conventions
+
+- Text inputs generally accept `list[str]`, one-sample-per-line `.txt` files, and `.json` files where supported by the evaluator.
+- Audio inputs generally accept folders, `list[str]`, `.txt` path lists, and `.json` path lists where supported by the evaluator.
+- For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
+- Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
+- Optional dependencies are loaded only when the corresponding evaluator needs them.
+
+
+## License
+
+OpenSTBench's original code is released under the MIT License. See [LICENSE](LICENSE).
+
+Some latency evaluation components include code adapted from [SimulEval](https://github.com/facebookresearch/SimulEval), which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for details.
+
+The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.

{openstbench-0.3.3 → openstbench-1.1.0}/src/OpenSTBench.egg-info/SOURCES.txt

@@ -1,3 +1,4 @@
+LICENSE
 README.md
 pyproject.toml
 src/OpenSTBench.egg-info/PKG-INFO
@@ -7,7 +8,6 @@ src/OpenSTBench.egg-info/requires.txt
 src/OpenSTBench.egg-info/top_level.txt
 src/openstbench/__init__.py
 src/openstbench/_model_loading.py
-src/openstbench/dataset.py
 src/openstbench/emotion_evaluator.py
 src/openstbench/paralinguistic_evaluator.py
 src/openstbench/speaker_similarity_evaluator.py

{openstbench-0.3.3 → openstbench-1.1.0}/src/OpenSTBench.egg-info/requires.txt

@@ -6,6 +6,7 @@ pandas
 tqdm
 scipy
 soundfile
+jiwer
 textgrid
 transformers
 matplotlib