PyPI - OpenSTBench - Versions diffs - 1.2.2__tar.gz → 1.2.4__tar.gz - Mend

OpenSTBench 1.2.2tar.gz → 1.2.4tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (28) hide show

{openstbench-1.2.2/src/OpenSTBench.egg-info → openstbench-1.2.4}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OpenSTBench
-Version: 1.2.2
+Version: 1.2.4
 Summary: Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings
 Author-email: Yanjie An <691476922@qq.com>
 License: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/sjtuayj/OpenSTBench
 Project-URL: Repository, https://github.com/sjtuayj/OpenSTBench
 Project-URL: Documentation, https://github.com/sjtuayj/OpenSTBench#readme
 Project-URL: Bug Tracker, https://github.com/sjtuayj/OpenSTBench/issues
-Keywords: translation,evaluation,BLEU,COMET,BLEURT,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
+Keywords: translation,evaluation,BLEU,COMET,BLEURT,MetricX,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
@@ -43,6 +43,12 @@ Requires-Dist: funasr; extra == "emotion"
 Requires-Dist: modelscope; extra == "emotion"
 Provides-Extra: paralinguistics
 Requires-Dist: transformers; extra == "paralinguistics"
+Provides-Extra: metricx
+Requires-Dist: transformers[torch]==4.30.2; extra == "metricx"
+Requires-Dist: sentencepiece==0.1.99; extra == "metricx"
+Requires-Dist: datasets==2.13.1; extra == "metricx"
+Requires-Dist: protobuf==3.20.3; extra == "metricx"
+Requires-Dist: accelerate>=0.26.0; extra == "metricx"
 Provides-Extra: all
 Requires-Dist: unbabel-comet>=2.2.2; extra == "all"
 Requires-Dist: openai-whisper; extra == "all"
@@ -50,13 +56,18 @@ Requires-Dist: jiwer; extra == "all"
 Requires-Dist: opencc-python-reimplemented; extra == "all"
 Requires-Dist: funasr; extra == "all"
 Requires-Dist: modelscope; extra == "all"
-Requires-Dist: transformers; extra == "all"
+Requires-Dist: transformers[torch]==4.30.2; extra == "all"
+Requires-Dist: sentencepiece==0.1.99; extra == "all"
+Requires-Dist: datasets==2.13.1; extra == "all"
+Requires-Dist: protobuf==3.20.3; extra == "all"
+Requires-Dist: accelerate>=0.26.0; extra == "all"
 Dynamic: license-file
 # OpenSTBench
 English | [中文](./README_zh.md)
+[![arXiv](https://img.shields.io/badge/arXiv-2605.30792-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2605.30792)
 [![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
 [![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
@@ -80,6 +91,10 @@ pip install OpenSTBench
 For local development:
 ```bash
+git clone https://github.com/sjtuayj/OpenSTBench.git
+cd OpenSTBench
+conda create -n openstbench python=3.10 -y
+conda activate openstbench
 pip install -e .
 ```
@@ -91,9 +106,15 @@ pip install "OpenSTBench[whisper]"
 pip install "OpenSTBench[speech_quality]"
 pip install "OpenSTBench[emotion]"
 pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[metricx]"
 pip install "OpenSTBench[all]"
 ```
+MetricX follows the official `google-research/metricx` runtime requirements. Installing
+`OpenSTBench[metricx]` or `OpenSTBench[all]` pins the MetricX-compatible stack,
+including `transformers[torch]==4.30.2`, `sentencepiece==0.1.99`,
+`datasets==2.13.1`, `protobuf==3.20.3`, and `accelerate>=0.26.0`.
 BLEURT is installed separately:
 ```bash
@@ -109,7 +130,7 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 | Dimension | Evaluator | System type | Main outputs |
 | :--- | :--- | :--- | :--- |
-| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT`, `MetricX`, `MetricX_QE` |
 | Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
 | Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
 | Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
@@ -119,6 +140,12 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+## Experimental Overview
+The radar plot below illustrates the multidimensional view produced by OpenSTBench for representative streaming and offline speech translation systems. It summarizes how systems can differ across translation quality, speech quality, and temporal quality: a system with strong translation quality may still show different behavior in speech realization, speaker or emotion preservation, paralinguistic fidelity, temporal consistency, and latency or efficiency.
+![OpenSTBench experimental radar overview](./radar.png)
 ## Datasets
 The paper uses the following datasets. Please follow the license and access terms of each original dataset.
@@ -126,7 +153,7 @@ The paper uses the following datasets. Please follow the license and access term
 | Dataset | Used for | Link |
 | :--- | :--- | :--- |
 | MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
-| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set is available on [Hugging Face Datasets](https://huggingface.co/datasets/ayj111/openstbench-paired-set); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
 | RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
 | MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
 | NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
@@ -142,6 +169,7 @@ evaluator = TranslationEvaluator(
     use_chrf=True,
     use_comet=False,
     use_bleurt=False,
+    use_metricx=True,
     device="cuda",
 )
@@ -183,7 +211,31 @@ python -m openstbench.latency.cli --help
 - For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
 - Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
 - Optional dependencies are loaded only when the corresponding evaluator needs them.
+- MetricX is enabled by default in `TranslationEvaluator`. It follows the official google-research/metricx README, uses text only, reports error scores in `[0, 25]` where lower is better, and can be disabled with `use_metricx=False`.
+## Acknowledgements
+- We especially thank [SimulEval](https://github.com/facebookresearch/SimulEval), from which parts of OpenSTBench's latency evaluation components are adapted
+- [sacreBLEU](https://github.com/mjpost/sacrebleu), [COMET](https://github.com/Unbabel/COMET), [MetricX](https://github.com/google-research/metricx), and [bleurt-pytorch](https://github.com/lucadiliello/bleurt-pytorch), a PyTorch port of [BLEURT](https://github.com/google-research/bleurt), for translation quality evaluation
+- [Whisper](https://github.com/openai/whisper), [SpeechMOS/UTMOS](https://github.com/tarepan/SpeechMOS), [Resemblyzer](https://github.com/resemble-ai/Resemblyzer), and [WavLM](https://github.com/microsoft/unilm/tree/master/wavlm) for speech quality and speaker similarity evaluation
+- [FunASR](https://github.com/modelscope/FunASR) and [Emotion2Vec](https://modelscope.cn/models/iic/emotion2vec_plus_large) for emotion preservation evaluation
+- [CLAP](https://huggingface.co/laion/clap-htsat-fused) and [Hugging Face Transformers](https://github.com/huggingface/transformers) for paralinguistic event evaluation
+## Citation
+If you find our work useful, please cite as：
+```bibtex
+@misc{an2026openstbenchsemanticevaluationspeech,
+      title={OpenSTBench: Beyond Semantic Evaluation for Speech Translation},
+      author={Yanjie An and Yuxiang Zhao and Yichi Zhang and Qixi Zheng and Yujie Tu and Keqi Deng and Kai Yu and Xie Chen},
+      year={2026},
+      eprint={2605.30792},
+      archivePrefix={arXiv},
+      primaryClass={eess.AS},
+      url={https://arxiv.org/abs/2605.30792},
+}
+```
 ## License

{openstbench-1.2.2 → openstbench-1.2.4}/README.md RENAMED Viewed

@@ -2,6 +2,7 @@
 English | [中文](./README_zh.md)
+[![arXiv](https://img.shields.io/badge/arXiv-2605.30792-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2605.30792)
 [![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
 [![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
@@ -25,6 +26,10 @@ pip install OpenSTBench
 For local development:
 ```bash
+git clone https://github.com/sjtuayj/OpenSTBench.git
+cd OpenSTBench
+conda create -n openstbench python=3.10 -y
+conda activate openstbench
 pip install -e .
 ```
@@ -36,9 +41,15 @@ pip install "OpenSTBench[whisper]"
 pip install "OpenSTBench[speech_quality]"
 pip install "OpenSTBench[emotion]"
 pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[metricx]"
 pip install "OpenSTBench[all]"
 ```
+MetricX follows the official `google-research/metricx` runtime requirements. Installing
+`OpenSTBench[metricx]` or `OpenSTBench[all]` pins the MetricX-compatible stack,
+including `transformers[torch]==4.30.2`, `sentencepiece==0.1.99`,
+`datasets==2.13.1`, `protobuf==3.20.3`, and `accelerate>=0.26.0`.
 BLEURT is installed separately:
 ```bash
@@ -54,7 +65,7 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 | Dimension | Evaluator | System type | Main outputs |
 | :--- | :--- | :--- | :--- |
-| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT`, `MetricX`, `MetricX_QE` |
 | Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
 | Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
 | Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
@@ -64,6 +75,12 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+## Experimental Overview
+The radar plot below illustrates the multidimensional view produced by OpenSTBench for representative streaming and offline speech translation systems. It summarizes how systems can differ across translation quality, speech quality, and temporal quality: a system with strong translation quality may still show different behavior in speech realization, speaker or emotion preservation, paralinguistic fidelity, temporal consistency, and latency or efficiency.
+![OpenSTBench experimental radar overview](./radar.png)
 ## Datasets
 The paper uses the following datasets. Please follow the license and access terms of each original dataset.
@@ -71,7 +88,7 @@ The paper uses the following datasets. Please follow the license and access term
 | Dataset | Used for | Link |
 | :--- | :--- | :--- |
 | MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
-| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set is available on [Hugging Face Datasets](https://huggingface.co/datasets/ayj111/openstbench-paired-set); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
 | RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
 | MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
 | NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
@@ -87,6 +104,7 @@ evaluator = TranslationEvaluator(
     use_chrf=True,
     use_comet=False,
     use_bleurt=False,
+    use_metricx=True,
     device="cuda",
 )
@@ -128,7 +146,31 @@ python -m openstbench.latency.cli --help
 - For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
 - Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
 - Optional dependencies are loaded only when the corresponding evaluator needs them.
+- MetricX is enabled by default in `TranslationEvaluator`. It follows the official google-research/metricx README, uses text only, reports error scores in `[0, 25]` where lower is better, and can be disabled with `use_metricx=False`.
+## Acknowledgements
+- We especially thank [SimulEval](https://github.com/facebookresearch/SimulEval), from which parts of OpenSTBench's latency evaluation components are adapted
+- [sacreBLEU](https://github.com/mjpost/sacrebleu), [COMET](https://github.com/Unbabel/COMET), [MetricX](https://github.com/google-research/metricx), and [bleurt-pytorch](https://github.com/lucadiliello/bleurt-pytorch), a PyTorch port of [BLEURT](https://github.com/google-research/bleurt), for translation quality evaluation
+- [Whisper](https://github.com/openai/whisper), [SpeechMOS/UTMOS](https://github.com/tarepan/SpeechMOS), [Resemblyzer](https://github.com/resemble-ai/Resemblyzer), and [WavLM](https://github.com/microsoft/unilm/tree/master/wavlm) for speech quality and speaker similarity evaluation
+- [FunASR](https://github.com/modelscope/FunASR) and [Emotion2Vec](https://modelscope.cn/models/iic/emotion2vec_plus_large) for emotion preservation evaluation
+- [CLAP](https://huggingface.co/laion/clap-htsat-fused) and [Hugging Face Transformers](https://github.com/huggingface/transformers) for paralinguistic event evaluation
+## Citation
+If you find our work useful, please cite as：
+```bibtex
+@misc{an2026openstbenchsemanticevaluationspeech,
+      title={OpenSTBench: Beyond Semantic Evaluation for Speech Translation},
+      author={Yanjie An and Yuxiang Zhao and Yichi Zhang and Qixi Zheng and Yujie Tu and Keqi Deng and Kai Yu and Xie Chen},
+      year={2026},
+      eprint={2605.30792},
+      archivePrefix={arXiv},
+      primaryClass={eess.AS},
+      url={https://arxiv.org/abs/2605.30792},
+}
+```
 ## License
@@ -136,4 +178,4 @@ OpenSTBench's original code is released under the MIT License. See [LICENSE](LIC
 Some latency evaluation components include code adapted from [SimulEval](https://github.com/facebookresearch/SimulEval), which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). Those adapted portions are distributed under CC BY-SA 4.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for details.
-The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.
+The datasets referenced by OpenSTBench, including the datasets used in the paper, are not covered by the OpenSTBench code license. They are provided by their original authors or distributors under their own licenses and terms of use. Some datasets are restricted to research or non-commercial use.

{openstbench-1.2.2 → openstbench-1.2.4}/pyproject.toml RENAMED Viewed

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "OpenSTBench"
-version = "1.2.2"
+version = "1.2.4"
 description = "Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings"
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.9,<3.11"
@@ -14,7 +14,7 @@ authors = [
 ]
 keywords = [
-    "translation", "evaluation", "BLEU", "COMET", "BLEURT",
+    "translation", "evaluation", "BLEU", "COMET", "BLEURT", "MetricX",
     "speech-translation", "S2TT", "S2ST", "streaming", "latency",
     "temporal-consistency", "UTMOS", "WER", "speaker-preservation",
     "emotion", "paralinguistics", "CLAP", "audio-events"
@@ -57,7 +57,26 @@ whisper = ["openai-whisper"]
 speech_quality = ["jiwer", "opencc-python-reimplemented"]
 emotion = ["funasr", "modelscope"]
 paralinguistics = ["transformers"]
-all = ["unbabel-comet>=2.2.2", "openai-whisper", "jiwer", "opencc-python-reimplemented", "funasr", "modelscope", "transformers"]
+metricx = [
+    "transformers[torch]==4.30.2",
+    "sentencepiece==0.1.99",
+    "datasets==2.13.1",
+    "protobuf==3.20.3",
+    "accelerate>=0.26.0",
+]
+all = [
+    "unbabel-comet>=2.2.2",
+    "openai-whisper",
+    "jiwer",
+    "opencc-python-reimplemented",
+    "funasr",
+    "modelscope",
+    "transformers[torch]==4.30.2",
+    "sentencepiece==0.1.99",
+    "datasets==2.13.1",
+    "protobuf==3.20.3",
+    "accelerate>=0.26.0",
+]
 [tool.setuptools]
 package-dir = {"" = "src"}

{openstbench-1.2.2 → openstbench-1.2.4/src/OpenSTBench.egg-info}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OpenSTBench
-Version: 1.2.2
+Version: 1.2.4
 Summary: Unified multidimensional evaluation toolkit for S2TT and S2ST systems in offline and streaming speech translation settings
 Author-email: Yanjie An <691476922@qq.com>
 License: MIT
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/sjtuayj/OpenSTBench
 Project-URL: Repository, https://github.com/sjtuayj/OpenSTBench
 Project-URL: Documentation, https://github.com/sjtuayj/OpenSTBench#readme
 Project-URL: Bug Tracker, https://github.com/sjtuayj/OpenSTBench/issues
-Keywords: translation,evaluation,BLEU,COMET,BLEURT,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
+Keywords: translation,evaluation,BLEU,COMET,BLEURT,MetricX,speech-translation,S2TT,S2ST,streaming,latency,temporal-consistency,UTMOS,WER,speaker-preservation,emotion,paralinguistics,CLAP,audio-events
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
@@ -43,6 +43,12 @@ Requires-Dist: funasr; extra == "emotion"
 Requires-Dist: modelscope; extra == "emotion"
 Provides-Extra: paralinguistics
 Requires-Dist: transformers; extra == "paralinguistics"
+Provides-Extra: metricx
+Requires-Dist: transformers[torch]==4.30.2; extra == "metricx"
+Requires-Dist: sentencepiece==0.1.99; extra == "metricx"
+Requires-Dist: datasets==2.13.1; extra == "metricx"
+Requires-Dist: protobuf==3.20.3; extra == "metricx"
+Requires-Dist: accelerate>=0.26.0; extra == "metricx"
 Provides-Extra: all
 Requires-Dist: unbabel-comet>=2.2.2; extra == "all"
 Requires-Dist: openai-whisper; extra == "all"
@@ -50,13 +56,18 @@ Requires-Dist: jiwer; extra == "all"
 Requires-Dist: opencc-python-reimplemented; extra == "all"
 Requires-Dist: funasr; extra == "all"
 Requires-Dist: modelscope; extra == "all"
-Requires-Dist: transformers; extra == "all"
+Requires-Dist: transformers[torch]==4.30.2; extra == "all"
+Requires-Dist: sentencepiece==0.1.99; extra == "all"
+Requires-Dist: datasets==2.13.1; extra == "all"
+Requires-Dist: protobuf==3.20.3; extra == "all"
+Requires-Dist: accelerate>=0.26.0; extra == "all"
 Dynamic: license-file
 # OpenSTBench
 English | [中文](./README_zh.md)
+[![arXiv](https://img.shields.io/badge/arXiv-2605.30792-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2605.30792)
 [![PyPI](https://img.shields.io/pypi/v/OpenSTBench?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/OpenSTBench/)
 [![Python](https://img.shields.io/badge/Python-3.9--3.10-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
@@ -80,6 +91,10 @@ pip install OpenSTBench
 For local development:
 ```bash
+git clone https://github.com/sjtuayj/OpenSTBench.git
+cd OpenSTBench
+conda create -n openstbench python=3.10 -y
+conda activate openstbench
 pip install -e .
 ```
@@ -91,9 +106,15 @@ pip install "OpenSTBench[whisper]"
 pip install "OpenSTBench[speech_quality]"
 pip install "OpenSTBench[emotion]"
 pip install "OpenSTBench[paralinguistics]"
+pip install "OpenSTBench[metricx]"
 pip install "OpenSTBench[all]"
 ```
+MetricX follows the official `google-research/metricx` runtime requirements. Installing
+`OpenSTBench[metricx]` or `OpenSTBench[all]` pins the MetricX-compatible stack,
+including `transformers[torch]==4.30.2`, `sentencepiece==0.1.99`,
+`datasets==2.13.1`, `protobuf==3.20.3`, and `accelerate>=0.26.0`.
 BLEURT is installed separately:
 ```bash
@@ -109,7 +130,7 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 | Dimension | Evaluator | System type | Main outputs |
 | :--- | :--- | :--- | :--- |
-| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT` |
+| Translation Quality | `TranslationEvaluator` | S2TT, S2ST transcripts | `sacreBLEU`, `chrF++`, `COMET`, `BLEURT`, `MetricX`, `MetricX_QE` |
 | Speech Quality | `SpeechQualityEvaluator` | S2ST | `UTMOS`, `WER_Consistency`, `CER_Consistency` |
 | Speech Quality | `SpeakerSimilarityEvaluator` | S2ST | `average_wavlm_large_similarity`, `average_resemblyzer_similarity` |
 | Speech Quality | `EmotionEvaluator` | S2ST | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
@@ -119,6 +140,12 @@ pip install git+https://github.com/lucadiliello/bleurt-pytorch.git
 Offline and streaming are supported system settings, not separate metric dimensions. Use the evaluators that match the available outputs: text, generated speech, source/target audio pairs, event annotations, or streaming traces.
+## Experimental Overview
+The radar plot below illustrates the multidimensional view produced by OpenSTBench for representative streaming and offline speech translation systems. It summarizes how systems can differ across translation quality, speech quality, and temporal quality: a system with strong translation quality may still show different behavior in speech realization, speaker or emotion preservation, paralinguistic fidelity, temporal consistency, and latency or efficiency.
+![OpenSTBench experimental radar overview](./radar.png)
 ## Datasets
 The paper uses the following datasets. Please follow the license and access terms of each original dataset.
@@ -126,7 +153,7 @@ The paper uses the following datasets. Please follow the license and access term
 | Dataset | Used for | Link |
 | :--- | :--- | :--- |
 | MSLT dev | Translation quality, speech quality, temporal consistency, latency | [Microsoft Speech Language Translation Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=54689) |
-| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set will be released through [GitHub Releases](https://github.com/sjtuayj/OpenSTBench/releases); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
+| LibriTTS-based paired speaker set | Speaker preservation | The constructed OpenSTBench paired set is available on [Hugging Face Datasets](https://huggingface.co/datasets/ayj111/openstbench-paired-set); the source corpus is [LibriTTS](https://www.openslr.org/60/) |
 | RAVDESS | Emotion preservation | [Audio_Speech_Actors_01-24.zip](https://zenodo.org/records/1188976) from the RAVDESS Zenodo record |
 | MCAE-SPPS | Emotion preservation | [MCAE-SPPS on OSF](https://doi.org/10.17605/OSF.IO/9JYZC) |
 | NonverbalTTS test | Paralinguistic fidelity | [deepvk/NonverbalTTS](https://huggingface.co/datasets/deepvk/NonverbalTTS) |
@@ -142,6 +169,7 @@ evaluator = TranslationEvaluator(
     use_chrf=True,
     use_comet=False,
     use_bleurt=False,
+    use_metricx=True,
     device="cuda",
 )
@@ -183,7 +211,31 @@ python -m openstbench.latency.cli --help
 - For `zh`, `ja`, and `ko`, speech consistency reports `CER_Consistency`; other languages report `WER_Consistency`.
 - Evaluators that accept pretrained model sources use a local-first rule. If the supplied local path exists, OpenSTBench uses it; otherwise it falls back to the configured remote model id.
 - Optional dependencies are loaded only when the corresponding evaluator needs them.
+- MetricX is enabled by default in `TranslationEvaluator`. It follows the official google-research/metricx README, uses text only, reports error scores in `[0, 25]` where lower is better, and can be disabled with `use_metricx=False`.
+## Acknowledgements
+- We especially thank [SimulEval](https://github.com/facebookresearch/SimulEval), from which parts of OpenSTBench's latency evaluation components are adapted
+- [sacreBLEU](https://github.com/mjpost/sacrebleu), [COMET](https://github.com/Unbabel/COMET), [MetricX](https://github.com/google-research/metricx), and [bleurt-pytorch](https://github.com/lucadiliello/bleurt-pytorch), a PyTorch port of [BLEURT](https://github.com/google-research/bleurt), for translation quality evaluation
+- [Whisper](https://github.com/openai/whisper), [SpeechMOS/UTMOS](https://github.com/tarepan/SpeechMOS), [Resemblyzer](https://github.com/resemble-ai/Resemblyzer), and [WavLM](https://github.com/microsoft/unilm/tree/master/wavlm) for speech quality and speaker similarity evaluation
+- [FunASR](https://github.com/modelscope/FunASR) and [Emotion2Vec](https://modelscope.cn/models/iic/emotion2vec_plus_large) for emotion preservation evaluation
+- [CLAP](https://huggingface.co/laion/clap-htsat-fused) and [Hugging Face Transformers](https://github.com/huggingface/transformers) for paralinguistic event evaluation
+## Citation
+If you find our work useful, please cite as：
+```bibtex
+@misc{an2026openstbenchsemanticevaluationspeech,
+      title={OpenSTBench: Beyond Semantic Evaluation for Speech Translation},
+      author={Yanjie An and Yuxiang Zhao and Yichi Zhang and Qixi Zheng and Yujie Tu and Keqi Deng and Kai Yu and Xie Chen},
+      year={2026},
+      eprint={2605.30792},
+      archivePrefix={arXiv},
+      primaryClass={eess.AS},
+      url={https://arxiv.org/abs/2605.30792},
+}
+```
 ## License

{openstbench-1.2.2 → openstbench-1.2.4}/src/OpenSTBench.egg-info/SOURCES.txt RENAMED Viewed

@@ -10,6 +10,7 @@ src/openstbench/__init__.py
 src/openstbench/_model_loading.py
 src/openstbench/emotion_evaluator.py
 src/openstbench/language_policy.py
+src/openstbench/metricx_evaluator.py
 src/openstbench/paralinguistic_evaluator.py
 src/openstbench/speaker_similarity_evaluator.py
 src/openstbench/speech_quality_evaluator.py

{openstbench-1.2.2 → openstbench-1.2.4}/src/OpenSTBench.egg-info/requires.txt RENAMED Viewed

@@ -20,7 +20,11 @@ jiwer
 opencc-python-reimplemented
 funasr
 modelscope
-transformers
+transformers[torch]==4.30.2
+sentencepiece==0.1.99
+datasets==2.13.1
+protobuf==3.20.3
+accelerate>=0.26.0
 [comet]
 unbabel-comet>=2.2.2
@@ -29,6 +33,13 @@ unbabel-comet>=2.2.2
 funasr
 modelscope
+[metricx]
+transformers[torch]==4.30.2
+sentencepiece==0.1.99
+datasets==2.13.1
+protobuf==3.20.3
+accelerate>=0.26.0
 [paralinguistics]
 transformers

{openstbench-1.2.2 → openstbench-1.2.4}/src/openstbench/__init__.py RENAMED Viewed

@@ -9,7 +9,7 @@ actually accessed.
 from importlib import import_module
 from typing import Dict, Tuple
-__version__ = "1.2.2"
+__version__ = "1.2.4"
 __all__ = [
     "TranslationEvaluator",

OpenSTBench 1.2.2__tar.gz → 1.2.4__tar.gz

OpenSTBench 1.2.2tar.gz → 1.2.4tar.gz