OpenSTBench 0.2.0__tar.gz → 0.3.0__tar.gz

{openstbench-0.2.0 → openstbench-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OpenSTBench
-Version: 0.2.0
+Version: 0.3.0
 Summary: Open speech and translation benchmarking toolkit supporting MT, ASR, TTS, SimulST, VC, and paralinguistics with optimized CJK language support
 Author-email: Yanjie An <691476922@qq.com>
 License: MIT
@@ -65,6 +65,7 @@ This project is best suited for these directions:
 - S2ST evaluation by combining text quality, speech quality, speaker similarity, and latency
 - Streaming or simultaneous speech translation latency evaluation with a custom agent
 - Preservation analysis for speech translation outputs, including speaker similarity, emotion, and paralinguistic similarity
+- Temporal consistency analysis for speech translation or dubbing outputs, including duration compliance and duration error
 
 ## Core Modules
 
@@ -75,6 +76,7 @@ This project is best suited for these directions:
 | `SpeakerSimilarityEvaluator` | Speaker preservation | `wavlm_similarity`, `resemblyzer_similarity` |
 | `EmotionEvaluator` | Emotion preservation or classification accuracy | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
 | `ParalinguisticEvaluator` | Non-verbal and paralinguistic preservation | `Paralinguistic_Fidelity_Cosine`, `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, `Acoustic_Event_Preservation_Macro_Recall`, `Event_Aligned_Preservation_Rate`, `Conditional_Relative_Onset_Error` |
+| `TemporalConsistencyEvaluator` | Source-target temporal structure consistency | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
 | `LatencyEvaluator` | Streaming / simultaneous translation latency | `StartOffset`, `ATD`, `CustomATD`, `RTF`, `Model_Generate_RTF` |
 
 ## Installation
@@ -119,7 +121,11 @@ openstbench
 Example:
 
 ```python
-from openstbench import TranslationEvaluator, SpeechQualityEvaluator
+from openstbench import (
+    TranslationEvaluator,
+    SpeechQualityEvaluator,
+    TemporalConsistencyEvaluator,
+)
 ```
 
 ## Quick Start
@@ -134,6 +140,7 @@ Python examples:
 - `examples/python/emotion_eval.py`
 - `examples/python/paralinguistic_eval.py`
 - `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
 - `examples/python/latency_eval.py`
 
 Shell examples:
@@ -141,6 +148,23 @@ Shell examples:
 - `examples/bash/install_extras.sh`
 - `examples/bash/run_latency_cli.sh`
 
+Minimal temporal consistency example:
+
+```python
+from openstbench import TemporalConsistencyEvaluator
+
+evaluator = TemporalConsistencyEvaluator(
+    thresholds=(0.2, 0.4),
+)
+
+results, diagnostics = evaluator.evaluate_all(
+    source_audio="./source_wavs",
+    target_audio="./generated_wavs",
+    sample_ids=["sample_1", "sample_2"],
+    return_diagnostics=True,
+)
+```
+
 Latency output distinguishes two RTF variants:
 
 - `Real_Time_Factor_(RTF)`: system-level RTF. This includes agent policy overhead, pre/post-processing, and other runtime costs around model inference.
@@ -166,6 +190,8 @@ Common audio inputs support:
 - For `zh` / `ja` / `ko`, the toolkit uses CJK-aware handling for text-side evaluation.
 - `SpeechQualityEvaluator` returns `CER_Consistency` for `zh` / `ja` / `ko`, and `WER_Consistency` for most other languages.
 - `ParalinguisticEvaluator` always supports `Paralinguistic_Fidelity_Cosine`, a continuous CLAP-based audio similarity score between source and target speech.
+- `TemporalConsistencyEvaluator` supports `List[str]`, audio folders, `.txt` path lists, and `.json` path lists for both `source_audio` and `target_audio`.
+- `TemporalConsistencyEvaluator` reports thresholded duration compliance metrics (`Duration_Consistency_SLC_*`).
 - The discrete preservation branch is an utterance-level single-label task. With source-side gold labels, it reports `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, and `Acoustic_Event_Preservation_Macro_Recall`.
 - If `source_onsets_ms` are available, the evaluator can also report alignment-aware metrics: `Event_Aligned_Preservation_Rate` and `Conditional_Relative_Onset_Error`.
 - Alignment is computed on relative onset position, not absolute wall-clock time. This makes it suitable for cross-lingual S2ST where source and target utterance durations naturally differ.
@@ -176,6 +202,11 @@ Common audio inputs support:
 - The default event localizer is also replaceable. Custom localizers only need to implement `localize(audio_paths, labels, candidate_labels)`.
 - Dataset-specific label mapping is intentionally outside the core package. Pass `candidate_labels` and `label_normalizer` at call time so the same evaluator works across datasets without changing core code.
 - For offline environments, `clap_model_path` accepts either a Hugging Face repo id or a local model directory or snapshot.
+- Model-loading parameters such as `clap_model_path`, `wavlm_model_path`, `whisper_model`, `e2v_model_path`, `comet_model`, and `bleurt_path` now use a consistent local-first rule: if the supplied local path exists, it is used; otherwise the evaluator falls back to the default remote model id.
 - In S2S latency evaluation, alignment prefers the model's native transcript when available. If the model is audio-only, the evaluator can optionally use ASR fallback to prepare alignment text.
 - For S2S forced alignment, pass language-appropriate MFA models through `alignment_acoustic_model` and `alignment_dictionary_model`. The defaults are English.
 - Some modules rely on optional dependencies or local model paths in offline environments.
+
+## License
+
+MIT License

{openstbench-0.2.0 → openstbench-0.3.0}/README.md

@@ -16,6 +16,7 @@ This project is best suited for these directions:
 - S2ST evaluation by combining text quality, speech quality, speaker similarity, and latency
 - Streaming or simultaneous speech translation latency evaluation with a custom agent
 - Preservation analysis for speech translation outputs, including speaker similarity, emotion, and paralinguistic similarity
+- Temporal consistency analysis for speech translation or dubbing outputs, including duration compliance and duration error
 
 ## Core Modules
 
@@ -26,6 +27,7 @@ This project is best suited for these directions:
 | `SpeakerSimilarityEvaluator` | Speaker preservation | `wavlm_similarity`, `resemblyzer_similarity` |
 | `EmotionEvaluator` | Emotion preservation or classification accuracy | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
 | `ParalinguisticEvaluator` | Non-verbal and paralinguistic preservation | `Paralinguistic_Fidelity_Cosine`, `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, `Acoustic_Event_Preservation_Macro_Recall`, `Event_Aligned_Preservation_Rate`, `Conditional_Relative_Onset_Error` |
+| `TemporalConsistencyEvaluator` | Source-target temporal structure consistency | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
 | `LatencyEvaluator` | Streaming / simultaneous translation latency | `StartOffset`, `ATD`, `CustomATD`, `RTF`, `Model_Generate_RTF` |
 
 ## Installation
@@ -70,7 +72,11 @@ openstbench
 Example:
 
 ```python
-from openstbench import TranslationEvaluator, SpeechQualityEvaluator
+from openstbench import (
+    TranslationEvaluator,
+    SpeechQualityEvaluator,
+    TemporalConsistencyEvaluator,
+)
 ```
 
 ## Quick Start
@@ -85,6 +91,7 @@ Python examples:
 - `examples/python/emotion_eval.py`
 - `examples/python/paralinguistic_eval.py`
 - `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
 - `examples/python/latency_eval.py`
 
 Shell examples:
@@ -92,6 +99,23 @@ Shell examples:
 - `examples/bash/install_extras.sh`
 - `examples/bash/run_latency_cli.sh`
 
+Minimal temporal consistency example:
+
+```python
+from openstbench import TemporalConsistencyEvaluator
+
+evaluator = TemporalConsistencyEvaluator(
+    thresholds=(0.2, 0.4),
+)
+
+results, diagnostics = evaluator.evaluate_all(
+    source_audio="./source_wavs",
+    target_audio="./generated_wavs",
+    sample_ids=["sample_1", "sample_2"],
+    return_diagnostics=True,
+)
+```
+
 Latency output distinguishes two RTF variants:
 
 - `Real_Time_Factor_(RTF)`: system-level RTF. This includes agent policy overhead, pre/post-processing, and other runtime costs around model inference.
@@ -117,6 +141,8 @@ Common audio inputs support:
 - For `zh` / `ja` / `ko`, the toolkit uses CJK-aware handling for text-side evaluation.
 - `SpeechQualityEvaluator` returns `CER_Consistency` for `zh` / `ja` / `ko`, and `WER_Consistency` for most other languages.
 - `ParalinguisticEvaluator` always supports `Paralinguistic_Fidelity_Cosine`, a continuous CLAP-based audio similarity score between source and target speech.
+- `TemporalConsistencyEvaluator` supports `List[str]`, audio folders, `.txt` path lists, and `.json` path lists for both `source_audio` and `target_audio`.
+- `TemporalConsistencyEvaluator` reports thresholded duration compliance metrics (`Duration_Consistency_SLC_*`).
 - The discrete preservation branch is an utterance-level single-label task. With source-side gold labels, it reports `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, and `Acoustic_Event_Preservation_Macro_Recall`.
 - If `source_onsets_ms` are available, the evaluator can also report alignment-aware metrics: `Event_Aligned_Preservation_Rate` and `Conditional_Relative_Onset_Error`.
 - Alignment is computed on relative onset position, not absolute wall-clock time. This makes it suitable for cross-lingual S2ST where source and target utterance durations naturally differ.
@@ -127,6 +153,11 @@ Common audio inputs support:
 - The default event localizer is also replaceable. Custom localizers only need to implement `localize(audio_paths, labels, candidate_labels)`.
 - Dataset-specific label mapping is intentionally outside the core package. Pass `candidate_labels` and `label_normalizer` at call time so the same evaluator works across datasets without changing core code.
 - For offline environments, `clap_model_path` accepts either a Hugging Face repo id or a local model directory or snapshot.
+- Model-loading parameters such as `clap_model_path`, `wavlm_model_path`, `whisper_model`, `e2v_model_path`, `comet_model`, and `bleurt_path` now use a consistent local-first rule: if the supplied local path exists, it is used; otherwise the evaluator falls back to the default remote model id.
 - In S2S latency evaluation, alignment prefers the model's native transcript when available. If the model is audio-only, the evaluator can optionally use ASR fallback to prepare alignment text.
 - For S2S forced alignment, pass language-appropriate MFA models through `alignment_acoustic_model` and `alignment_dictionary_model`. The defaults are English.
 - Some modules rely on optional dependencies or local model paths in offline environments.
+
+## License
+
+MIT License

{openstbench-0.2.0 → openstbench-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "OpenSTBench"
-version = "0.2.0"  
+version = "0.3.0"  
 description = "Open speech and translation benchmarking toolkit supporting MT, ASR, TTS, SimulST, VC, and paralinguistics with optimized CJK language support"
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.8"

{openstbench-0.2.0 → openstbench-0.3.0}/src/OpenSTBench.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: OpenSTBench
-Version: 0.2.0
+Version: 0.3.0
 Summary: Open speech and translation benchmarking toolkit supporting MT, ASR, TTS, SimulST, VC, and paralinguistics with optimized CJK language support
 Author-email: Yanjie An <691476922@qq.com>
 License: MIT
@@ -65,6 +65,7 @@ This project is best suited for these directions:
 - S2ST evaluation by combining text quality, speech quality, speaker similarity, and latency
 - Streaming or simultaneous speech translation latency evaluation with a custom agent
 - Preservation analysis for speech translation outputs, including speaker similarity, emotion, and paralinguistic similarity
+- Temporal consistency analysis for speech translation or dubbing outputs, including duration compliance and duration error
 
 ## Core Modules
 
@@ -75,6 +76,7 @@ This project is best suited for these directions:
 | `SpeakerSimilarityEvaluator` | Speaker preservation | `wavlm_similarity`, `resemblyzer_similarity` |
 | `EmotionEvaluator` | Emotion preservation or classification accuracy | `Emotion2Vec_Cosine_Similarity`, `Audio_Emotion_Accuracy` |
 | `ParalinguisticEvaluator` | Non-verbal and paralinguistic preservation | `Paralinguistic_Fidelity_Cosine`, `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, `Acoustic_Event_Preservation_Macro_Recall`, `Event_Aligned_Preservation_Rate`, `Conditional_Relative_Onset_Error` |
+| `TemporalConsistencyEvaluator` | Source-target temporal structure consistency | `Duration_Consistency_SLC_0.2`, `Duration_Consistency_SLC_0.4` |
 | `LatencyEvaluator` | Streaming / simultaneous translation latency | `StartOffset`, `ATD`, `CustomATD`, `RTF`, `Model_Generate_RTF` |
 
 ## Installation
@@ -119,7 +121,11 @@ openstbench
 Example:
 
 ```python
-from openstbench import TranslationEvaluator, SpeechQualityEvaluator
+from openstbench import (
+    TranslationEvaluator,
+    SpeechQualityEvaluator,
+    TemporalConsistencyEvaluator,
+)
 ```
 
 ## Quick Start
@@ -134,6 +140,7 @@ Python examples:
 - `examples/python/emotion_eval.py`
 - `examples/python/paralinguistic_eval.py`
 - `examples/python/paralinguistic_identity_baseline.py`
+- `examples/python/temporal_consistency_eval.py`
 - `examples/python/latency_eval.py`
 
 Shell examples:
@@ -141,6 +148,23 @@ Shell examples:
 - `examples/bash/install_extras.sh`
 - `examples/bash/run_latency_cli.sh`
 
+Minimal temporal consistency example:
+
+```python
+from openstbench import TemporalConsistencyEvaluator
+
+evaluator = TemporalConsistencyEvaluator(
+    thresholds=(0.2, 0.4),
+)
+
+results, diagnostics = evaluator.evaluate_all(
+    source_audio="./source_wavs",
+    target_audio="./generated_wavs",
+    sample_ids=["sample_1", "sample_2"],
+    return_diagnostics=True,
+)
+```
+
 Latency output distinguishes two RTF variants:
 
 - `Real_Time_Factor_(RTF)`: system-level RTF. This includes agent policy overhead, pre/post-processing, and other runtime costs around model inference.
@@ -166,6 +190,8 @@ Common audio inputs support:
 - For `zh` / `ja` / `ko`, the toolkit uses CJK-aware handling for text-side evaluation.
 - `SpeechQualityEvaluator` returns `CER_Consistency` for `zh` / `ja` / `ko`, and `WER_Consistency` for most other languages.
 - `ParalinguisticEvaluator` always supports `Paralinguistic_Fidelity_Cosine`, a continuous CLAP-based audio similarity score between source and target speech.
+- `TemporalConsistencyEvaluator` supports `List[str]`, audio folders, `.txt` path lists, and `.json` path lists for both `source_audio` and `target_audio`.
+- `TemporalConsistencyEvaluator` reports thresholded duration compliance metrics (`Duration_Consistency_SLC_*`).
 - The discrete preservation branch is an utterance-level single-label task. With source-side gold labels, it reports `Acoustic_Event_Preservation_Rate`, `Acoustic_Event_Preservation_Macro_F1`, and `Acoustic_Event_Preservation_Macro_Recall`.
 - If `source_onsets_ms` are available, the evaluator can also report alignment-aware metrics: `Event_Aligned_Preservation_Rate` and `Conditional_Relative_Onset_Error`.
 - Alignment is computed on relative onset position, not absolute wall-clock time. This makes it suitable for cross-lingual S2ST where source and target utterance durations naturally differ.
@@ -176,6 +202,11 @@ Common audio inputs support:
 - The default event localizer is also replaceable. Custom localizers only need to implement `localize(audio_paths, labels, candidate_labels)`.
 - Dataset-specific label mapping is intentionally outside the core package. Pass `candidate_labels` and `label_normalizer` at call time so the same evaluator works across datasets without changing core code.
 - For offline environments, `clap_model_path` accepts either a Hugging Face repo id or a local model directory or snapshot.
+- Model-loading parameters such as `clap_model_path`, `wavlm_model_path`, `whisper_model`, `e2v_model_path`, `comet_model`, and `bleurt_path` now use a consistent local-first rule: if the supplied local path exists, it is used; otherwise the evaluator falls back to the default remote model id.
 - In S2S latency evaluation, alignment prefers the model's native transcript when available. If the model is audio-only, the evaluator can optionally use ASR fallback to prepare alignment text.
 - For S2S forced alignment, pass language-appropriate MFA models through `alignment_acoustic_model` and `alignment_dictionary_model`. The defaults are English.
 - Some modules rely on optional dependencies or local model paths in offline environments.
+
+## License
+
+MIT License

{openstbench-0.2.0 → openstbench-0.3.0}/src/OpenSTBench.egg-info/SOURCES.txt

@@ -6,11 +6,13 @@ src/OpenSTBench.egg-info/dependency_links.txt
 src/OpenSTBench.egg-info/requires.txt
 src/OpenSTBench.egg-info/top_level.txt
 src/openstbench/__init__.py
+src/openstbench/_model_loading.py
 src/openstbench/dataset.py
 src/openstbench/emotion_evaluator.py
 src/openstbench/paralinguistic_evaluator.py
 src/openstbench/speaker_similarity_evaluator.py
 src/openstbench/speech_quality_evaluator.py
+src/openstbench/temporal_consistency_evaluator.py
 src/openstbench/translation_evaluator.py
 src/openstbench/latency/__init__.py
 src/openstbench/latency/agent.py

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/__init__.py

@@ -9,7 +9,7 @@ actually accessed.
 from importlib import import_module
 from typing import Dict, Tuple
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     "TranslationEvaluator",
@@ -31,6 +31,7 @@ __all__ = [
     "build_paralinguistic_inputs",
     "evaluate_paralinguistic_dataset",
     "SpeechQualityEvaluator",
+    "TemporalConsistencyEvaluator",
     "LatencyEvaluator",
     "SpeakerSimilarityEvaluator",
     "GenericAgent",
@@ -79,6 +80,7 @@ _EXPORT_SPECS: Tuple[Tuple[str, Tuple[str, ...]], ...] = (
         ),
     ),
     ("speech_quality_evaluator", ("SpeechQualityEvaluator",)),
+    ("temporal_consistency_evaluator", ("TemporalConsistencyEvaluator",)),
     ("speaker_similarity_evaluator", ("SpeakerSimilarityEvaluator",)),
     (
         "dataset",

openstbench-0.3.0/src/openstbench/_model_loading.py

@@ -0,0 +1,62 @@
+from pathlib import Path
+from typing import Optional, Tuple
+
+
+_LOCAL_MODEL_FILE_SUFFIXES = {
+    ".bin",
+    ".ckpt",
+    ".onnx",
+    ".pb",
+    ".pt",
+    ".pth",
+    ".safetensors",
+}
+
+
+def _is_explicit_local_reference(model_source: str) -> bool:
+    normalized = model_source.strip().replace("\\", "/")
+    if not normalized:
+        return False
+
+    if normalized.startswith(("./", "../", "~/", "/")):
+        return True
+
+    if len(normalized) >= 3 and normalized[1] == ":" and normalized[2] == "/":
+        return True
+
+    if "\\" in model_source:
+        return True
+
+    return Path(model_source).suffix.lower() in _LOCAL_MODEL_FILE_SUFFIXES
+
+
+def resolve_pretrained_source(
+    preferred_source: Optional[str],
+    *,
+    fallback_source: Optional[str] = None,
+) -> Tuple[str, str]:
+    """Resolve a model source with local-first behavior.
+
+    Returns:
+        Tuple[str, str]: (resolved_source, source_kind), where source_kind is
+        either "local" or "remote".
+    """
+
+    if preferred_source:
+        candidate = Path(preferred_source).expanduser()
+        if candidate.exists():
+            return str(candidate.resolve()), "local"
+
+        if not _is_explicit_local_reference(preferred_source):
+            return preferred_source, "remote"
+
+    if fallback_source:
+        fallback_candidate = Path(fallback_source).expanduser()
+        if fallback_candidate.exists():
+            return str(fallback_candidate.resolve()), "local"
+        return fallback_source, "remote"
+
+    if preferred_source:
+        return preferred_source, "remote"
+
+    raise ValueError("At least one model source must be provided.")

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/emotion_evaluator.py

@@ -7,6 +7,8 @@ from typing import List, Dict, Optional, Union
 from tqdm import tqdm
 from numpy.linalg import norm
 
+from ._model_loading import resolve_pretrained_source
+
 def _load_data_list(
     input_data: Union[str, List[str]], 
     name: str,
@@ -97,7 +99,12 @@ class EmotionEvaluator:
         print(f"⏳ 正在加载 Emotion2Vec+ 大模型: {self.e2v_model_path}")
         try:
             from funasr import AutoModel
-            self.e2v_model = AutoModel(model=self.e2v_model_path, device=self.device, disable_update=True)
+            model_source, source_kind = resolve_pretrained_source(
+                self.e2v_model_path,
+                fallback_source=self.DEFAULT_E2V_MODEL,
+            )
+            print(f"Loading Emotion2Vec+ ({source_kind}) from {model_source}...")
+            self.e2v_model = AutoModel(model=model_source, device=self.device, disable_update=True)
             print("✅ Emotion2Vec+ 大模型加载成功！")
         except ImportError:
             print("❌ Emotion2Vec+ 依赖缺失。请执行: pip install funasr modelscope")
@@ -259,4 +266,4 @@ class EmotionEvaluator:
                 print("\n   [深度学习离散情感识别分类率] (数值越高代表预设标签识别越准)")
                 print(f"   - Audio Emotion Accuracy (离散情感识别准确率): {results['Audio_Emotion_Accuracy']:.2%}")
                 
-        return results
+        return results

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/latency/cli.py

@@ -104,14 +104,14 @@ class LatencyEvaluator:
         # 2. 智能化筛选最高级/最准确的一个版本展示给用户
         cleaned_results = {}
         
-        # [A] 评估: 第一声开口延迟 (StartOffset/ALAL)
+        # [A] 评估: 第一声开口延迟 (StartOffset)
         so_key = "StartOffset_CA" if computation_aware else "StartOffset"
         align_so_key = "StartOffset_SpeechAlign_CA" if computation_aware else "StartOffset_SpeechAlign"
         
         if results.get(align_so_key) is not None and results[align_so_key] > 0:
-            cleaned_results["First_Audio_Delay_(ALAL_ms)"] = results[align_so_key]
+            cleaned_results["First_Audio_Delay_(StartOffset_ms)"] = results[align_so_key]
         else:
-            cleaned_results["First_Audio_Delay_(ALAL_ms)"] = results.get(so_key, 0)
+            cleaned_results["First_Audio_Delay_(StartOffset_ms)"] = results.get(so_key, 0)
             
         # [B] 评估: 整句同传综合延迟 (标准版 ATD: 连同阅读/播放音频时间一起计算)
         atd_key = "ATD_CA" if computation_aware else "ATD"

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/latency/utils.py

@@ -7,6 +7,8 @@ from pathlib import Path
 import matplotlib.pyplot as plt
 import textgrid
 
+from .._model_loading import resolve_pretrained_source
+
 try:
     import whisper
 except ImportError:
@@ -98,10 +100,14 @@ def _load_whisper_model(model_name="medium", device=None):
         return None
     key = (model_name, device or "auto")
     if key not in _WHISPER_MODELS:
+        resolved_model_name, _source_kind = resolve_pretrained_source(
+            model_name,
+            fallback_source="medium",
+        )
         kwargs = {}
         if device:
             kwargs["device"] = device
-        _WHISPER_MODELS[key] = whisper.load_model(model_name, **kwargs)
+        _WHISPER_MODELS[key] = whisper.load_model(resolved_model_name, **kwargs)
     return _WHISPER_MODELS[key]
 
 

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/paralinguistic_evaluator.py

@@ -9,8 +9,11 @@ import torch
 import torchaudio
 from tqdm import tqdm
 
+from ._model_loading import resolve_pretrained_source
+
 
 LabelNormalizer = Optional[Union[Dict[str, Optional[str]], Callable[[str], Optional[str]]]]
+DEFAULT_CLAP_MODEL_SOURCE = "laion/clap-htsat-fused"
 
 
 @dataclass(frozen=True)
@@ -524,7 +527,7 @@ class ClapAudioEventPredictor(BaseAudioEventPredictor):
     def __init__(
         self,
         *,
-        model_path: str = "laion/clap-htsat-fused",
+        model_path: str = DEFAULT_CLAP_MODEL_SOURCE,
         config: Optional[EventPredictionConfig] = None,
         device: Optional[str] = None,
     ) -> None:
@@ -544,8 +547,13 @@ class ClapAudioEventPredictor(BaseAudioEventPredictor):
         except ImportError as exc:
             raise RuntimeError("CLAP-based event prediction requires `transformers`.") from exc
 
-        self._processor = ClapProcessor.from_pretrained(self.model_path)
-        self._model = ClapModel.from_pretrained(self.model_path).to(self.device).eval()
+        model_source, source_kind = resolve_pretrained_source(
+            self.model_path,
+            fallback_source=DEFAULT_CLAP_MODEL_SOURCE,
+        )
+        print(f"Loading CLAP ({source_kind}) from {model_source}...")
+        self._processor = ClapProcessor.from_pretrained(model_source)
+        self._model = ClapModel.from_pretrained(model_source).to(self.device).eval()
 
     @staticmethod
     def _normalize_embedding(embedding: np.ndarray) -> np.ndarray:
@@ -691,7 +699,7 @@ class ClapSlidingWindowEventLocalizer(BaseAudioEventLocalizer):
     def __init__(
         self,
         *,
-        model_path: str = "laion/clap-htsat-fused",
+        model_path: str = DEFAULT_CLAP_MODEL_SOURCE,
         prediction_config: Optional[EventPredictionConfig] = None,
         localization_config: Optional[EventLocalizationConfig] = None,
         device: Optional[str] = None,
@@ -801,7 +809,7 @@ class ClapSlidingWindowEventLocalizer(BaseAudioEventLocalizer):
 
 
 class ParalinguisticEvaluator:
-    DEFAULT_CLAP_MODEL = "laion/clap-htsat-fused"
+    DEFAULT_CLAP_MODEL = DEFAULT_CLAP_MODEL_SOURCE
 
     def __init__(
         self,
@@ -842,8 +850,13 @@ class ParalinguisticEvaluator:
         except ImportError as exc:
             raise RuntimeError("Paralinguistic_Fidelity_Cosine requires `transformers` to load CLAP.") from exc
 
-        self._clap_processor = ClapProcessor.from_pretrained(self.clap_model_path)
-        self._clap_model = ClapModel.from_pretrained(self.clap_model_path).to(self.device).eval()
+        model_source, source_kind = resolve_pretrained_source(
+            self.clap_model_path,
+            fallback_source=self.DEFAULT_CLAP_MODEL,
+        )
+        print(f"Loading CLAP ({source_kind}) from {model_source}...")
+        self._clap_processor = ClapProcessor.from_pretrained(model_source)
+        self._clap_model = ClapModel.from_pretrained(model_source).to(self.device).eval()
 
     def _extract_clap_embeddings(self, audio_paths: Sequence[str]) -> List[np.ndarray]:
         self._load_clap()

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/speaker_similarity_evaluator.py

@@ -5,6 +5,10 @@ import os
 import librosa
 from tqdm import tqdm
 
+from ._model_loading import resolve_pretrained_source
+
+DEFAULT_WAVLM_MODEL_SOURCE = "microsoft/wavlm-base-plus-sv"
+
 # 如果你想用本地的 Resemblyzer，确保之前项目能 import 到
 try:
     from resemblyzer import preprocess_wav, VoiceEncoder
@@ -26,7 +30,7 @@ class SpeakerSimilarityEvaluator:
     def __init__(self, 
                  model_type="wavlm", 
                  device=None, 
-                 wavlm_model_path="microsoft/wavlm-base-plus-sv",
+                 wavlm_model_path=DEFAULT_WAVLM_MODEL_SOURCE,
                  resemblyzer_weights_path="pretrained.pt"):
         """
         初始化打分器，仅在初始化时加载一次模型。
@@ -50,9 +54,13 @@ class SpeakerSimilarityEvaluator:
         if self.model_type in ["wavlm", "both"]:
             if not TRANSFORMERS_AVAILABLE:
                 raise ImportError("Please install transformers to use WavLM: pip install transformers")
-            print(f"Loading WavLM from {wavlm_model_path}...")
-            self.wavlm_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(wavlm_model_path)
-            self.wavlm_model = WavLMForXVector.from_pretrained(wavlm_model_path).to(self.device).eval()
+            model_source, source_kind = resolve_pretrained_source(
+                wavlm_model_path,
+                fallback_source=DEFAULT_WAVLM_MODEL_SOURCE,
+            )
+            print(f"Loading WavLM ({source_kind}) from {model_source}...")
+            self.wavlm_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_source)
+            self.wavlm_model = WavLMForXVector.from_pretrained(model_source).to(self.device).eval()
 
         if self.model_type in ["resemblyzer", "both"]:
             if not RESEMBLYZER_AVAILABLE:
@@ -144,4 +152,4 @@ class SpeakerSimilarityEvaluator:
         if res_scores:
             batch_results["average_resemblyzer_similarity"] = sum(res_scores) / len(res_scores)
             
-        return batch_results
+        return batch_results

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/speech_quality_evaluator.py

@@ -7,6 +7,7 @@ from typing import List, Dict, Optional, Union
 from tqdm import tqdm
 
 # 复用现有的加载工具
+from ._model_loading import resolve_pretrained_source
 from .translation_evaluator import load_text_from_file_or_list, load_audio_from_folder
 
 try:
@@ -15,6 +16,7 @@ except ImportError:
     whisper = None
 
 class SpeechQualityEvaluator:
+    DEFAULT_WHISPER_MODEL = "medium"
     """
     语音质量与一致性评测器
     专门用于：
@@ -24,7 +26,7 @@ class SpeechQualityEvaluator:
     def __init__(self, 
                  use_wer: bool = True,
                  use_utmos: bool = True,
-                 whisper_model: str = "medium",
+                 whisper_model: str = DEFAULT_WHISPER_MODEL,
                  utmos_model_path: Optional[str] = None,
                  utmos_ckpt_path: Optional[str] = None,
                  device: Optional[str] = None):
@@ -56,7 +58,12 @@ class SpeechQualityEvaluator:
                 self.use_wer = False
                 return
             print(f"⏳ 正在加载 Whisper ({self.whisper_model_name})...")
-            self.whisper_model = whisper.load_model(self.whisper_model_name, device=self.device)
+            model_source, source_kind = resolve_pretrained_source(
+                self.whisper_model_name,
+                fallback_source=self.DEFAULT_WHISPER_MODEL,
+            )
+            print(f"Loading Whisper ({source_kind}) from {model_source}...")
+            self.whisper_model = whisper.load_model(model_source, device=self.device)
 
     def _load_utmos(self):
         if self.utmos_model is None and self.use_utmos:
@@ -168,4 +175,4 @@ class SpeechQualityEvaluator:
                 else:
                     print("   ⚠️ 提供的生成文本清洗后为空，无法计算分布错误率。")
 
-        return results
+        return results

openstbench-0.3.0/src/openstbench/temporal_consistency_evaluator.py

@@ -0,0 +1,219 @@
+import json
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
+
+import torchaudio
+
+
+class TemporalConsistencyEvaluator:
+    """Evaluate source-target temporal consistency at the utterance level."""
+
+    def __init__(
+        self,
+        thresholds: Sequence[float] = (0.2, 0.4),
+    ) -> None:
+        if not thresholds:
+            raise ValueError("thresholds must contain at least one value.")
+
+        normalized_thresholds: List[float] = []
+        for threshold in thresholds:
+            try:
+                value = float(threshold)
+            except (TypeError, ValueError) as exc:
+                raise ValueError("thresholds must be numeric.") from exc
+            if value < 0.0 or value >= 1.0:
+                raise ValueError("each threshold must satisfy 0 <= threshold < 1.")
+            normalized_thresholds.append(value)
+
+        self.thresholds = tuple(sorted(set(normalized_thresholds)))
+
+    def _load_audio_from_folder(self, folder_path: str, name: str) -> List[str]:
+        folder = Path(folder_path)
+        if not folder.exists():
+            raise FileNotFoundError(f"{name} not found: {folder_path}")
+        if not folder.is_dir():
+            raise ValueError(f"{name} must be a directory when using folder input: {folder_path}")
+
+        audio_files: List[Path] = []
+        for extension in (".wav", ".flac", ".mp3", ".ogg", ".m4a"):
+            audio_files.extend(folder.glob(f"*{extension}"))
+        audio_files = sorted(audio_files, key=lambda item: item.stem)
+        if not audio_files:
+            raise ValueError(f"No audio files found under {name}: {folder_path}")
+        return [str(item.resolve()) for item in audio_files]
+
+    def _resolve_audio_paths(self, values: Sequence[str], name: str) -> List[str]:
+        resolved: List[str] = []
+        for index, value in enumerate(values):
+            path = Path(str(value))
+            if not path.exists():
+                raise FileNotFoundError(f"{name}[{index}] not found: {value}")
+            resolved.append(str(path.resolve()))
+        return resolved
+
+    def _load_audio_list(self, data: Union[List[str], str], name: str) -> List[str]:
+        if isinstance(data, list):
+            return self._resolve_audio_paths(data, name)
+
+        if not isinstance(data, str):
+            raise ValueError(f"{name} must be a path or a list of paths.")
+
+        path = Path(data)
+        if not path.exists():
+            raise FileNotFoundError(f"{name} not found: {data}")
+
+        if path.is_dir():
+            return self._load_audio_from_folder(data, name)
+
+        suffix = path.suffix.lower()
+        if suffix == ".json":
+            with open(path, "r", encoding="utf-8") as file:
+                payload = json.load(file)
+
+            candidate_keys = ("audio", "path", "file", "wav", "mp3", "source_audio", "target_audio")
+            if isinstance(payload, list):
+                if not payload:
+                    return []
+                if isinstance(payload[0], str):
+                    return self._resolve_audio_paths(payload, name)
+                if isinstance(payload[0], dict):
+                    for key in candidate_keys:
+                        if key in payload[0]:
+                            return self._resolve_audio_paths([str(item[key]) for item in payload], name)
+                raise ValueError(f"{name} JSON list must contain strings or dicts with one of {candidate_keys}.")
+
+            if isinstance(payload, dict):
+                plural_keys = tuple(f"{key}s" for key in candidate_keys)
+                for key in plural_keys:
+                    if key in payload:
+                        return self._resolve_audio_paths(payload[key], name)
+                raise ValueError(f"{name} JSON object must contain one of {plural_keys}.")
+
+            raise ValueError(f"{name} JSON payload must be a list or dict.")
+
+        if suffix == ".txt":
+            with open(path, "r", encoding="utf-8") as file:
+                lines = [line.strip() for line in file if line.strip()]
+            return self._resolve_audio_paths(lines, name)
+
+        return self._resolve_audio_paths([data], name)
+
+    def _get_audio_duration_ms(self, audio_path: str) -> float:
+        try:
+            info = torchaudio.info(audio_path)
+            if info.sample_rate > 0 and info.num_frames >= 0:
+                return float(info.num_frames / info.sample_rate * 1000.0)
+        except Exception:
+            pass
+
+        waveform, sample_rate = torchaudio.load(audio_path)
+        if sample_rate <= 0:
+            return 0.0
+        return float(waveform.shape[-1] / sample_rate * 1000.0)
+
+    @staticmethod
+    def _threshold_suffix(threshold: float) -> str:
+        return f"{threshold:.1f}"
+
+    def _compute_metrics(
+        self,
+        source_durations_ms: Sequence[float],
+        target_durations_ms: Sequence[float],
+        sample_ids: Optional[Sequence[str]] = None,
+    ) -> Tuple[Dict[str, float], Dict[str, Any]]:
+        metrics: Dict[str, float] = {}
+        sample_records: List[Dict[str, Any]] = []
+        valid_ratios: List[float] = []
+        hit_counts = {threshold: 0 for threshold in self.thresholds}
+        skipped = 0
+
+        for index, (source_ms, target_ms) in enumerate(zip(source_durations_ms, target_durations_ms)):
+            sample_id = sample_ids[index] if sample_ids is not None else str(index)
+            record: Dict[str, Any] = {
+                "sample_index": int(index),
+                "sample_id": str(sample_id),
+                "source_duration_ms": round(float(source_ms), 4),
+                "target_duration_ms": round(float(target_ms), 4),
+            }
+
+            if source_ms <= 0.0 or target_ms <= 0.0:
+                skipped += 1
+                record["valid"] = False
+                record["skip_reason"] = "non_positive_duration"
+                sample_records.append(record)
+                continue
+
+            ratio = float(target_ms / source_ms)
+            valid_ratios.append(ratio)
+
+            slc_hits: Dict[str, bool] = {}
+            for threshold in self.thresholds:
+                hit = (1.0 - threshold) <= ratio <= (1.0 + threshold)
+                slc_hits[self._threshold_suffix(threshold)] = bool(hit)
+                if hit:
+                    hit_counts[threshold] += 1
+
+            record.update(
+                {
+                    "valid": True,
+                    "duration_ratio": round(ratio, 6),
+                    "slc_hits": slc_hits,
+                }
+            )
+            sample_records.append(record)
+
+        num_evaluated = len(valid_ratios)
+        for threshold in self.thresholds:
+            suffix = self._threshold_suffix(threshold)
+            score = float(hit_counts[threshold] / num_evaluated) if num_evaluated > 0 else 0.0
+            metrics[f"Duration_Consistency_SLC_{suffix}"] = round(score, 4)
+
+        diagnostics: Dict[str, Any] = {
+            "num_samples": len(source_durations_ms),
+            "num_evaluated": num_evaluated,
+            "num_skipped": skipped,
+            "thresholds": [round(float(value), 4) for value in self.thresholds],
+            "samples": sample_records,
+        }
+        return metrics, diagnostics
+
+    def evaluate_all(
+        self,
+        source_audio: Union[List[str], str],
+        target_audio: Union[List[str], str],
+        *,
+        sample_ids: Optional[Sequence[str]] = None,
+        verbose: bool = True,
+        return_diagnostics: bool = False,
+    ) -> Union[Tuple[Dict[str, float], Dict[str, Any]], Dict[str, float]]:
+        source_audio_paths = self._load_audio_list(source_audio, "source_audio")
+        target_audio_paths = self._load_audio_list(target_audio, "target_audio")
+
+        if len(source_audio_paths) != len(target_audio_paths):
+            raise ValueError(
+                f"Source and target size mismatch: {len(source_audio_paths)} vs {len(target_audio_paths)}"
+            )
+        if not source_audio_paths:
+            raise ValueError("No samples found for temporal consistency evaluation.")
+        if sample_ids is not None and len(sample_ids) != len(source_audio_paths):
+            raise ValueError(f"sample_ids size mismatch: {len(sample_ids)} vs {len(source_audio_paths)}")
+
+        source_durations_ms = [self._get_audio_duration_ms(path) for path in source_audio_paths]
+        target_durations_ms = [self._get_audio_duration_ms(path) for path in target_audio_paths]
+        metrics, diagnostics = self._compute_metrics(
+            source_durations_ms,
+            target_durations_ms,
+            sample_ids=sample_ids,
+        )
+
+        if verbose:
+            print("\n[TemporalConsistencyEvaluator] Summary")
+            print(f"  Samples: {diagnostics['num_samples']}")
+            print(f"  Evaluated: {diagnostics['num_evaluated']}")
+            print(f"  Skipped: {diagnostics['num_skipped']}")
+            for metric_name, score in metrics.items():
+                print(f"  {metric_name}: {score}")
+
+        if return_diagnostics:
+            return metrics, diagnostics
+        return metrics

{openstbench-0.2.0 → openstbench-0.3.0}/src/openstbench/translation_evaluator.py

@@ -10,6 +10,8 @@ import torch
 from typing import Dict, List, Optional, Union
 from pathlib import Path
 
+from ._model_loading import resolve_pretrained_source
+
 # ==================== 配置 ====================
 
 CACHE_PATHS = {
@@ -88,6 +90,7 @@ def load_audio_from_folder(folder_path: str, extensions: tuple = (".wav", ".mp3"
 
 # ==================== 评测器核心类 ====================
 
+DEFAULT_COMET_MODEL = "Unbabel/wmt22-comet-da"
 DEFAULT_BLEURT_MODEL = "lucadiliello/BLEURT-20"
 
 class TranslationEvaluator:
@@ -100,7 +103,7 @@ class TranslationEvaluator:
                  use_chrf: bool = True,
                  use_comet: bool = True,      
                  use_bleurt: bool = False,    
-                 comet_model: str = "Unbabel/wmt22-comet-da",
+                 comet_model: str = DEFAULT_COMET_MODEL,
                  bleurt_path: Optional[str] = None,
                  bleurt_model: Optional[str] = None,
                  device: Optional[str] = None):
@@ -144,15 +147,43 @@ class TranslationEvaluator:
             torch.cuda.empty_cache()
         gc.collect()
 
+    def _resolve_local_comet_checkpoint(self, model_name: str) -> Optional[str]:
+        candidate = Path(model_name).expanduser()
+        if not candidate.exists():
+            return None
+
+        if candidate.is_file():
+            return str(candidate.resolve())
+
+        common_ckpts = [
+            candidate / "checkpoints" / "model.ckpt",
+            candidate / "model.ckpt",
+        ]
+        for ckpt in common_ckpts:
+            if ckpt.exists() and ckpt.is_file():
+                return str(ckpt.resolve())
+        return None
+
     def _load_comet(self, model_name: str):
         if not download_model:
             print("⚠️ COMET 未安装，跳过")
             return None
         try:
-            cache = os.path.join(CACHE_PATHS["huggingface"], f"models--{model_name.replace('/', '--')}")
-            status = "[Local]" if os.path.exists(cache) else "[Online]"
-            print(f"⏳ {status} 加载 COMET: {model_name}")
-            model = load_from_checkpoint(download_model(model_name))
+            model_source, source_kind = resolve_pretrained_source(
+                model_name,
+                fallback_source=DEFAULT_COMET_MODEL,
+            )
+            local_ckpt = self._resolve_local_comet_checkpoint(model_source)
+            if local_ckpt is not None:
+                print(f"⏳ [Local] 加载 COMET: {local_ckpt}")
+                model = load_from_checkpoint(local_ckpt)
+            else:
+                remote_source = model_source if source_kind == "remote" else DEFAULT_COMET_MODEL
+                cache = os.path.join(CACHE_PATHS["huggingface"], f"models--{remote_source.replace('/', '--')}")
+                status = "[Local]" if os.path.exists(cache) else "[Online]"
+                print(f"⏳ {status} 加载 COMET: {model_name}")
+                print(f"Loading COMET ({status}) from {remote_source}")
+                model = load_from_checkpoint(download_model(remote_source))
             if self.device.startswith("cuda"):
                 model = model.to(self.device)
             return model
@@ -165,7 +196,13 @@ class TranslationEvaluator:
             print("⚠️ bleurt-pytorch 未安装，跳过加载")
             return
         
-        model_source = path if (path and os.path.exists(path)) else (model_name or DEFAULT_BLEURT_MODEL)
+        if path:
+            model_source, _source_kind = resolve_pretrained_source(
+                path,
+                fallback_source=model_name or DEFAULT_BLEURT_MODEL,
+            )
+        else:
+            model_source, _source_kind = resolve_pretrained_source(model_name or DEFAULT_BLEURT_MODEL)
         print(f"⏳ 加载 BLEURT: {model_source}")
         
         try:
@@ -254,4 +291,4 @@ class TranslationEvaluator:
             except Exception as e:
                 results["COMET"] = -1.0
         
-        return {k: round(v, 4) if v >= 0 else v for k, v in results.items()}
+        return {k: round(v, 4) if v >= 0 else v for k, v in results.items()}