extended-wer-normalizer 0.1.0__tar.gz

extended_wer_normalizer-0.1.0/.claude/settings.local.json

@@ -0,0 +1,7 @@
+{
+  "permissions": {
+    "allow": [
+      "WebFetch(domain:jitsi.github.io)"
+    ]
+  }
+}

extended_wer_normalizer-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv

extended_wer_normalizer-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

extended_wer_normalizer-0.1.0/PKG-INFO

@@ -0,0 +1,111 @@
+Metadata-Version: 2.4
+Name: extended-wer-normalizer
+Version: 0.1.0
+Summary: jiwer-compatible WER normalizer with number, email, URL, filler, and symbol normalization for voice AI evaluation
+Project-URL: Homepage, https://github.com/ai-coustics/extended-wer-normalizer
+License: MIT
+Keywords: asr,jiwer,normalization,speech recognition,voice ai,wer
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: jiwer>=3.0
+Requires-Dist: num2words>=0.5
+Requires-Dist: whisper-normalizer>=0.1.12
+Description-Content-Type: text/markdown
+
+# extended-wer-normalizer
+
+jiwer-compatible text normalizer for Word Error Rate (WER) evaluation in voice AI.
+
+Extends [jiwer](https://github.com/jitsi/jiwer)'s built-in transforms with normalizations that matter for real-world ASR evaluation: phone numbers, emails, URLs, currency, percentages, ordinals, filler words, and stuttering.
+
+## Installation
+
+```bash
+pip install extended-wer-normalizer
+```
+
+## Quick start
+
+```python
+from extended_wer_normalizer import normalize_for_wer
+
+normalize_for_wer("Call 0176 or email info@example.com, it costs $5.99")
+# → "call 0 1 7 6 or email info at example dot com it costs five dollars ninety nine cents"
+
+normalize_for_wer("Um, 1st place goes to Dr. Smith with 50% accuracy")
+# → "first place goes to doctor smith with fifty percent accuracy"
+```
+
+## jiwer integration
+
+Every normalization is a `jiwer.AbstractTransform` subclass — compose them freely:
+
+```python
+import jiwer
+from extended_wer_normalizer.transforms import NormalizeEmails, ExpandDigitRuns
+
+pipeline = jiwer.Compose([
+    NormalizeEmails(),
+    ExpandDigitRuns(),
+    jiwer.ToLowerCase(),
+    jiwer.RemovePunctuation(),
+    jiwer.ReduceToListOfListOfWords(),
+])
+
+wer = jiwer.wer("info at example dot com", "info@example.com", hypothesis_transform=pipeline)
+```
+
+Use the pre-built pipeline directly with `jiwer.wer`:
+
+```python
+import jiwer
+from extended_wer_normalizer import english_wer_pipeline
+
+wer = jiwer.wer(reference, hypothesis, reference_transform=english_wer_pipeline, hypothesis_transform=english_wer_pipeline)
+```
+
+## Available transforms
+
+| Transform | Example |
+|---|---|
+| `ExpandDigitRuns` | `"0176"` → `"0 1 7 6"` |
+| `DigitWordsToChars` | `"zero one seven"` → `"0 1 7"` |
+| `WhisperEnglishNormalize` | lowercase, punctuation, contractions, compound numbers |
+| `WhisperBasicNormalize` | language-agnostic basic normalization |
+| `FinalDigitWordCleanup` | residual digit-word sweep after compound resolution |
+| `NormalizeEmails` | `"user@example.com"` → `"user at example dot com"` |
+| `NormalizeURLs` | `"https://example.com/path"` → `"example dot com"` |
+| `NormalizeCurrency` | `"$5.99"` → `"five dollars ninety nine cents"` |
+| `NormalizePercentages` | `"50%"` → `"fifty percent"` |
+| `NormalizeOrdinals` | `"1st"` → `"first"`, `"15th"` → `"fifteenth"` |
+| `ExpandAbbreviations` | `"Dr."` → `"doctor"`, `"vs."` → `"versus"` |
+| `NormalizeSymbols` | `"cats & dogs"` → `"cats and dogs"` |
+| `RemoveFillerWords` | removes `um`, `uh`, `hmm`, `er`, `ah`, … |
+| `CollapseRepetitions` | `"I I I think"` → `"I think"` |
+
+## Pipeline design
+
+The English pipeline applies transforms in an order that ensures idempotence and correct interaction with Whisper's `EnglishTextNormalizer`:
+
+1. **Pre-Whisper**: email, URL, symbol, abbreviation (patterns Whisper would mangle)
+2. **Digit preparation**: expand digit runs, convert digit words
+3. **Core**: `WhisperEnglishNormalize` (lowercase, punctuation, contractions, compound numbers → digits)
+4. **Post-Whisper**: digit run expansion, digit word cleanup, currency, percentage, ordinal (patterns Whisper preserves or compacts)
+5. **Cleanup**: filler words, repetition collapse
+
+## Non-English
+
+For non-English text, pass `language` to get Whisper's `BasicTextNormalizer`:
+
+```python
+normalize_for_wer("Das kostet fünf Euro", language="de")
+```

extended_wer_normalizer-0.1.0/README.md

@@ -0,0 +1,88 @@
+# extended-wer-normalizer
+
+jiwer-compatible text normalizer for Word Error Rate (WER) evaluation in voice AI.
+
+Extends [jiwer](https://github.com/jitsi/jiwer)'s built-in transforms with normalizations that matter for real-world ASR evaluation: phone numbers, emails, URLs, currency, percentages, ordinals, filler words, and stuttering.
+
+## Installation
+
+```bash
+pip install extended-wer-normalizer
+```
+
+## Quick start
+
+```python
+from extended_wer_normalizer import normalize_for_wer
+
+normalize_for_wer("Call 0176 or email info@example.com, it costs $5.99")
+# → "call 0 1 7 6 or email info at example dot com it costs five dollars ninety nine cents"
+
+normalize_for_wer("Um, 1st place goes to Dr. Smith with 50% accuracy")
+# → "first place goes to doctor smith with fifty percent accuracy"
+```
+
+## jiwer integration
+
+Every normalization is a `jiwer.AbstractTransform` subclass — compose them freely:
+
+```python
+import jiwer
+from extended_wer_normalizer.transforms import NormalizeEmails, ExpandDigitRuns
+
+pipeline = jiwer.Compose([
+    NormalizeEmails(),
+    ExpandDigitRuns(),
+    jiwer.ToLowerCase(),
+    jiwer.RemovePunctuation(),
+    jiwer.ReduceToListOfListOfWords(),
+])
+
+wer = jiwer.wer("info at example dot com", "info@example.com", hypothesis_transform=pipeline)
+```
+
+Use the pre-built pipeline directly with `jiwer.wer`:
+
+```python
+import jiwer
+from extended_wer_normalizer import english_wer_pipeline
+
+wer = jiwer.wer(reference, hypothesis, reference_transform=english_wer_pipeline, hypothesis_transform=english_wer_pipeline)
+```
+
+## Available transforms
+
+| Transform | Example |
+|---|---|
+| `ExpandDigitRuns` | `"0176"` → `"0 1 7 6"` |
+| `DigitWordsToChars` | `"zero one seven"` → `"0 1 7"` |
+| `WhisperEnglishNormalize` | lowercase, punctuation, contractions, compound numbers |
+| `WhisperBasicNormalize` | language-agnostic basic normalization |
+| `FinalDigitWordCleanup` | residual digit-word sweep after compound resolution |
+| `NormalizeEmails` | `"user@example.com"` → `"user at example dot com"` |
+| `NormalizeURLs` | `"https://example.com/path"` → `"example dot com"` |
+| `NormalizeCurrency` | `"$5.99"` → `"five dollars ninety nine cents"` |
+| `NormalizePercentages` | `"50%"` → `"fifty percent"` |
+| `NormalizeOrdinals` | `"1st"` → `"first"`, `"15th"` → `"fifteenth"` |
+| `ExpandAbbreviations` | `"Dr."` → `"doctor"`, `"vs."` → `"versus"` |
+| `NormalizeSymbols` | `"cats & dogs"` → `"cats and dogs"` |
+| `RemoveFillerWords` | removes `um`, `uh`, `hmm`, `er`, `ah`, … |
+| `CollapseRepetitions` | `"I I I think"` → `"I think"` |
+
+## Pipeline design
+
+The English pipeline applies transforms in an order that ensures idempotence and correct interaction with Whisper's `EnglishTextNormalizer`:
+
+1. **Pre-Whisper**: email, URL, symbol, abbreviation (patterns Whisper would mangle)
+2. **Digit preparation**: expand digit runs, convert digit words
+3. **Core**: `WhisperEnglishNormalize` (lowercase, punctuation, contractions, compound numbers → digits)
+4. **Post-Whisper**: digit run expansion, digit word cleanup, currency, percentage, ordinal (patterns Whisper preserves or compacts)
+5. **Cleanup**: filler words, repetition collapse
+
+## Non-English
+
+For non-English text, pass `language` to get Whisper's `BasicTextNormalizer`:
+
+```python
+normalize_for_wer("Das kostet fünf Euro", language="de")
+```

extended_wer_normalizer-0.1.0/main.py

@@ -0,0 +1,6 @@
+def main():
+    print("Hello from aic-wer-normalizer!")
+
+
+if __name__ == "__main__":
+    main()

extended_wer_normalizer-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/extended_wer_normalizer"]
+
+[project]
+name = "extended-wer-normalizer"
+version = "0.1.0"
+description = "jiwer-compatible WER normalizer with number, email, URL, filler, and symbol normalization for voice AI evaluation"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+keywords = ["wer", "speech recognition", "asr", "normalization", "jiwer", "voice ai"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Multimedia :: Sound/Audio :: Speech",
+]
+dependencies = [
+    "jiwer>=3.0",
+    "whisper-normalizer>=0.1.12",
+    "num2words>=0.5",
+]
+
+[project.urls]
+Homepage = "https://github.com/ai-coustics/extended-wer-normalizer"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.3",
+]

extended_wer_normalizer-0.1.0/src/extended_wer_normalizer/__init__.py

@@ -0,0 +1,83 @@
+"""extended-wer-normalizer: jiwer-compatible WER normalization for voice AI."""
+
+from __future__ import annotations
+
+import jiwer
+
+from .transforms import (
+    CollapseRepetitions,
+    DigitWordsToChars,
+    ExpandAbbreviations,
+    ExpandDigitRuns,
+    FinalDigitWordCleanup,
+    NormalizeCurrency,
+    NormalizeEmails,
+    NormalizeOrdinals,
+    NormalizePercentages,
+    NormalizeSymbols,
+    NormalizeURLs,
+    RemoveFillerWords,
+    WhisperBasicNormalize,
+    WhisperEnglishNormalize,
+)
+
+__all__ = [
+    "normalize_for_wer",
+    "english_wer_pipeline",
+    "CollapseRepetitions",
+    "DigitWordsToChars",
+    "ExpandAbbreviations",
+    "ExpandDigitRuns",
+    "FinalDigitWordCleanup",
+    "NormalizeCurrency",
+    "NormalizeEmails",
+    "NormalizeOrdinals",
+    "NormalizePercentages",
+    "NormalizeSymbols",
+    "NormalizeURLs",
+    "RemoveFillerWords",
+    "WhisperBasicNormalize",
+    "WhisperEnglishNormalize",
+]
+
+english_wer_pipeline = jiwer.Compose([
+    # Pre-WhisperEN: handle patterns WhisperEN would mangle or remove
+    NormalizeEmails(),        # must be first: WhisperEN strips @ and dots in emails
+    NormalizeURLs(),          # WhisperEN strips slashes and scheme prefixes
+    NormalizeSymbols(),       # WhisperEN removes & and + as punctuation
+    ExpandAbbreviations(),    # expand Dr./Mr. before WhisperEN lowercases
+    # Core Whisper normalization (lowercase, punctuation, contractions, compound numbers)
+    ExpandDigitRuns(),        # preserve leading zeros before WhisperEN sees e.g. "0176"
+    DigitWordsToChars(),      # "zero one seven" → "0 1 7" before compound resolution
+    WhisperEnglishNormalize(),
+    # Post-WhisperEN: handle patterns WhisperEN preserves as-is or converts to compact form
+    # (WhisperEN preserves $5.99, 50%, 1st; and converts spoken forms back to compact)
+    ExpandDigitRuns(),        # split any remaining digit runs (e.g. from compound numbers)
+    FinalDigitWordCleanup(),  # final digit-word sweep — must run before currency/% so
+                              # it doesn't convert "nine" inside "ninety nine cents"
+    NormalizeCurrency(),      # $5.99 → five dollars ninety nine cents
+    NormalizePercentages(),   # 50% → fifty percent
+    NormalizeOrdinals(),      # 1st → first
+    RemoveFillerWords(),
+    CollapseRepetitions(),
+])
+
+
+def normalize_for_wer(text: str, language: str = "en") -> str:
+    """Normalize text for WER comparison.
+
+    English pipeline applies, in order:
+      email → URL → symbol → currency → percentage → ordinal → abbreviation
+      → digit-run expansion → digit-word-to-char → Whisper EnglishTextNormalizer
+      → digit-run expansion → final digit cleanup → filler removal → repetition collapse
+
+    All number representations collapse to space-separated single digits, so
+    "0176", "zero one seven six", and "21" all compare correctly.
+
+    For non-English text, Whisper's BasicTextNormalizer is applied instead.
+    """
+    if language == "en":
+        result = english_wer_pipeline([text])
+        return result[0].strip()
+    from whisper_normalizer.basic import BasicTextNormalizer
+    return BasicTextNormalizer()(text).strip()