phonebox 0.1.0__tar.gz

phonebox-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021-2024 Kevin Lenzo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

phonebox-0.1.0/MANIFEST.in

@@ -0,0 +1,19 @@
+# Include package data
+include README.md
+include LICENSE
+include requirements.txt
+include requirements-dev.txt
+
+# Include all configuration files
+recursive-include g2p/config *.json *.xlit
+
+# Include examples
+recursive-include examples *.py
+
+# Include tests
+recursive-include tests *.py
+
+# Exclude build artifacts
+global-exclude __pycache__
+global-exclude *.py[co]
+global-exclude .DS_Store

phonebox-0.1.0/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: phonebox
+Version: 0.1.0
+Summary: Fast, lightweight grapheme-to-phoneme conversion using decision trees
+Author: Kevin Lenzo (lenzo-ka)
+License: MIT
+Project-URL: Homepage, https://github.com/lenzo-ka/phonebox
+Project-URL: Bug Reports, https://github.com/lenzo-ka/phonebox/issues
+Project-URL: Source, https://github.com/lenzo-ka/phonebox
+Keywords: g2p,grapheme-to-phoneme,pronunciation,phonetics,speech,nlp,phonebox
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: cartlet>=0.5.0
+Requires-Dist: icukit>=0.1.2
+Provides-Extra: icu
+Requires-Dist: PyICU>=2.11; extra == "icu"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: pre-commit>=3.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: sphinx>=5.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
+Requires-Dist: myst-parser>=0.18; extra == "docs"
+Provides-Extra: viz
+Requires-Dist: graphviz>=0.20; extra == "viz"
+Requires-Dist: matplotlib>=3.5.0; extra == "viz"
+Dynamic: license-file
+
+# Phonebox: Grapheme-to-Phoneme Conversion
+
+Fast, lightweight grapheme-to-phoneme (G2P) conversion using decision trees and
+EM alignment. The package also includes **MultigramG2P** (n:m joint Viterbi) in
+the library; 1:1 CLI commands use the decision tree path.
+
+Decision-tree G2P is fast, compact, and interpretable; it trades some accuracy
+for those properties, and neural G2P methods will generally score better.
+CMUdict / PocketSphinx workflows are the main English use case; measured phone
+error rates and IPA locale benchmarks are in
+[`docs/G2P_EVAL.md`](docs/G2P_EVAL.md).
+
+## Features
+
+- **Measured accuracy**: reported phone error rates on CMUdict (see docs/BENCHMARKS.md); neural G2P methods can be more accurate
+- **Compact Models**: < 1MB model size for typical 1:1 trees
+- **MultigramG2P**: n:m joint EM + Viterbi decode (`MultigramG2P` in Python API)
+- **Zero Dependencies**: Bundled Python executable works standalone
+- **CMUdict Support**: English pronunciation with PocketSphinx compatibility
+
+## Installation
+
+```bash
+pip install phonebox
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/lenzo-ka/phonebox.git
+cd phonebox
+pip install -e .
+```
+
+## Quick Start
+
+### One Command
+
+```bash
+# Build G2P from CMUdict, bundle as Python executable
+phonebox recipe cmudict pocketsphinx -o g2p.py
+
+# Use it
+python g2p.py "Hello, world!"
+```
+
+### TTS Preset (keeps stress)
+
+```bash
+phonebox recipe cmudict tts -o g2p.py
+```
+
+## Using Bundled G2P
+
+### Command Line
+
+```bash
+python g2p.py "Hello, world!"
+# hello   HH AH L OW
+# world   W ER L D
+
+# Raw mode (no text normalization)
+python g2p.py -r "Hello,"
+```
+
+### Python Library
+
+```python
+from g2p import G2PPredictor
+
+g2p = G2PPredictor.from_embedded()
+phones = g2p.pronounce("hello")  # ['HH', 'AH', 'L', 'OW']
+
+# Process text (tokenizes automatically)
+for word, phones in g2p.pronounce_text("Hello, world!"):
+    print(f"{word}: {' '.join(phones)}")
+```
+
+## CLI Commands
+
+```
+Quick Start:
+  recipe       Build complete G2P from dictionary (one command)
+
+Using Models:
+  pronounce    Get pronunciations for words
+  normalize    Preview text normalization/tokenization
+  bundle       Create standalone executable with embedded model
+
+Building Models:
+  model        Model operations (build, train, convert, benchmark)
+  dict         Dictionary operations (fetch, export-vectors)
+
+Low-Level:
+  align        Align letters to phonemes (EM algorithm)
+  vectorize    Convert alignments to feature vectors
+
+Quality:
+  check          Validate lexicon against phoneset
+  suggest-joins  Discover join candidates (multigram EM)
+  compare        1:1 vs n:m eval (locale or all IPA locales)
+  train-multigram  Train/export MultigramG2P
+```
+
+### G2P evaluation (IPA locales)
+
+```bash
+phonebox compare all                    # docs/G2P_COMPARE.md
+phonebox compare locale --lexicon … --locale it_IT
+phonebox train-multigram --locale it_IT --lexicon it_ipa.tsv -o model.g2p.gz
+```
+
+Repo wrappers: `compare_g2p_all.py`, `compare_g2p_sweep.py`, `dump_units.py`.
+See [`docs/G2P_EVAL.md`](docs/G2P_EVAL.md).
+
+### Examples
+
+```bash
+# Preview text normalization
+phonebox normalize "Hello, world!"
+
+# Pronounce with existing model
+phonebox pronounce hello world -m model.g2p.gz
+
+# Benchmark model
+phonebox model benchmark model.g2p.gz
+
+# Fetch dictionary manually
+phonebox dict fetch cmudict
+```
+
+## Python API
+
+```python
+from phonebox import G2P, MultigramG2P
+
+# 1:1 decision tree (CLI: phonebox pronounce)
+g2p = G2P(model="model.g2p.gz")
+phones = g2p.pronounce("hello")
+print(phones)  # ['HH', 'AH', 'L', 'OW']
+
+# n:m multigram (train via CLI or library)
+mg = MultigramG2P(max_letter_span=2, max_phone_span=2)
+mg.train_from_dict("lexicon.tsv")
+# phonebox train-multigram … ; phonebox pronounce -m model.g2p.gz (sidecar auto-detect)
+
+# N-best alternatives
+for pron, score in g2p.pronounce_nbest("read", n=3):
+    print(f"{pron} ({score:.3f})")
+```
+
+## Step-by-Step Training
+
+For debugging or custom workflows:
+
+```bash
+# 1. Fetch dictionary
+phonebox dict fetch cmudict
+
+# 2. Align letters to phonemes
+phonebox align data/cmudict/cmudict.dict -o alignments.txt --remove-stress
+
+# 3. Vectorize alignments
+phonebox vectorize alignments.txt -o vectors.txt
+
+# 4. Train from vectors
+phonebox model train en_US --vectors vectors.txt -o model.g2p.gz
+
+# 5. Bundle
+phonebox bundle model.g2p.gz -o g2p.py
+```
+
+## Standalone Deployment
+
+Bundled files have zero dependencies beyond the Python standard library:
+
+```bash
+phonebox bundle model.g2p.gz -o g2p.py
+python g2p.py "test"
+```
+
+## Algorithm
+
+1. **EM Alignment**: Expectation-Maximization aligns letters to phonemes
+2. **Feature Extraction**: 7-gram letter windows create feature vectors
+3. **Decision Tree**: ID3-style tree trained on aligned data
+4. **Prediction**: Tree traversal based on letter context
+
+Based on research from CMU:
+- [CMU G2P Research](http://www.cs.cmu.edu/afs/cs.cmu.edu/user/lenzo/html/areas/t2p/)
+- [ICSLP 1998 Paper](https://www.isca-speech.org/archive/icslp_1998/i98_0561.html)
+
+## License
+
+MIT License - see LICENSE file.
+
+## Author
+
+Kevin Lenzo ([@lenzo-ka](https://github.com/lenzo-ka))
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Add tests for new functionality
+4. Submit a pull request

phonebox-0.1.0/README.md

@@ -0,0 +1,206 @@
+# Phonebox: Grapheme-to-Phoneme Conversion
+
+Fast, lightweight grapheme-to-phoneme (G2P) conversion using decision trees and
+EM alignment. The package also includes **MultigramG2P** (n:m joint Viterbi) in
+the library; 1:1 CLI commands use the decision tree path.
+
+Decision-tree G2P is fast, compact, and interpretable; it trades some accuracy
+for those properties, and neural G2P methods will generally score better.
+CMUdict / PocketSphinx workflows are the main English use case; measured phone
+error rates and IPA locale benchmarks are in
+[`docs/G2P_EVAL.md`](docs/G2P_EVAL.md).
+
+## Features
+
+- **Measured accuracy**: reported phone error rates on CMUdict (see docs/BENCHMARKS.md); neural G2P methods can be more accurate
+- **Compact Models**: < 1MB model size for typical 1:1 trees
+- **MultigramG2P**: n:m joint EM + Viterbi decode (`MultigramG2P` in Python API)
+- **Zero Dependencies**: Bundled Python executable works standalone
+- **CMUdict Support**: English pronunciation with PocketSphinx compatibility
+
+## Installation
+
+```bash
+pip install phonebox
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/lenzo-ka/phonebox.git
+cd phonebox
+pip install -e .
+```
+
+## Quick Start
+
+### One Command
+
+```bash
+# Build G2P from CMUdict, bundle as Python executable
+phonebox recipe cmudict pocketsphinx -o g2p.py
+
+# Use it
+python g2p.py "Hello, world!"
+```
+
+### TTS Preset (keeps stress)
+
+```bash
+phonebox recipe cmudict tts -o g2p.py
+```
+
+## Using Bundled G2P
+
+### Command Line
+
+```bash
+python g2p.py "Hello, world!"
+# hello   HH AH L OW
+# world   W ER L D
+
+# Raw mode (no text normalization)
+python g2p.py -r "Hello,"
+```
+
+### Python Library
+
+```python
+from g2p import G2PPredictor
+
+g2p = G2PPredictor.from_embedded()
+phones = g2p.pronounce("hello")  # ['HH', 'AH', 'L', 'OW']
+
+# Process text (tokenizes automatically)
+for word, phones in g2p.pronounce_text("Hello, world!"):
+    print(f"{word}: {' '.join(phones)}")
+```
+
+## CLI Commands
+
+```
+Quick Start:
+  recipe       Build complete G2P from dictionary (one command)
+
+Using Models:
+  pronounce    Get pronunciations for words
+  normalize    Preview text normalization/tokenization
+  bundle       Create standalone executable with embedded model
+
+Building Models:
+  model        Model operations (build, train, convert, benchmark)
+  dict         Dictionary operations (fetch, export-vectors)
+
+Low-Level:
+  align        Align letters to phonemes (EM algorithm)
+  vectorize    Convert alignments to feature vectors
+
+Quality:
+  check          Validate lexicon against phoneset
+  suggest-joins  Discover join candidates (multigram EM)
+  compare        1:1 vs n:m eval (locale or all IPA locales)
+  train-multigram  Train/export MultigramG2P
+```
+
+### G2P evaluation (IPA locales)
+
+```bash
+phonebox compare all                    # docs/G2P_COMPARE.md
+phonebox compare locale --lexicon … --locale it_IT
+phonebox train-multigram --locale it_IT --lexicon it_ipa.tsv -o model.g2p.gz
+```
+
+Repo wrappers: `compare_g2p_all.py`, `compare_g2p_sweep.py`, `dump_units.py`.
+See [`docs/G2P_EVAL.md`](docs/G2P_EVAL.md).
+
+### Examples
+
+```bash
+# Preview text normalization
+phonebox normalize "Hello, world!"
+
+# Pronounce with existing model
+phonebox pronounce hello world -m model.g2p.gz
+
+# Benchmark model
+phonebox model benchmark model.g2p.gz
+
+# Fetch dictionary manually
+phonebox dict fetch cmudict
+```
+
+## Python API
+
+```python
+from phonebox import G2P, MultigramG2P
+
+# 1:1 decision tree (CLI: phonebox pronounce)
+g2p = G2P(model="model.g2p.gz")
+phones = g2p.pronounce("hello")
+print(phones)  # ['HH', 'AH', 'L', 'OW']
+
+# n:m multigram (train via CLI or library)
+mg = MultigramG2P(max_letter_span=2, max_phone_span=2)
+mg.train_from_dict("lexicon.tsv")
+# phonebox train-multigram … ; phonebox pronounce -m model.g2p.gz (sidecar auto-detect)
+
+# N-best alternatives
+for pron, score in g2p.pronounce_nbest("read", n=3):
+    print(f"{pron} ({score:.3f})")
+```
+
+## Step-by-Step Training
+
+For debugging or custom workflows:
+
+```bash
+# 1. Fetch dictionary
+phonebox dict fetch cmudict
+
+# 2. Align letters to phonemes
+phonebox align data/cmudict/cmudict.dict -o alignments.txt --remove-stress
+
+# 3. Vectorize alignments
+phonebox vectorize alignments.txt -o vectors.txt
+
+# 4. Train from vectors
+phonebox model train en_US --vectors vectors.txt -o model.g2p.gz
+
+# 5. Bundle
+phonebox bundle model.g2p.gz -o g2p.py
+```
+
+## Standalone Deployment
+
+Bundled files have zero dependencies beyond the Python standard library:
+
+```bash
+phonebox bundle model.g2p.gz -o g2p.py
+python g2p.py "test"
+```
+
+## Algorithm
+
+1. **EM Alignment**: Expectation-Maximization aligns letters to phonemes
+2. **Feature Extraction**: 7-gram letter windows create feature vectors
+3. **Decision Tree**: ID3-style tree trained on aligned data
+4. **Prediction**: Tree traversal based on letter context
+
+Based on research from CMU:
+- [CMU G2P Research](http://www.cs.cmu.edu/afs/cs.cmu.edu/user/lenzo/html/areas/t2p/)
+- [ICSLP 1998 Paper](https://www.isca-speech.org/archive/icslp_1998/i98_0561.html)
+
+## License
+
+MIT License - see LICENSE file.
+
+## Author
+
+Kevin Lenzo ([@lenzo-ka](https://github.com/lenzo-ka))
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Add tests for new functionality
+4. Submit a pull request

phonebox-0.1.0/examples/library_usage.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python
+"""
+Examples of using phonebox as a Python library.
+"""
+
+from phonebox import G2P, Dictionary
+
+
+def example_simple_usage():
+    """Simplest way to use phonebox."""
+    print("=== Simple Usage ===")
+
+    # Load pre-trained model
+    g2p = G2P(model="models/en_US_nostress.g2p.gz")
+
+    # Pronounce a word
+    pronunciation = g2p.pronounce("hello")
+    print(f"hello: {pronunciation}")
+
+    # Or use as callable
+    print(f"world: {g2p('world')}")
+
+    # Batch pronunciation
+    words = ["python", "grapheme", "phoneme"]
+    for word, phones in g2p.pronounce_batch(words):
+        print(f"{word}: {' '.join(phones)}")
+
+
+def example_pocketsphinx_setup():
+    """One-command PocketSphinx setup."""
+    print("\n=== PocketSphinx Setup ===")
+
+    # This fetches CMUdict, processes it, and trains a model
+    g2p = G2P.from_pocketsphinx()
+
+    # Use it
+    print(f"test: {g2p('test')}")
+
+
+def example_dictionary_class():
+    """Using the Dictionary class."""
+    print("\n=== Dictionary Class ===")
+
+    # Fetch a dictionary
+    dict = Dictionary.fetch("cmudict")
+    print(f"Dictionary: {dict}")
+    print(f"Size: {len(dict)} entries")
+
+    # Process it
+    processed = dict.process(remove_stress=True, output="processed.dict")
+    print(f"Processed: {processed}")
+
+    # Train a model
+    _ = processed.train_g2p_model(locale="en_US", output="my_model.g2p.gz")
+    print("Model trained")
+
+
+def example_train_from_custom_dict():
+    """Train from your own dictionary."""
+    print("\n=== Train from Custom Dictionary ===")
+
+    # Create a small test dictionary
+    test_dict = """
+hello HH AH L OW
+world W ER L D
+python P AY TH AA N
+    """.strip()
+
+    with open("test_dict.txt", "w") as f:
+        f.write(test_dict)
+
+    # Train directly
+    g2p = G2P.train(dictionary="test_dict.txt", locale="en_US", remove_stress=False)
+
+    # Use it
+    print(f"hello: {g2p('hello')}")
+
+    # Clean up
+    import os
+
+    os.remove("test_dict.txt")
+
+
+def example_method_chaining():
+    """Fluent API with method chaining."""
+    print("\n=== Method Chaining ===")
+
+    # Fetch, process, and train in one line
+    _ = (
+        Dictionary.fetch("cmudict")
+        .process(remove_stress=True)
+        .train_g2p_model("en_US", output="model.g2p.gz")
+    )
+
+    print("Model trained via chaining")
+
+
+def example_advanced_usage():
+    """Advanced features."""
+    print("\n=== Advanced Usage ===")
+
+    # Direct DecisionTree access for full control
+    from phonebox import DecisionTree
+
+    dt = DecisionTree(
+        locale="en_US",
+        phoneset_name="cmu",
+        remove_stress=True,
+        max_iterations=50,
+        min_samples_split=5,
+    )
+
+    # Load and train
+    with open("data/cmudict/cmudict_nostress.dict") as f:
+        dt.load_prondict(f)
+    dt.align()
+    dt.train()
+
+    # Use it
+    print(f"pronunciation: {dt.pronounce('pronunciation')}")
+
+
+if __name__ == "__main__":
+    # Run only the simple example by default
+    # Uncomment others as needed
+
+    example_simple_usage()
+
+    # example_pocketsphinx_setup()  # Requires internet
+    # example_dictionary_class()     # Requires internet
+    # example_train_from_custom_dict()
+    # example_method_chaining()      # Requires internet
+    # example_advanced_usage()       # Requires data/