simphone 0.1.0__tar.gz

simphone-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 nguyenvulebinh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

simphone-0.1.0/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: simphone
+Version: 0.1.0
+Summary: Feature-weighted phonetic edit-distance search over a word or span list.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/nguyenvulebinh/simphone
+Project-URL: Repository, https://github.com/nguyenvulebinh/simphone
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Requires-Dist: panphon>=0.20
+Requires-Dist: rapidfuzz>=3
+Requires-Dist: numba>=0.57
+Requires-Dist: tqdm>=4.65
+Requires-Dist: sea-g2p>=0.9.1
+Provides-Extra: en
+Requires-Dist: text2phonemesequence>=0.1.4; extra == "en"
+Requires-Dist: transformers>=4.44; extra == "en"
+Requires-Dist: segments; extra == "en"
+Requires-Dist: huggingface_hub>=0.23; extra == "en"
+Requires-Dist: WeTextProcessing>=1.0; extra == "en"
+Dynamic: license-file
+
+# simphone
+
+Find words or spans that sound alike. Each span is turned into phones, then ranked with a feature-weighted edit distance (panphon articulatory cost, plus a penalty when tones differ).
+
+Vietnamese phones come from [sea-g2p](https://github.com/pnnbao97/sea-g2p). English phones come from CharsiuG2P.
+
+## Vietnamese reading
+
+Vietnamese build and search run sea-g2p in two steps, both with `punc_norm=False` (no forced trailing period):
+
+1. `Normalizer` expands the spoken form: numbers, dates, units, and abbreviations. `21` becomes `hai mươi mốt`, `km` becomes `ki lô mét`, `COVID-19` becomes `covid mười chín`.
+2. `G2P` turns that spoken form into phones. `21` and `hai mươi mốt` therefore share `hˈaːj mˈyəj mˈoɜt̪`.
+
+sea-g2p wraps an English run in `<en>...</en>` (`CPU` becomes `<en>c p u</en>`). `normalize` drops those markers and keeps the words. Search still passes the marked text to G2P, so the English run is read as English and the markers never become phones.
+
+## English reading
+
+English build and search run two steps:
+
+1. [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) English text normalization expands the spoken form (cardinals, ordinals, decimals, dates, times, measures, money). `21` becomes `twenty one`. A number plus a unit expands (`5 km` becomes `five kilometers`). A bare `km` stays `km`.
+2. CharsiuG2P turns each spoken word into phones. `21` and `twenty one` therefore share those phones.
+
+## Install
+
+`pip install simphone` installs Vietnamese only. Vietnamese search does not need a GPU.
+
+English is a separate extra. Install PyTorch yourself first, so a CUDA build already in the environment is kept. This package will not install torch.
+
+```bash
+# CPU. Skip this if torch is already installed.
+pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+pip install "simphone[en]"
+```
+
+The English extra needs Linux x86_64. It uses WeTextProcessing, which depends on pynini, and pynini publishes wheels for that platform only.
+
+From a checkout of this repo:
+
+```bash
+pip install -e .
+pip install -e ".[en]"   # after torch, Linux x86_64
+```
+
+Calling English without the extra raises an error that tells you to install `simphone[en]`. English out-of-dictionary words use Charsiu's byT5 model; `--device cuda:0` speeds that step only.
+
+## Python
+
+```python
+from simphone import PhoneticIndex
+
+index = PhoneticIndex.build(
+    ["mắc", "mác", "mức", "việt nam"],
+    lang="vi",
+    tone_weight=0.5,
+)
+index.save("vi.index")
+
+index = PhoneticIndex.load("vi.index")  # skips G2P and the cost matrix
+hits = index.search("mắc", topk=50, prefilter=300, sim_min=0.8)
+# [{"text": "...", "sim": ..., "n_units": 1, "dlen": 0}, ...]
+
+batch = index.search_many(
+    ["mắc", "việt nam"],
+    topk=50,
+    workers=0,       # 0 = all CPUs
+    block_size=512,
+)
+```
+
+`sim` is `1 - distance / max(len_a, len_b)`. `dlen` is the difference in whitespace-separated unit counts.
+
+Spoken form only, without phones or an index. One string per input, same order:
+
+```python
+from simphone import normalize
+
+normalize(["21", "km"], lang="vi")
+# ["hai mươi mốt", "ki lô mét"]
+
+normalize(["21", "5 km"], lang="en")
+# ["twenty one", "five kilometers"]
+```
+
+English normalization needs `simphone[en]` and does not load torch.
+
+## Command line
+
+Inventory and query files are one span per line. Blank lines and `#` comments are ignored.
+
+```bash
+simphone norm --lang vi --text "21" --text "km"
+simphone norm --lang en --text "21" --text "5 km"
+simphone norm --lang vi --queries spans.txt
+
+simphone build --lang vi --inventory spans.txt --save vi.index --tone-weight 0.5
+
+simphone search --load vi.index --query "mắc" --topk 50 --sim-min 0.8 --workers 0
+
+simphone search --load vi.index --queries queries.txt --workers 0 --block-size 512 --out hits.jsonl
+```
+
+`--inventory` on `search` builds in memory when you have no saved index. `--tone-weight` is fixed at build time and stored in the index.
+
+## Large lists
+
+Build the inventory once and `save` it. A later `load` does not phonemize those spans and does not rebuild the panphon matrix. A new query is phonemized on its own, including the Vietnamese or English normalizer above. If that query uses a phone the inventory never saw, only that phone's cost row is added.
+
+Searching many queries one call at a time repeats a full scan of the inventory and leaves cores idle. `search_many` / `--queries` is the batch path:
+
+- Several queries (at least as many as workers): the queries are split across processes. Each process keeps the inventory in shared memory and runs rapidfuzz on blocks of 512 queries, then reranks the top `--prefilter` with the feature-plus-tone distance.
+- One query, or fewer queries than workers: the inventory is split instead, so that query still uses more than one core.
+
+Passing the inventory back in as the query list returns a neighbor list for every span. A span is not returned as its own neighbor.
+
+## Tones
+
+panphon ignores tone. Vietnamese sea-g2p writes one syllable per space, with stress and a tone mark inside the syllable (`mà` → `mˌaː2`, `má` → `mˈaːɜ`). Those marks are peeled off and attached to the vowel before the distance is computed. The six tones on `ma mà mả mã má mạ` are: no mark, `2`, `4`, `5`, `ɜ` (sắc), `6`. English Charsiu tones stay Chao letters and use the same penalty. `--tone-weight` (default `0.5`) is added when two phones differ only in tone, or in tone as well as in the base phone.

simphone-0.1.0/README.md

@@ -0,0 +1,118 @@
+# simphone
+
+Find words or spans that sound alike. Each span is turned into phones, then ranked with a feature-weighted edit distance (panphon articulatory cost, plus a penalty when tones differ).
+
+Vietnamese phones come from [sea-g2p](https://github.com/pnnbao97/sea-g2p). English phones come from CharsiuG2P.
+
+## Vietnamese reading
+
+Vietnamese build and search run sea-g2p in two steps, both with `punc_norm=False` (no forced trailing period):
+
+1. `Normalizer` expands the spoken form: numbers, dates, units, and abbreviations. `21` becomes `hai mươi mốt`, `km` becomes `ki lô mét`, `COVID-19` becomes `covid mười chín`.
+2. `G2P` turns that spoken form into phones. `21` and `hai mươi mốt` therefore share `hˈaːj mˈyəj mˈoɜt̪`.
+
+sea-g2p wraps an English run in `<en>...</en>` (`CPU` becomes `<en>c p u</en>`). `normalize` drops those markers and keeps the words. Search still passes the marked text to G2P, so the English run is read as English and the markers never become phones.
+
+## English reading
+
+English build and search run two steps:
+
+1. [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing) English text normalization expands the spoken form (cardinals, ordinals, decimals, dates, times, measures, money). `21` becomes `twenty one`. A number plus a unit expands (`5 km` becomes `five kilometers`). A bare `km` stays `km`.
+2. CharsiuG2P turns each spoken word into phones. `21` and `twenty one` therefore share those phones.
+
+## Install
+
+`pip install simphone` installs Vietnamese only. Vietnamese search does not need a GPU.
+
+English is a separate extra. Install PyTorch yourself first, so a CUDA build already in the environment is kept. This package will not install torch.
+
+```bash
+# CPU. Skip this if torch is already installed.
+pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+pip install "simphone[en]"
+```
+
+The English extra needs Linux x86_64. It uses WeTextProcessing, which depends on pynini, and pynini publishes wheels for that platform only.
+
+From a checkout of this repo:
+
+```bash
+pip install -e .
+pip install -e ".[en]"   # after torch, Linux x86_64
+```
+
+Calling English without the extra raises an error that tells you to install `simphone[en]`. English out-of-dictionary words use Charsiu's byT5 model; `--device cuda:0` speeds that step only.
+
+## Python
+
+```python
+from simphone import PhoneticIndex
+
+index = PhoneticIndex.build(
+    ["mắc", "mác", "mức", "việt nam"],
+    lang="vi",
+    tone_weight=0.5,
+)
+index.save("vi.index")
+
+index = PhoneticIndex.load("vi.index")  # skips G2P and the cost matrix
+hits = index.search("mắc", topk=50, prefilter=300, sim_min=0.8)
+# [{"text": "...", "sim": ..., "n_units": 1, "dlen": 0}, ...]
+
+batch = index.search_many(
+    ["mắc", "việt nam"],
+    topk=50,
+    workers=0,       # 0 = all CPUs
+    block_size=512,
+)
+```
+
+`sim` is `1 - distance / max(len_a, len_b)`. `dlen` is the difference in whitespace-separated unit counts.
+
+Spoken form only, without phones or an index. One string per input, same order:
+
+```python
+from simphone import normalize
+
+normalize(["21", "km"], lang="vi")
+# ["hai mươi mốt", "ki lô mét"]
+
+normalize(["21", "5 km"], lang="en")
+# ["twenty one", "five kilometers"]
+```
+
+English normalization needs `simphone[en]` and does not load torch.
+
+## Command line
+
+Inventory and query files are one span per line. Blank lines and `#` comments are ignored.
+
+```bash
+simphone norm --lang vi --text "21" --text "km"
+simphone norm --lang en --text "21" --text "5 km"
+simphone norm --lang vi --queries spans.txt
+
+simphone build --lang vi --inventory spans.txt --save vi.index --tone-weight 0.5
+
+simphone search --load vi.index --query "mắc" --topk 50 --sim-min 0.8 --workers 0
+
+simphone search --load vi.index --queries queries.txt --workers 0 --block-size 512 --out hits.jsonl
+```
+
+`--inventory` on `search` builds in memory when you have no saved index. `--tone-weight` is fixed at build time and stored in the index.
+
+## Large lists
+
+Build the inventory once and `save` it. A later `load` does not phonemize those spans and does not rebuild the panphon matrix. A new query is phonemized on its own, including the Vietnamese or English normalizer above. If that query uses a phone the inventory never saw, only that phone's cost row is added.
+
+Searching many queries one call at a time repeats a full scan of the inventory and leaves cores idle. `search_many` / `--queries` is the batch path:
+
+- Several queries (at least as many as workers): the queries are split across processes. Each process keeps the inventory in shared memory and runs rapidfuzz on blocks of 512 queries, then reranks the top `--prefilter` with the feature-plus-tone distance.
+- One query, or fewer queries than workers: the inventory is split instead, so that query still uses more than one core.
+
+Passing the inventory back in as the query list returns a neighbor list for every span. A span is not returned as its own neighbor.
+
+## Tones
+
+panphon ignores tone. Vietnamese sea-g2p writes one syllable per space, with stress and a tone mark inside the syllable (`mà` → `mˌaː2`, `má` → `mˈaːɜ`). Those marks are peeled off and attached to the vowel before the distance is computed. The six tones on `ma mà mả mã má mạ` are: no mark, `2`, `4`, `5`, `ɜ` (sắc), `6`. English Charsiu tones stay Chao letters and use the same penalty. `--tone-weight` (default `0.5`) is added when two phones differ only in tone, or in tone as well as in the base phone.

simphone-0.1.0/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "simphone"
+version = "0.1.0"
+description = "Feature-weighted phonetic edit-distance search over a word or span list."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Text Processing :: Linguistic",
+]
+# torch is not a dependency. English CharsiuG2P needs the build already in the
+# environment. Vietnamese uses sea-g2p and does not.
+dependencies = [
+    "numpy>=1.22",
+    "panphon>=0.20",
+    "rapidfuzz>=3",
+    "numba>=0.57",
+    "tqdm>=4.65",
+    "sea-g2p>=0.9.1",
+]
+
+[project.optional-dependencies]
+en = [
+    "text2phonemesequence>=0.1.4",
+    "transformers>=4.44",
+    "segments",
+    "huggingface_hub>=0.23",
+    "WeTextProcessing>=1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/nguyenvulebinh/simphone"
+Repository = "https://github.com/nguyenvulebinh/simphone"
+
+[project.scripts]
+simphone = "simphone.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

simphone-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

simphone-0.1.0/src/simphone/__init__.py

@@ -0,0 +1,13 @@
+"""Phonetic similarity search over a list of words or spans."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from simphone.g2p import normalize
+from simphone.index import PhoneticIndex
+
+try:
+    __version__ = version("simphone")
+except PackageNotFoundError:
+    __version__ = "0.1.0"
+
+__all__ = ["PhoneticIndex", "normalize"]

simphone-0.1.0/src/simphone/__main__.py

@@ -0,0 +1,8 @@
+"""``python -m simphone``."""
+
+import sys
+
+from simphone.cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

simphone-0.1.0/src/simphone/cli.py

@@ -0,0 +1,174 @@
+"""Command line for building and searching a phonetic index."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from simphone.g2p import normalize
+from simphone.index import PhoneticIndex, read_span_file
+
+
+def _queries(args: argparse.Namespace) -> list[str]:
+    out = list(args.query or [])
+    if args.queries:
+        out.extend(read_span_file(args.queries))
+    return out
+
+
+def _print_hits(query: str, hits: list[dict]) -> None:
+    print(query)
+    if not hits:
+        print("  (none)")
+        return
+    for h in hits:
+        dlen = h["dlen"]
+        sign = f"{dlen:+d}"
+        print(f"  {h['text']}\t{h['sim']:.4f}\tn={h['n_units']}\tdlen={sign}")
+
+
+def cmd_norm(args: argparse.Namespace) -> int:
+    texts = list(args.text or [])
+    if args.queries:
+        texts.extend(read_span_file(args.queries))
+    if not texts:
+        print("pass --text and/or --queries", file=sys.stderr)
+        return 1
+    for line in normalize(texts, args.lang, workers=args.workers):
+        print(line)
+    return 0
+
+
+def cmd_build(args: argparse.Namespace) -> int:
+    spans = read_span_file(args.inventory)
+    if not spans:
+        print(f"no spans in {args.inventory}", file=sys.stderr)
+        return 1
+    print(f"phonemizing {len(spans)} spans ({args.lang})", flush=True)
+    index = PhoneticIndex.build(
+        spans,
+        args.lang,
+        tone_weight=args.tone_weight,
+        device=args.device,
+        cache_dir=args.cache_dir,
+        en_locale=args.en_locale,
+        workers=args.workers,
+    )
+    index.save(args.save)
+    print(f"saved {len(index.spans)} spans -> {args.save}")
+    return 0
+
+
+def cmd_search(args: argparse.Namespace) -> int:
+    queries = _queries(args)
+    if not queries:
+        print("pass --query and/or --queries", file=sys.stderr)
+        return 1
+    if args.load:
+        index = PhoneticIndex.load(args.load)
+    else:
+        if not args.inventory or not args.lang:
+            print("pass --load, or --inventory with --lang", file=sys.stderr)
+            return 1
+        spans = read_span_file(args.inventory)
+        print(f"phonemizing {len(spans)} spans ({args.lang})", flush=True)
+        index = PhoneticIndex.build(
+            spans,
+            args.lang,
+            tone_weight=args.tone_weight,
+            device=args.device,
+            cache_dir=args.cache_dir,
+            en_locale=args.en_locale,
+            workers=args.workers,
+        )
+    hits = index.search_many(
+        queries,
+        topk=args.topk,
+        prefilter=args.prefilter,
+        sim_min=args.sim_min,
+        workers=args.workers,
+        block_size=args.block_size,
+        device=args.device,
+        cache_dir=args.cache_dir,
+        en_locale=args.en_locale,
+    )
+    rows = [{"query": q, "neighbors": h} for q, h in zip(queries, hits)]
+    for row in rows:
+        _print_hits(row["query"], row["neighbors"])
+    sys.stdout.flush()
+    if args.out:
+        path = Path(args.out)
+        path.parent.mkdir(parents=True, exist_ok=True)
+        with path.open("w", encoding="utf-8") as fh:
+            for row in rows:
+                fh.write(json.dumps(row, ensure_ascii=False))
+                fh.write("\n")
+        print(f"wrote {len(rows)} rows -> {path}", file=sys.stderr)
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="simphone",
+        description="Find phonetically similar words or spans with a feature-weighted edit distance.",
+    )
+    sub = p.add_subparsers(dest="command", required=True)
+
+    n = sub.add_parser("norm", help="expand spans to their spoken form")
+    n.add_argument("--lang", required=True, choices=["vi", "en"])
+    n.add_argument("--text", action="append", default=None, help="repeatable span")
+    n.add_argument("--queries", default=None, help="file of spans, one per line")
+    n.add_argument(
+        "--workers",
+        type=int,
+        default=0,
+        help="sea-g2p Rayon threads for Vietnamese (0 = os.cpu_count()). Ignored for English.",
+    )
+    n.set_defaults(func=cmd_norm)
+
+    b = sub.add_parser("build", help="phonemize an inventory and save the index")
+    b.add_argument("--lang", required=True, choices=["vi", "en"])
+    b.add_argument("--inventory", required=True, help="one span per line")
+    b.add_argument("--save", required=True, help="directory to write")
+    b.add_argument("--tone-weight", type=float, default=0.5)
+    b.add_argument("--device", default="cpu", help="English out-of-dictionary G2P device")
+    b.add_argument(
+        "--workers",
+        type=int,
+        default=0,
+        help="CPU workers for Vietnamese segmentation (0 = os.cpu_count()). "
+             "Also sets sea-g2p's Rayon thread count.",
+    )
+    b.add_argument("--cache-dir", default=None, help="Charsiu model cache (English)")
+    b.add_argument("--en-locale", default="eng-us")
+    b.set_defaults(func=cmd_build)
+
+    s = sub.add_parser("search", help="search one or more queries against an inventory")
+    s.add_argument("--load", default=None, help="index directory from build")
+    s.add_argument("--lang", choices=["vi", "en"], help="required with --inventory")
+    s.add_argument("--inventory", default=None, help="build in memory instead of --load")
+    s.add_argument("--query", action="append", default=None, help="repeatable span")
+    s.add_argument("--queries", default=None, help="file of query spans, one per line")
+    s.add_argument("--topk", type=int, default=50)
+    s.add_argument("--prefilter", type=int, default=300)
+    s.add_argument("--sim-min", type=float, default=0.8)
+    s.add_argument("--workers", type=int, default=0, help="0 = os.cpu_count()")
+    s.add_argument("--block-size", type=int, default=512)
+    s.add_argument("--tone-weight", type=float, default=0.5)
+    s.add_argument("--device", default="cpu")
+    s.add_argument("--cache-dir", default=None)
+    s.add_argument("--en-locale", default="eng-us")
+    s.add_argument("--out", default=None, help="JSONL of query rows")
+    s.set_defaults(func=cmd_search)
+    return p
+
+
+def main(argv=None) -> int:
+    args = build_parser().parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())