scriptconv 0.0.4a2__tar.gz → 0.0.4a4__tar.gz

{scriptconv-0.0.4a2/scriptconv.egg-info → scriptconv-0.0.4a4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scriptconv
-Version: 0.0.4a2
+Version: 0.0.4a4
 Summary: Zero-dependency script & phoneme-notation core — ISO-15924 detection & metadata, IPA↔ARPABET/X-SAMPA/Lexique/Kirshenbaum/Cotovía/RFE, Buckwalter↔Arabic, Hangul→jamo, kana
 License: Apache-2.0
 Project-URL: Homepage, https://github.com/TigreGotico/scriptconv
@@ -78,6 +78,12 @@ Requires-Dist: viphoneme; extra == "vi"
 Provides-Extra: tashkeel
 Requires-Dist: scriptconv[phonemizers]; extra == "tashkeel"
 Requires-Dist: text2tashkeel; extra == "tashkeel"
+Provides-Extra: stress
+Requires-Dist: scriptconv[phonemizers]; extra == "stress"
+Requires-Dist: stressonnx>=0.0.2; extra == "stress"
+Provides-Extra: pt
+Requires-Dist: scriptconv[phonemizers]; extra == "pt"
+Requires-Dist: bifonia>=0.1.1; extra == "pt"
 Provides-Extra: ja-phonemizers
 Requires-Dist: scriptconv[phonemizers]; extra == "ja-phonemizers"
 Requires-Dist: pyopenjtalk; extra == "ja-phonemizers"
@@ -115,6 +121,7 @@ Requires-Dist: quebra-frases; extra == "test"
 Requires-Dist: langcodes; extra == "test"
 Requires-Dist: regex; extra == "test"
 Requires-Dist: ovos-number-parser; extra == "test"
+Requires-Dist: bifonia>=0.1.1; extra == "test"
 Dynamic: license-file
 
 <div align="center">
@@ -369,6 +376,38 @@ Two design points worth knowing:
   local `model=` path; resolving and caching model files is the caller's
   concern.
 
+**Pre-G2P disambiguation.** `add_diacritics(text, lang, model=None)` restores
+information ordinary orthography omits but a G2P needs, before phonemization:
+Hebrew niqqud (phonikud), Arabic tashkeel (text2tashkeel), word stress across
+26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene,
+Latvian, Armenian, Georgian, and Turkic/Caucasian languages (`scriptconv[stress]`),
+and European-Portuguese heterophonic-homograph sense diacritics (bifonia —
+`scriptconv[pt]`, never applied to `pt-BR`, whose vowel system differs):
+
+```python
+from scriptconv.phonemizers import GraphemePhonemizer
+
+p = GraphemePhonemizer()
+p.add_diacritics("Tenho muita sede hoje.", "pt")   # 'Tenho muita sêde hoje.' (thirst)
+```
+
+Diacritization also joins the graph, like phonemization, via
+`scriptconv.diacritics.register` — a `text -> text-diacritized` edge, opt-in,
+with `text -> ipa` routing unchanged:
+
+```python
+from scriptconv import diacritics
+g = DEFAULT_GRAPH.extend(diacritics.register).extend(phonemizers.register)
+g.convert("Tenho muita sede hoje.", "text", "text-diacritized", lang="pt")
+# 'Tenho muita sêde hoje.'
+```
+
+Stress is unwritten or under-marked in all 26 covered languages; East Slavic
+is the clearest case, where unstressed vowels also reduce (e.g. Russian
+о→[ɐ]/[ə]), so a missing mark corrupts more than prosody there. stressonnx is
+optional (not yet on PyPI) and emits the standard combining acute (U+0301)
+after the stressed vowel.
+
 Details: [docs/phonemizers.md](docs/phonemizers.md).
 
 ## Fidelity guarantees
@@ -412,6 +451,8 @@ The core installs with zero dependencies. Capabilities opt in:
 | `espeak`, `gruut`, `goruut`, `epitran`, `transphone`, `misaki`, `byt5` | Multilingual phonemizer backends |
 | `en-phonemizers`, `ja-phonemizers`, `zh-phonemizers`, `ko`, `ar-phonemizers`, `eu`, `pt-phonemizers`, `gl`, `he`, `fa`, `vi`, `mwl`, `shami`, `o2i` | Per-language phonemizer backends |
 | `tashkeel` | Arabic diacritization for the phonemizer pipeline (text2tashkeel) |
+| `stress` | Word-stress restoration for 26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene, Latvian, Armenian, Georgian, Turkic/Caucasian — for the phonemizer pipeline (stressonnx; not yet on PyPI) |
+| `pt` | European-Portuguese heterophonic-homograph sense diacritics for the phonemizer pipeline (bifonia) |
 
 ## Licensing
 
@@ -433,29 +474,17 @@ build on it:
   text-to-speech; consumes scriptconv for scripts, notation, conventions and
   the whole phonemizer layer.
 - [orthography2ipa](https://github.com/TigreGotico/orthography2ipa) —
-  data-driven orthography→IPA engine; the usual per-language phonemizer
-  default.
-- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic tokenization and
-  phonemization; the Arabic default.
+  data-driven orthography→IPA engine; the usual per-language phonemizer default.
+- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic phonemizer (dialect-aware).
+- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque phonemizer (dialect-aware).
+- [tugaphone](https://github.com/TigreGotico/tugaphone) — Portuguese phonemizer (dialect-aware).
+- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) — Mirandese phonemizer.
+- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) — Barranquenho phonemizer.
+- [pycotovia](https://github.com/TigreGotico/pycotovia) — Pure-Python phonemizer port
+  of the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
 - [espyak](https://github.com/TigreGotico/espyak) — pure-Python port of
   espeak-ng's G2P; the espeak fallback when the binary is absent.
-- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque
-  phonemizer (dialect-aware).
-- [tugaphone](https://github.com/TigreGotico/tugaphone) — European
-  Portuguese phonemizer.
-- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) —
-  Mirandese phonemizer.
-- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) —
-  Barranquenho phonemizer.
-- [pycotovia](https://github.com/TigreGotico/pycotovia) — Python bindings
-  for the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
-- [stressonnx](https://github.com/TigreGotico/stressonnx) — East Slavic
-  stress annotation; shares scriptconv's script tags.
-- [text2tashkeel](https://github.com/TigreGotico/text2tashkeel) — Arabic
-  diacritization models (the prediction side of the `tashkeel` convention).
-- [text2nikkud](https://github.com/TigreGotico/text2nikkud) — Hebrew
-  niqqud restoration (the prediction side of the `niqqud` convention).
-
+  
 ## Development
 
 ```bash

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/README.md

@@ -250,6 +250,38 @@ Two design points worth knowing:
   local `model=` path; resolving and caching model files is the caller's
   concern.
 
+**Pre-G2P disambiguation.** `add_diacritics(text, lang, model=None)` restores
+information ordinary orthography omits but a G2P needs, before phonemization:
+Hebrew niqqud (phonikud), Arabic tashkeel (text2tashkeel), word stress across
+26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene,
+Latvian, Armenian, Georgian, and Turkic/Caucasian languages (`scriptconv[stress]`),
+and European-Portuguese heterophonic-homograph sense diacritics (bifonia —
+`scriptconv[pt]`, never applied to `pt-BR`, whose vowel system differs):
+
+```python
+from scriptconv.phonemizers import GraphemePhonemizer
+
+p = GraphemePhonemizer()
+p.add_diacritics("Tenho muita sede hoje.", "pt")   # 'Tenho muita sêde hoje.' (thirst)
+```
+
+Diacritization also joins the graph, like phonemization, via
+`scriptconv.diacritics.register` — a `text -> text-diacritized` edge, opt-in,
+with `text -> ipa` routing unchanged:
+
+```python
+from scriptconv import diacritics
+g = DEFAULT_GRAPH.extend(diacritics.register).extend(phonemizers.register)
+g.convert("Tenho muita sede hoje.", "text", "text-diacritized", lang="pt")
+# 'Tenho muita sêde hoje.'
+```
+
+Stress is unwritten or under-marked in all 26 covered languages; East Slavic
+is the clearest case, where unstressed vowels also reduce (e.g. Russian
+о→[ɐ]/[ə]), so a missing mark corrupts more than prosody there. stressonnx is
+optional (not yet on PyPI) and emits the standard combining acute (U+0301)
+after the stressed vowel.
+
 Details: [docs/phonemizers.md](docs/phonemizers.md).
 
 ## Fidelity guarantees
@@ -293,6 +325,8 @@ The core installs with zero dependencies. Capabilities opt in:
 | `espeak`, `gruut`, `goruut`, `epitran`, `transphone`, `misaki`, `byt5` | Multilingual phonemizer backends |
 | `en-phonemizers`, `ja-phonemizers`, `zh-phonemizers`, `ko`, `ar-phonemizers`, `eu`, `pt-phonemizers`, `gl`, `he`, `fa`, `vi`, `mwl`, `shami`, `o2i` | Per-language phonemizer backends |
 | `tashkeel` | Arabic diacritization for the phonemizer pipeline (text2tashkeel) |
+| `stress` | Word-stress restoration for 26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene, Latvian, Armenian, Georgian, Turkic/Caucasian — for the phonemizer pipeline (stressonnx; not yet on PyPI) |
+| `pt` | European-Portuguese heterophonic-homograph sense diacritics for the phonemizer pipeline (bifonia) |
 
 ## Licensing
 
@@ -314,29 +348,17 @@ build on it:
   text-to-speech; consumes scriptconv for scripts, notation, conventions and
   the whole phonemizer layer.
 - [orthography2ipa](https://github.com/TigreGotico/orthography2ipa) —
-  data-driven orthography→IPA engine; the usual per-language phonemizer
-  default.
-- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic tokenization and
-  phonemization; the Arabic default.
+  data-driven orthography→IPA engine; the usual per-language phonemizer default.
+- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic phonemizer (dialect-aware).
+- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque phonemizer (dialect-aware).
+- [tugaphone](https://github.com/TigreGotico/tugaphone) — Portuguese phonemizer (dialect-aware).
+- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) — Mirandese phonemizer.
+- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) — Barranquenho phonemizer.
+- [pycotovia](https://github.com/TigreGotico/pycotovia) — Pure-Python phonemizer port
+  of the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
 - [espyak](https://github.com/TigreGotico/espyak) — pure-Python port of
   espeak-ng's G2P; the espeak fallback when the binary is absent.
-- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque
-  phonemizer (dialect-aware).
-- [tugaphone](https://github.com/TigreGotico/tugaphone) — European
-  Portuguese phonemizer.
-- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) —
-  Mirandese phonemizer.
-- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) —
-  Barranquenho phonemizer.
-- [pycotovia](https://github.com/TigreGotico/pycotovia) — Python bindings
-  for the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
-- [stressonnx](https://github.com/TigreGotico/stressonnx) — East Slavic
-  stress annotation; shares scriptconv's script tags.
-- [text2tashkeel](https://github.com/TigreGotico/text2tashkeel) — Arabic
-  diacritization models (the prediction side of the `tashkeel` convention).
-- [text2nikkud](https://github.com/TigreGotico/text2nikkud) — Hebrew
-  niqqud restoration (the prediction side of the `niqqud` convention).
-
+  
 ## Development
 
 ```bash

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/pyproject.toml

@@ -53,13 +53,15 @@ he = ["scriptconv[phonemizers]", "phonikud-onnx"]
 fa = ["scriptconv[phonemizers]", "persian_phonemizer"]
 vi = ["scriptconv[phonemizers]", "viphoneme"]
 tashkeel = ["scriptconv[phonemizers]", "text2tashkeel"]
+stress = ["scriptconv[phonemizers]", "stressonnx>=0.0.2"]
+pt = ["scriptconv[phonemizers]", "bifonia>=0.1.1"]
 ja-phonemizers = ["scriptconv[phonemizers]", "pyopenjtalk", "cutlet", "pykakasi>=2.3,<3"]
 ko = ["scriptconv[phonemizers]", "g2pk", "regex"]
 zh-phonemizers = ["scriptconv[phonemizers]", "jieba", "pypinyin>=0.50,<1", "xpinyin", "g2pM", "pinyin_to_ipa"]
 en-phonemizers = ["scriptconv[phonemizers]", "g2p_en", "openphonemizer", "deep-phonemizer"]
 ar-phonemizers = ["scriptconv[phonemizers]", "arbtok", "ovos-number-parser"]
 shami = ["scriptconv[phonemizers]", "regex"]
-test = ["pytest", "pytest-timeout", "pykakasi>=2.3,<3", "pypinyin>=0.50,<1", "quebra-frases", "langcodes", "regex", "ovos-number-parser"]
+test = ["pytest", "pytest-timeout", "pykakasi>=2.3,<3", "pypinyin>=0.50,<1", "quebra-frases", "langcodes", "regex", "ovos-number-parser", "bifonia>=0.1.1"]
 
 [project.urls]
 Homepage = "https://github.com/TigreGotico/scriptconv"

scriptconv-0.0.4a4/scriptconv/diacritics.py

@@ -0,0 +1,153 @@
+"""Diacritization as a graph extension — parallel to :mod:`scriptconv.phonemizers`.
+
+Diacritization (Arabic tashkeel, Hebrew niqqud, East-Slavic/Turkic/Caucasian
+word stress, European-Portuguese homograph sense marks) is, architecturally,
+just another transform between text representations: it maps the ``"text"``
+node to a lang-contextual ``"text-diacritized"`` node.  Like
+:func:`scriptconv.phonemizers.register` it is opt-in — ``DEFAULT_GRAPH`` stays
+free of it until a caller extends a graph::
+
+    from scriptconv.graph import DEFAULT_GRAPH
+    from scriptconv import diacritics, phonemizers
+    g = DEFAULT_GRAPH.extend(diacritics.register).extend(phonemizers.register)
+
+    g.convert("замок стоит на горе", "text", "text-diacritized", lang="ru")
+    # 'за́мок сто́ит на горе́'    — just the diacritized text
+
+    g.convert("Tenho muita sede hoje.", "text", "text-diacritized", lang="pt")
+    # 'Tenho muita sêde hoje.'
+
+Routing ``"text" -> "ipa"`` still takes the direct phonemization edge by
+default: the diacritization edge is model-based (``lossless=False``), so it
+never out-prices a direct phonemization, and enabling this extension does not
+silently change phonemization output.  The diacritized route is taken only
+when a caller asks for ``"text-diacritized"`` explicitly.
+
+A future per-engine *stance* would push this further into topology rather than
+a flag: an engine that *requires* vocalized input would carry only a
+``"text-diacritized" -> "ipa"`` edge (forcing the detour), one that *forbids*
+diacritics would carry only ``"text" -> "ipa"`` (making the detour
+unroutable), and a tolerant engine (e.g. arbtok, which self-vocalizes) carries
+both.  The strip direction (``"text-diacritized" -> "text"``) is registered too, but
+gated to languages whose marks are removable *overlay* diacritics —
+Arabic/Hebrew vocalization and East-Slavic/Turkic/Caucasian stress. Strip
+removes only the specific overlay codepoints each backend adds (combining
+acute/grave for stress, tashkeel for Arabic, niqqud for Hebrew) — never a
+blanket combining-mark filter — so precomposed native letters survive
+(Cyrillic й/ё, Latvian macrons, Azerbaijani ç/ö, Arabic hamza carriers أ إ آ ؤ
+ئ). The gate uses exact primary-subtag matching, so Aragonese (``arg``),
+Herero (``her``), Mapudungun (``arn``) etc. are never misread as Arabic/Hebrew.
+Languages whose diacritics are native orthography (European Portuguese, via
+bifonia) refuse the strip with :class:`ValueError` instead of silently
+corrupting the spelling (``café`` must never become ``cafe``). This split
+tracks which backend/model produced the marks: overlay backends (phonikud,
+tashkeel, stressonnx) are strippable; spelling-integral backends (bifonia) are
+not.
+"""
+from scriptconv.graph import Edge
+from scriptconv.phonemizers.base import STRESS_LANGS, _primary_subtag
+
+#: The lang-contextual node produced by diacritization.  Like ``"text"`` it is
+#: meaningful only with ``lang=`` context and exists only in opted-in graphs.
+DIACRITIZED = "text-diacritized"
+
+_DEFAULT_PHONEMIZER = None
+
+
+def _get_phonemizer(phonikud_model=None):
+    global _DEFAULT_PHONEMIZER
+    if phonikud_model:
+        from scriptconv.phonemizers.base import GraphemePhonemizer
+        return GraphemePhonemizer(phonikud_model=phonikud_model)
+    if _DEFAULT_PHONEMIZER is None:
+        from scriptconv.phonemizers.base import GraphemePhonemizer
+        _DEFAULT_PHONEMIZER = GraphemePhonemizer()
+    return _DEFAULT_PHONEMIZER
+
+
+def diacritize(text: str, lang: str = "und", model=None, **context) -> str:
+    """Add pronunciation-disambiguating diacritics to *text* for *lang*.
+
+    Thin wrapper over :meth:`BasePhonemizer.add_diacritics` — the single
+    dispatch (he→phonikud, ar→tashkeel, ru/uk/be/…→stressonnx, pt→bifonia).
+    Reuses a cached default phonemizer so repeated calls don't reload models; a
+    per-call Hebrew ``phonikud_model`` path (via context) builds a fresh one.
+    """
+    return _get_phonemizer(context.get("phonikud_model")).add_diacritics(text, lang, model)
+
+
+# Combining marks each diacritization backend overlays onto the bare text.
+# Strip removes ONLY these codepoints — never a blanket combining-mark filter,
+# which (via NFD) would decompose and destroy precomposed native letters:
+# Cyrillic й/ё, Latvian ī, Azerbaijani ç/ö, and Arabic hamza carriers أ إ آ ؤ ئ.
+_STRESS_MARKS = frozenset({0x0300, 0x0301})            # combining grave / acute
+_ARABIC_MARKS = frozenset(range(0x064B, 0x0660)) | frozenset({0x0670})  # tashkeel + dagger alef
+_HEBREW_MARKS = (frozenset(range(0x05B0, 0x05BE))
+                 | frozenset({0x05BF, 0x05C1, 0x05C2, 0x05C4, 0x05C5, 0x05C7}))  # niqqud
+
+
+def _overlay_marks(lang: str) -> "frozenset | None":
+    """The overlay codepoints for *lang*'s diacritization backend, or None.
+
+    Uses exact primary-subtag matching (never ``startswith``) so Aragonese
+    (``arg``), Herero (``her``), Mapudungun (``arn``) etc. are NOT misread as
+    Arabic/Hebrew and stripped.
+    """
+    p = _primary_subtag(lang)
+    if p in STRESS_LANGS:
+        return _STRESS_MARKS
+    if p == "ar":
+        return _ARABIC_MARKS
+    if p == "he":
+        return _HEBREW_MARKS
+    return None
+
+
+def _supports_strip(lang: str) -> bool:
+    """True for languages whose diacritics are removable overlays (stress marks,
+    Arabic tashkeel, Hebrew niqqud) rather than native orthography."""
+    return _overlay_marks(lang) is not None
+
+
+def strip_diacritics(text: str, lang: str = "und", **_) -> str:
+    """Remove the overlay diacritics *lang*'s backend adds, recovering the bare
+    text WITHOUT touching native letters.
+
+    Removes only the specific overlay codepoints (combining acute/grave for
+    stress, tashkeel for Arabic, niqqud for Hebrew) — precomposed native
+    letters (Cyrillic й/ё, Latvian ī, Azerbaijani ç, Arabic hamza carriers) are
+    left intact. Raises :class:`ValueError` for languages whose diacritics are
+    part of the native orthography (e.g. European Portuguese), where any removal
+    would corrupt the spelling.
+    """
+    marks = _overlay_marks(lang)
+    if marks is None:
+        raise ValueError(
+            f"cannot strip diacritics for lang={lang!r}: its diacritics are "
+            "part of the native orthography, so removal would corrupt the "
+            "spelling. strip is defined only for overlay diacritics "
+            "(Arabic/Hebrew vocalization, East-Slavic/Turkic stress).")
+    return "".join(c for c in text if ord(c) not in marks)
+
+
+def register(graph) -> None:
+    """Opt-in graph integration: add the diacritize/strip edge pair.
+
+    ``text -> text-diacritized`` is model-based (``lossless=False``), so a
+    direct ``text -> ipa`` phonemization always out-prices the detour —
+    enabling this extension is safe and non-invasive.  Pair with
+    :func:`scriptconv.phonemizers.register` to make
+    ``"text-diacritized" -> "ipa"`` reachable.
+
+    ``text-diacritized -> text`` (:func:`strip_diacritics`) is lossless and
+    cheap, but gated: it raises for languages whose diacritics are native
+    orthography rather than removable overlays.
+    """
+    graph.register(
+        Edge("text", DIACRITIZED,
+             lambda text, lang="und", model=None, **c: diacritize(text, lang, model, **c),
+             lossless=False))
+    graph.register(
+        Edge(DIACRITIZED, "text",
+             lambda text, lang="und", **_: strip_diacritics(text, lang),
+             lossless=True))

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/scriptconv/phonemizers/base.py

@@ -34,6 +34,45 @@ RawPhonemizedChunks = List[Tuple[str, str, bool]]
 
 PhonemizedChunks = list[list[str]]
 
+# Across East Slavic, Bulgarian/Macedonian/Slovene, Latvian, Armenian,
+# Georgian, and several Turkic/Caucasian languages, lexical word stress is
+# free (not fixed to a syllable) and ordinary orthography leaves it unwritten
+# or under-marked. The clearest case is East Slavic: stress is also mobile
+# (it shifts between forms of the same word) and unstressed vowels *reduce*
+# — Russian unstressed "о" surfaces as [ɐ] or [ə] depending on distance from
+# the stress, not [o] — so a wrong or missing mark there corrupts the vowel
+# quality of the whole word, not just its prosody. Other families in this set
+# don't necessarily reduce vowels, but still need the mark for correct stress
+# placement and prosody. stressonnx restores it as a combining acute (U+0301)
+# after the stressed vowel, covering 26 BCP-47 tags across these families
+# (24 primary subtags; Azerbaijani and Uzbek each have Cyrillic/Latin script
+# variants routed by the full tag).
+STRESS_LANGS = {
+    "az", "ba", "be", "bg", "cv", "hy", "ka", "kbd", "kjh", "kk", "ky", "lv",
+    "mdf", "mk", "myv", "ru", "sah", "sl", "tg", "tt", "udm", "uk", "uz", "xal",
+}
+
+
+def _primary_subtag(lang: str) -> str:
+    """Lowercase, ``_``→``-`` normalized primary language subtag.
+
+    Used for exact-match routing (``STRESS_LANGS`` membership) rather than
+    ``str.startswith``, so e.g. Berber (``ber``) never false-matches
+    Belarusian (``be``).
+    """
+    return lang.lower().replace("_", "-").split("-")[0]
+
+
+def _is_european_portuguese(lang: str) -> bool:
+    """True for European Portuguese (``pt`` or a ``pt-PT`` region tag).
+
+    False for Brazilian Portuguese (``pt-BR``) and everything else — the two
+    varieties' vowel systems differ, and bifonia's open/closed diacritics are
+    only valid for European Portuguese phonology.
+    """
+    norm = lang.lower().replace("_", "-")
+    return norm == "pt" or norm == "pt-pt"
+
 
 class BasePhonemizer(metaclass=abc.ABCMeta):
     def __init__(self, alphabet: Alphabet = Alphabet.UNICODE,
@@ -94,6 +133,41 @@ class BasePhonemizer(metaclass=abc.ABCMeta):
             self._tashkeel[model] = Diacritizer(model)
         return self._tashkeel[model]
 
+    def _stress(self, text: str, lang: str, model: Optional[str] = None) -> str:
+        """Word-stress restoration via stressonnx, for the 26 language tags
+        it covers (see ``STRESS_LANGS``) — East Slavic, Bulgarian/Macedonian/
+        Slovene, Latvian, Armenian, Georgian, and Turkic/Caucasian languages.
+
+        stressonnx is not on PyPI yet; install straight from source. Install
+        with ``pip install scriptconv[stress]`` (or ``pip install
+        stressonnx``)."""
+        try:
+            from stressonnx import stress
+        except ImportError as e:
+            raise ImportError(
+                "stress restoration requires the stressonnx package: "
+                "pip install scriptconv[stress]  (or pip install stressonnx)"
+            ) from e
+        return stress(text, lang, model=model)
+
+    def _sense_diacritics_pt(self, text: str) -> str:
+        """European-Portuguese heterophonic-homograph sense diacritics via bifonia.
+
+        Rewrites homographs whose pronunciation depends on meaning (e.g.
+        "sede" thirst/closed vs. seat/open) with an explicit open/closed
+        vowel diacritic. These are ordinary Portuguese orthographic marks,
+        chosen so any downstream G2P — rule-based, neural, or espeak —
+        reads them correctly. Install with ``pip install scriptconv[pt]``
+        (or ``pip install bifonia``)."""
+        try:
+            from bifonia import add_extra_diacritics
+        except ImportError as e:
+            raise ImportError(
+                "European-Portuguese sense diacritics require the bifonia package: "
+                "pip install scriptconv[pt]  (or pip install bifonia)"
+            ) from e
+        return add_extra_diacritics(text)
+
     @abc.abstractmethod
     def phonemize_string(self, text: str, lang: str) -> str:
         raise NotImplementedError
@@ -103,10 +177,36 @@ class BasePhonemizer(metaclass=abc.ABCMeta):
 
     def add_diacritics(self, text: str, lang: str,
                        model: Optional[str] = None) -> str:
-        if lang.startswith("he"):
+        """Disambiguate pronunciation before G2P by adding diacritics.
+
+        Four backends, each restoring information ordinary orthography
+        omits but downstream G2P needs:
+
+        - Hebrew (``he``) — niqqud via phonikud (``phonikud_model=``).
+        - Arabic (``ar``) — tashkeel via text2tashkeel (``[tashkeel]``).
+        - East Slavic, Bulgarian/Macedonian/Slovene, Latvian, Armenian,
+          Georgian, and Turkic/Caucasian languages (``STRESS_LANGS``, 26
+          stressonnx tags) — word stress via stressonnx (``[stress]``);
+          stress is unwritten or under-marked in these languages, and in
+          East Slavic unstressed vowels also reduce, so a missing mark can
+          corrupt more than prosody.
+        - European Portuguese (``pt``/``pt-PT``, never ``pt-BR``) —
+          heterophonic-homograph sense diacritics via bifonia (``[pt]``);
+          ordinary Portuguese orthographic marks that any downstream G2P
+          reads correctly.
+
+        Unrecognized languages are returned unchanged. Each backend raises
+        ``ImportError`` naming its extra when the optional dependency is
+        missing — scriptconv never installs anything on the caller's behalf.
+        """
+        if _primary_subtag(lang) == "he":
             return self.phonikud.add_diacritics(text)
-        elif lang.startswith("ar"):
+        elif _primary_subtag(lang) == "ar":
             return self.tashkeel(model).diacritize(text)
+        elif _primary_subtag(lang) in STRESS_LANGS:
+            return self._stress(text, lang, model)
+        elif _is_european_portuguese(lang):
+            return self._sense_diacritics_pt(text)
         return text
 
     def phonemize(self, text: str, lang: str) -> PhonemizedChunks:

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/scriptconv/phonemizers/registry.py

@@ -219,7 +219,9 @@ def register(graph) -> None:
     routing context, and present only in graphs that opted in (the
     :data:`scriptconv.graph.DEFAULT_GRAPH` stays orthography-only by
     design) — and one dispatching ``text -> ipa`` edge that resolves the
-    per-language default (honouring an ``override=`` context key).
+    per-language default (honouring an ``override=`` context key). It also
+    accepts already-diacritized input via a ``"text-diacritized" -> "ipa"``
+    edge (the same phonemization; pair with :func:`scriptconv.diacritics.register`).
 
     Usage::
 
@@ -236,3 +238,4 @@ def register(graph) -> None:
                                    override).phonemize_string(text, lang)
 
     graph.register(Edge("text", "ipa", _text_to_ipa, lossless=False))
+    graph.register(Edge("text-diacritized", "ipa", _text_to_ipa, lossless=False))

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/scriptconv/version.py

@@ -2,7 +2,7 @@
 VERSION_MAJOR = 0
 VERSION_MINOR = 0
 VERSION_BUILD = 4
-VERSION_ALPHA = 2
+VERSION_ALPHA = 4
 # END_VERSION_BLOCK
 
 VERSION_STR = f"{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_BUILD}"

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4/scriptconv.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scriptconv
-Version: 0.0.4a2
+Version: 0.0.4a4
 Summary: Zero-dependency script & phoneme-notation core — ISO-15924 detection & metadata, IPA↔ARPABET/X-SAMPA/Lexique/Kirshenbaum/Cotovía/RFE, Buckwalter↔Arabic, Hangul→jamo, kana
 License: Apache-2.0
 Project-URL: Homepage, https://github.com/TigreGotico/scriptconv
@@ -78,6 +78,12 @@ Requires-Dist: viphoneme; extra == "vi"
 Provides-Extra: tashkeel
 Requires-Dist: scriptconv[phonemizers]; extra == "tashkeel"
 Requires-Dist: text2tashkeel; extra == "tashkeel"
+Provides-Extra: stress
+Requires-Dist: scriptconv[phonemizers]; extra == "stress"
+Requires-Dist: stressonnx>=0.0.2; extra == "stress"
+Provides-Extra: pt
+Requires-Dist: scriptconv[phonemizers]; extra == "pt"
+Requires-Dist: bifonia>=0.1.1; extra == "pt"
 Provides-Extra: ja-phonemizers
 Requires-Dist: scriptconv[phonemizers]; extra == "ja-phonemizers"
 Requires-Dist: pyopenjtalk; extra == "ja-phonemizers"
@@ -115,6 +121,7 @@ Requires-Dist: quebra-frases; extra == "test"
 Requires-Dist: langcodes; extra == "test"
 Requires-Dist: regex; extra == "test"
 Requires-Dist: ovos-number-parser; extra == "test"
+Requires-Dist: bifonia>=0.1.1; extra == "test"
 Dynamic: license-file
 
 <div align="center">
@@ -369,6 +376,38 @@ Two design points worth knowing:
   local `model=` path; resolving and caching model files is the caller's
   concern.
 
+**Pre-G2P disambiguation.** `add_diacritics(text, lang, model=None)` restores
+information ordinary orthography omits but a G2P needs, before phonemization:
+Hebrew niqqud (phonikud), Arabic tashkeel (text2tashkeel), word stress across
+26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene,
+Latvian, Armenian, Georgian, and Turkic/Caucasian languages (`scriptconv[stress]`),
+and European-Portuguese heterophonic-homograph sense diacritics (bifonia —
+`scriptconv[pt]`, never applied to `pt-BR`, whose vowel system differs):
+
+```python
+from scriptconv.phonemizers import GraphemePhonemizer
+
+p = GraphemePhonemizer()
+p.add_diacritics("Tenho muita sede hoje.", "pt")   # 'Tenho muita sêde hoje.' (thirst)
+```
+
+Diacritization also joins the graph, like phonemization, via
+`scriptconv.diacritics.register` — a `text -> text-diacritized` edge, opt-in,
+with `text -> ipa` routing unchanged:
+
+```python
+from scriptconv import diacritics
+g = DEFAULT_GRAPH.extend(diacritics.register).extend(phonemizers.register)
+g.convert("Tenho muita sede hoje.", "text", "text-diacritized", lang="pt")
+# 'Tenho muita sêde hoje.'
+```
+
+Stress is unwritten or under-marked in all 26 covered languages; East Slavic
+is the clearest case, where unstressed vowels also reduce (e.g. Russian
+о→[ɐ]/[ə]), so a missing mark corrupts more than prosody there. stressonnx is
+optional (not yet on PyPI) and emits the standard combining acute (U+0301)
+after the stressed vowel.
+
 Details: [docs/phonemizers.md](docs/phonemizers.md).
 
 ## Fidelity guarantees
@@ -412,6 +451,8 @@ The core installs with zero dependencies. Capabilities opt in:
 | `espeak`, `gruut`, `goruut`, `epitran`, `transphone`, `misaki`, `byt5` | Multilingual phonemizer backends |
 | `en-phonemizers`, `ja-phonemizers`, `zh-phonemizers`, `ko`, `ar-phonemizers`, `eu`, `pt-phonemizers`, `gl`, `he`, `fa`, `vi`, `mwl`, `shami`, `o2i` | Per-language phonemizer backends |
 | `tashkeel` | Arabic diacritization for the phonemizer pipeline (text2tashkeel) |
+| `stress` | Word-stress restoration for 26 stressonnx language tags — East Slavic, Bulgarian/Macedonian/Slovene, Latvian, Armenian, Georgian, Turkic/Caucasian — for the phonemizer pipeline (stressonnx; not yet on PyPI) |
+| `pt` | European-Portuguese heterophonic-homograph sense diacritics for the phonemizer pipeline (bifonia) |
 
 ## Licensing
 
@@ -433,29 +474,17 @@ build on it:
   text-to-speech; consumes scriptconv for scripts, notation, conventions and
   the whole phonemizer layer.
 - [orthography2ipa](https://github.com/TigreGotico/orthography2ipa) —
-  data-driven orthography→IPA engine; the usual per-language phonemizer
-  default.
-- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic tokenization and
-  phonemization; the Arabic default.
+  data-driven orthography→IPA engine; the usual per-language phonemizer default.
+- [arbtok](https://github.com/TigreGotico/arbtok) — Arabic phonemizer (dialect-aware).
+- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque phonemizer (dialect-aware).
+- [tugaphone](https://github.com/TigreGotico/tugaphone) — Portuguese phonemizer (dialect-aware).
+- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) — Mirandese phonemizer.
+- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) — Barranquenho phonemizer.
+- [pycotovia](https://github.com/TigreGotico/pycotovia) — Pure-Python phonemizer port
+  of the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
 - [espyak](https://github.com/TigreGotico/espyak) — pure-Python port of
   espeak-ng's G2P; the espeak fallback when the binary is absent.
-- [euskaphone](https://github.com/TigreGotico/euskaphone) — Basque
-  phonemizer (dialect-aware).
-- [tugaphone](https://github.com/TigreGotico/tugaphone) — European
-  Portuguese phonemizer.
-- [mwl_phonemizer](https://github.com/TigreGotico/mwl_phonemizer) —
-  Mirandese phonemizer.
-- [g2p_barranquenho](https://github.com/TigreGotico/g2p_barranquenho) —
-  Barranquenho phonemizer.
-- [pycotovia](https://github.com/TigreGotico/pycotovia) — Python bindings
-  for the Cotovía Galician TTS engine, whose notation scriptconv transcodes.
-- [stressonnx](https://github.com/TigreGotico/stressonnx) — East Slavic
-  stress annotation; shares scriptconv's script tags.
-- [text2tashkeel](https://github.com/TigreGotico/text2tashkeel) — Arabic
-  diacritization models (the prediction side of the `tashkeel` convention).
-- [text2nikkud](https://github.com/TigreGotico/text2nikkud) — Hebrew
-  niqqud restoration (the prediction side of the `niqqud` convention).
-
+  
 ## Development
 
 ```bash

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/scriptconv.egg-info/SOURCES.txt

@@ -6,6 +6,7 @@ scriptconv/__init__.py
 scriptconv/__main__.py
 scriptconv/cangjie.py
 scriptconv/conventions.py
+scriptconv/diacritics.py
 scriptconv/graph.py
 scriptconv/notation.py
 scriptconv/py.typed
@@ -77,6 +78,7 @@ tests/test_arpa_stress.py
 tests/test_cangjie.py
 tests/test_cli.py
 tests/test_conventions.py
+tests/test_diacritics_graph.py
 tests/test_errors_policy.py
 tests/test_examples.py
 tests/test_graph.py

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/scriptconv.egg-info/requires.txt

@@ -78,6 +78,10 @@ orthography2ipa
 quebra-frases
 langcodes
 
+[pt]
+scriptconv[phonemizers]
+bifonia>=0.1.1
+
 [pt-phonemizers]
 scriptconv[phonemizers]
 tugaphone
@@ -87,6 +91,10 @@ g2p_barranquenho
 scriptconv[phonemizers]
 regex
 
+[stress]
+scriptconv[phonemizers]
+stressonnx>=0.0.2
+
 [tashkeel]
 scriptconv[phonemizers]
 text2tashkeel
@@ -100,6 +108,7 @@ quebra-frases
 langcodes
 regex
 ovos-number-parser
+bifonia>=0.1.1
 
 [transphone]
 scriptconv[phonemizers]

scriptconv-0.0.4a4/tests/test_diacritics_graph.py

@@ -0,0 +1,104 @@
+import sys
+import types
+import unittest
+from unittest import mock
+
+from scriptconv.graph import DEFAULT_GRAPH
+from scriptconv import diacritics, phonemizers
+
+
+class TestDiacriticsGraphExtension(unittest.TestCase):
+    def setUp(self):
+        self.graph = DEFAULT_GRAPH.extend(diacritics.register).extend(phonemizers.register)
+
+    def test_default_graph_has_no_diacritized_node(self):
+        self.assertFalse(DEFAULT_GRAPH.can_convert("text", "text-diacritized"))
+
+    def test_extend_adds_diacritized_node(self):
+        self.assertTrue(self.graph.can_convert("text", "text-diacritized"))
+        self.assertTrue(self.graph.can_convert("text-diacritized", "ipa"))
+
+    def test_text_to_ipa_prefers_direct_route(self):
+        route = self.graph.route("text", "ipa")
+        self.assertEqual(len(route), 1)
+        edge = route[0]
+        self.assertEqual(edge.src, "text")
+        self.assertEqual(edge.dst, "ipa")
+
+    def test_portuguese_diacritization_via_graph(self):
+        out = self.graph.convert("Tenho muita sede hoje.", "text",
+                                  "text-diacritized", lang="pt")
+        self.assertEqual(out, "Tenho muita sêde hoje.")
+        out2 = self.graph.convert("A sede da empresa fica em Lisboa.", "text",
+                                   "text-diacritized", lang="pt")
+        self.assertEqual(out2, "A séde da empresa fica em Lisboa.")
+
+    def test_slavic_diacritization_routes_to_stressonnx_stub(self):
+        calls = []
+        mod = types.ModuleType("stressonnx")
+
+        def stress(text, lang, model=None):
+            calls.append((text, lang, model))
+            return "STRESSED"
+
+        mod.stress = stress
+        with mock.patch.dict(sys.modules, {"stressonnx": mod}):
+            out = self.graph.convert("замок стоит", "text",
+                                      "text-diacritized", lang="ru")
+        self.assertEqual(out, "STRESSED")
+        self.assertEqual(calls, [("замок стоит", "ru", None)])
+
+    def test_strip_recovers_bare_russian(self):
+        out = self.graph.convert("за́мок", "text-diacritized", "text", lang="ru")
+        self.assertEqual(out, "замок")
+
+    def test_strip_recovers_bare_arabic(self):
+        vocalized = "مُحَمَّد"
+        bare = "".join(c for c in vocalized if not (0x064B <= ord(c) <= 0x065F
+                                                     or ord(c) == 0x0670))
+        out = self.graph.convert(vocalized, "text-diacritized", "text", lang="ar")
+        self.assertEqual(out, bare)
+        self.assertEqual(out, "محمد")
+
+    def test_strip_refused_for_portuguese(self):
+        with self.assertRaises(ValueError) as ctx:
+            self.graph.convert("sêde", "text-diacritized", "text", lang="pt")
+        self.assertIn("native orthography", str(ctx.exception))
+
+    def test_strip_edge_does_not_change_text_to_ipa_route(self):
+        route = self.graph.route("text", "ipa")
+        self.assertEqual(len(route), 1)
+        edge = route[0]
+        self.assertEqual(edge.src, "text")
+        self.assertEqual(edge.dst, "ipa")
+
+    def test_strip_preserves_cyrillic_native_letters(self):
+        out = self.graph.convert("мой родно́й край", "text-diacritized",
+                                  "text", lang="ru")
+        self.assertEqual(out, "мой родной край")
+        out2 = self.graph.convert("ёлка", "text-diacritized", "text", lang="ru")
+        self.assertEqual(out2, "ёлка")
+
+    def test_strip_preserves_latin_stress_lang_diacritics(self):
+        # native macron (ī) plus an added combining acute overlay
+        out = self.graph.convert("Rī́ga", "text-diacritized", "text",
+                                  lang="lv")
+        self.assertEqual(out, "Rīga")
+        out2 = self.graph.convert("Rīga", "text-diacritized", "text", lang="lv")
+        self.assertEqual(out2, "Rīga")
+
+    def test_strip_preserves_arabic_hamza(self):
+        out = self.graph.convert("أَحْمَد", "text-diacritized", "text", lang="ar")
+        self.assertEqual(out, "أحمد")
+
+    def test_strip_refuses_aragonese(self):
+        with self.assertRaises(ValueError):
+            self.graph.convert("Cristián", "text-diacritized", "text", lang="arg")
+
+    def test_add_diacritics_does_not_misroute_herero(self):
+        from scriptconv.phonemizers.base import GraphemePhonemizer
+        self.assertEqual(GraphemePhonemizer().add_diacritics("teste", "her"), "teste")
+
+
+if __name__ == "__main__":
+    unittest.main()

{scriptconv-0.0.4a2 → scriptconv-0.0.4a4}/tests/test_phonemizers_base.py

@@ -221,3 +221,85 @@ class TestModelForwarding(unittest.TestCase):
     def test_model_none_forwards_nothing(self):
         g = get_phonemizer(Phonemizer.GRAPHEMES, model=None)
         self.assertIsInstance(g, GraphemePhonemizer)
+
+
+class TestEuropeanPortugueseSenseDiacritics(unittest.TestCase):
+    """Real bifonia — it is on PyPI, so this exercises the actual backend."""
+
+    def test_thirst_sense_gets_closed_vowel(self):
+        out = GraphemePhonemizer().add_diacritics(
+            "Tenho muita sede hoje.", "pt")
+        self.assertEqual(out, "Tenho muita sêde hoje.")
+
+    def test_seat_sense_gets_open_vowel(self):
+        out = GraphemePhonemizer().add_diacritics(
+            "A sede da empresa fica em Lisboa.", "pt")
+        self.assertEqual(out, "A séde da empresa fica em Lisboa.")
+
+    def test_no_homograph_unchanged(self):
+        text = "O cão correu no jardim."
+        self.assertEqual(GraphemePhonemizer().add_diacritics(text, "pt"), text)
+
+    def test_brazilian_portuguese_excluded(self):
+        text = "Tenho muita sede hoje."
+        self.assertEqual(GraphemePhonemizer().add_diacritics(text, "pt-BR"), text)
+
+
+class TestEastSlavicStressRouting(unittest.TestCase):
+    """stressonnx is not yet on PyPI — routing is verified against a stub."""
+
+    def _stub(self, calls):
+        import types
+        mod = types.ModuleType("stressonnx")
+
+        def stress(text, lang, model=None):
+            calls.append((text, lang, model))
+            return "STRESSED"
+
+        mod.stress = stress
+        return mod
+
+    def test_russian_routes_to_stress_backend(self):
+        import sys
+        from unittest import mock
+        calls = []
+        with mock.patch.dict(sys.modules, {"stressonnx": self._stub(calls)}):
+            out = GraphemePhonemizer().add_diacritics(
+                "замок стоит", "ru", model="silero")
+        self.assertEqual(out, "STRESSED")
+        self.assertEqual(calls, [("замок стоит", "ru", "silero")])
+
+    def test_ukrainian_and_belarusian_route_too(self):
+        import sys
+        from unittest import mock
+        for lang in ("uk", "be"):
+            calls = []
+            with mock.patch.dict(sys.modules, {"stressonnx": self._stub(calls)}):
+                GraphemePhonemizer().add_diacritics("текст", lang, model="ruaccent")
+            self.assertEqual(calls, [("текст", lang, "ruaccent")], lang)
+
+    def test_additional_stressonnx_languages_route_too(self):
+        import sys
+        from unittest import mock
+        for lang in ("kk", "hy", "az-Latn"):
+            calls = []
+            with mock.patch.dict(sys.modules, {"stressonnx": self._stub(calls)}):
+                GraphemePhonemizer().add_diacritics("text", lang, model="simple")
+            self.assertEqual(calls, [("text", lang, "simple")], lang)
+
+    def test_berber_does_not_false_match_belarusian(self):
+        import sys
+        from unittest import mock
+        calls = []
+        with mock.patch.dict(sys.modules, {"stressonnx": self._stub(calls)}):
+            out = GraphemePhonemizer().add_diacritics("azul", "ber")
+        self.assertEqual(out, "azul")
+        self.assertEqual(calls, [])
+
+    def test_missing_stressonnx_raises_named_importerror(self):
+        import sys
+        from unittest import mock
+        with mock.patch.dict(sys.modules, {"stressonnx": None}):
+            with self.assertRaises(ImportError) as ctx:
+                GraphemePhonemizer().add_diacritics("замок", "ru")
+        self.assertIn("stressonnx", str(ctx.exception))