alifbe 0.2.0__tar.gz

alifbe-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

alifbe-0.2.0/PKG-INFO

@@ -0,0 +1,177 @@
+Metadata-Version: 2.4
+Name: alifbe
+Version: 0.2.0
+Summary: Robust normalization for real-world Uzbek text: apostrophe chaos, sh/s+h word boundaries, ş/ș confusables, the Turkish-I casing bug, ambiguous Cyrillic letters, and conversion between the pre-2026 and Sept-2026-reform Latin alphabets.
+Author: you
+License: MIT
+Project-URL: Homepage, https://github.com/yourname/alifbe
+Project-URL: Repository, https://github.com/yourname/alifbe
+Keywords: uzbek,unicode,normalization,transliteration,i18n,nlp,cyrillic,latin-alphabet
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Topic :: Software Development :: Internationalization
+Classifier: Typing :: Typed
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# alifbe
+
+Robust text normalization for real-world Uzbek text — the eight problems
+that quietly corrupt Uzbek data in most pipelines.
+
+| Problem | What alifbe does |
+|---|---|
+| 8+ different "apostrophe" characters (`'` `‘` `’` `` ` `` `´` ...) all meaning oʻ / gʻ / tutuq belgisi | Normalizes all of them to the two *correct* Unicode letters, based on context |
+| `"Isʼhoq"` → `"Işoq"` | Detects s+h morpheme boundaries (via apostrophe *and* a known-word list) so `sh` isn't wrongly merged into one letter |
+| `ş` vs `ș` (cedilla vs. comma-below — different code points, identical glyph) | Detects and normalizes confusable characters so search/dedup actually works |
+| The "Turkish I" bug (`.upper()`/`.lower()` under some locales turns `i` into `İ`) | Locale-independent, explicit-table casing that never produces `İ`/`ı` |
+| Cyrillic `е`, `ц`, `ё` are position/origin-dependent | Transliteration always returns warnings for ambiguous letters instead of silently guessing (and can `raise` instead, if you'd rather fail loudly) |
+| The **Sept 2026 alphabet reform** (sh→ş, ch→ç, oʻ→ö, gʻ→ğ) | `to_new_latin()` / `to_old_latin()` convert between the two orthographies, with brand-name/URL/code protection |
+| "Which script is this text even in?" | `detect_alphabet()` — cyrillic / old-latin / new-latin / mixed / unknown |
+| "oʻzbek", "özbek", and "ўзбек" are the same word, but `==` doesn't think so | `fold_search_key()` gives every spelling the same canonical key |
+
+## Install
+
+Not yet published to PyPI — the name `alifbe` is free, checked via
+`pip download alifbe` returning no match, but publishing itself is a
+manual step (PyPI account + 2FA + `twine upload`). Install locally for now:
+
+```bash
+pip install /path/to/alifbe          # normal install
+pip install -e /path/to/alifbe       # editable, for development
+```
+
+This also registers an `alifbe` command-line tool (see below).
+
+## Quick tour
+
+```python
+import alifbe as uz
+
+# 1. Apostrophes: every variant collapses to the correct, same string
+uz.normalize_apostrophes("o'zbek")     # -> "oʻzbek"  (U+02BB, turned comma)
+uz.normalize_apostrophes("o‘zbek")     # -> "oʻzbek"  (same result)
+uz.normalize_apostrophes("san'at")     # -> "sanʼat"  (U+02BC — different rule, not after o/g)
+
+# 2. "Isʼhoq" stays "Isʼhoq" -- sh is not wrongly merged into ш
+uz.latin_to_cyrillic("Isʼhoq").text        # -> "Исъҳоқ"  (not "Ишоқ")
+uz.latin_to_cyrillic("Ishoq").warnings     # non-empty: flags this as a known
+                                            # s+h boundary word even without
+                                            # the apostrophe
+
+# 3. ş vs ș
+uz.find_confusables("Kraiova munșasi")           # -> [Confusable(char='ș', codepoint='U+0219', ...)]
+uz.normalize_confusables("munşa") == uz.normalize_confusables("munșa")  # -> True
+
+# 4. Turkish I bug
+uz.uz_upper("olib")                        # -> "OLIB"  (never "OLİB")
+uz.find_turkish_i_corruption("OLİB")       # -> flags the İ as corruption
+uz.fix_turkish_i_corruption("OLİB")        # -> "OLIB"
+
+# 5. Ambiguous Cyrillic letters warn instead of silently guessing
+result = uz.cyrillic_to_latin("центр")
+result.text                                # -> "tsentr"  (best guess)
+result.warnings                            # -> non-empty, explains ц is ambiguous
+uz.cyrillic_to_latin("ёлғон", on_ambiguous="raise")  # -> raises instead
+
+# 6. The Sept 2026 alphabet reform
+uz.to_new_latin("Shahzoda Oʻzbekistonda choy ichdi").text
+# -> "Şahzoda Özbekistonda çoy içdi"
+uz.to_old_latin("Şahzoda Özbekistonda çoy içdi").text
+# -> "Shahzoda Oʻzbekistonda choy ichdi"
+
+# ... with brand names / URLs / code protected from conversion
+uz.to_new_latin("MyShop: sotib oling", protected_terms=["MyShop"]).text
+# -> "MyShop: sotib oling"   (MyShop untouched, rest still converts if applicable)
+uz.to_new_latin("See https://x.com/shahar for info").text
+# -> "See https://x.com/shahar for info"   (URL untouched)
+
+# 7. What script is this?
+uz.detect_alphabet("Shahzoda")   # -> AlphabetDetection(alphabet='old-latin', confidence=0.65)
+uz.detect_alphabet("Şahzoda")    # -> AlphabetDetection(alphabet='new-latin', confidence=0.7)
+uz.detect_alphabet("Шаҳзода")    # -> AlphabetDetection(alphabet='cyrillic', confidence=1.0)
+
+# 8. One search key regardless of script or apostrophe style
+uz.fold_search_key("o'zbek") == uz.fold_search_key("özbek") == uz.fold_search_key("ўзбек")
+# -> True (all fold to "özbek")
+```
+
+## Command line
+
+```bash
+echo "o'zbek" | alifbe normalize-apostrophes          # oʻzbek
+alifbe to-new-latin "Shahzoda choy ichdi"              # Şahzoda çoy içdi
+alifbe to-cyrillic "Ishoq"                              # Исъҳоқ  (+ warning on stderr)
+alifbe detect "Шаҳзода"                                 # cyrillic (confidence: 1.0)
+alifbe fold-key "oʻzbek"                                 # özbek
+alifbe check "OLİB"                                       # corruption: 'İ' (U+0130) at index 2
+```
+
+Every subcommand reads from the positional argument if given, or stdin
+otherwise — so it pipes cleanly. Conversion warnings go to stderr, so
+stdout stays clean for piping the result onward.
+
+## Design principle
+
+Every function that could plausibly get something wrong either:
+- makes the correct choice deterministically (apostrophes, casing,
+  alphabet-reform digraphs), or
+- tells you it's not sure, instead of guessing silently (Cyrillic
+  е/ц/ё, the sh/s+h boundary when no apostrophe survives).
+
+That second category is deliberate: a library that *always* looks
+confident is more dangerous than one that sometimes says "I'm not sure,
+here's my best guess and why." All conversion functions return the same
+`Result(text, warnings)` shape — `warnings` is empty when the library is
+confident, and populated (with a machine-readable `rule` plus a
+human-readable `message`) when it made a judgment call.
+
+## On the Sept 2026 alphabet reform
+
+Uzbekistan's Senate approved a bill on 10 September 2026 replacing the
+sh/ch/oʻ/gʻ digraphs with single letters ş/ç/ö/ğ. As of this writing the
+bill has been sent to the president and is **not yet in force** — school
+materials are expected to transition starting 2027. `to_new_latin()` is a
+forward-looking convenience, not a claim about which spelling is
+currently mandatory. Check lex.uz or the Ministry of Education for the
+authoritative status before treating conversion as required.
+
+## Known simplifications (read before relying on this for anything critical)
+
+- `SH_BOUNDARY_EXCEPTIONS` (in `exceptions_data.py`) is a small starting
+  list, not a linguistic corpus — extend it via `extra=` for your data,
+  and get a native speaker to review it before production use.
+- Cyrillic `ц` has no reliable positional rule (can be "s" or "ts"
+  depending on the word's origin); alifbe defaults to "ts" and always
+  warns. Cyrillic `е` uses a positional heuristic (ye at word start/after
+  a vowel, else e) which is usually right but isn't a certainty for
+  borrowed words.
+- `latin_to_cyrillic("c")` maps to "к" (since old-orthography Uzbek Latin
+  has no standalone "c" outside the "ch" digraph) — for text with
+  loanwords spelled with a bare "c", double-check the output.
+- `fold_search_key_loose()` is intentionally lossy (strips ö/ğ/ş/ç
+  diacritics and apostrophes) — never use it as a unique key, only for
+  fuzzy "did you mean" style matching.
+
+## Extending the sh-boundary word list
+
+```python
+uz.latin_to_cyrillic("Asʼhad", extra_sh_exceptions={"ashad"})
+```
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+102 tests across normalization, transliteration, the alphabet-reform
+converter, script detection, search-key folding, and the CLI.

alifbe-0.2.0/README.md

@@ -0,0 +1,154 @@
+# alifbe
+
+Robust text normalization for real-world Uzbek text — the eight problems
+that quietly corrupt Uzbek data in most pipelines.
+
+| Problem | What alifbe does |
+|---|---|
+| 8+ different "apostrophe" characters (`'` `‘` `’` `` ` `` `´` ...) all meaning oʻ / gʻ / tutuq belgisi | Normalizes all of them to the two *correct* Unicode letters, based on context |
+| `"Isʼhoq"` → `"Işoq"` | Detects s+h morpheme boundaries (via apostrophe *and* a known-word list) so `sh` isn't wrongly merged into one letter |
+| `ş` vs `ș` (cedilla vs. comma-below — different code points, identical glyph) | Detects and normalizes confusable characters so search/dedup actually works |
+| The "Turkish I" bug (`.upper()`/`.lower()` under some locales turns `i` into `İ`) | Locale-independent, explicit-table casing that never produces `İ`/`ı` |
+| Cyrillic `е`, `ц`, `ё` are position/origin-dependent | Transliteration always returns warnings for ambiguous letters instead of silently guessing (and can `raise` instead, if you'd rather fail loudly) |
+| The **Sept 2026 alphabet reform** (sh→ş, ch→ç, oʻ→ö, gʻ→ğ) | `to_new_latin()` / `to_old_latin()` convert between the two orthographies, with brand-name/URL/code protection |
+| "Which script is this text even in?" | `detect_alphabet()` — cyrillic / old-latin / new-latin / mixed / unknown |
+| "oʻzbek", "özbek", and "ўзбек" are the same word, but `==` doesn't think so | `fold_search_key()` gives every spelling the same canonical key |
+
+## Install
+
+Not yet published to PyPI — the name `alifbe` is free, checked via
+`pip download alifbe` returning no match, but publishing itself is a
+manual step (PyPI account + 2FA + `twine upload`). Install locally for now:
+
+```bash
+pip install /path/to/alifbe          # normal install
+pip install -e /path/to/alifbe       # editable, for development
+```
+
+This also registers an `alifbe` command-line tool (see below).
+
+## Quick tour
+
+```python
+import alifbe as uz
+
+# 1. Apostrophes: every variant collapses to the correct, same string
+uz.normalize_apostrophes("o'zbek")     # -> "oʻzbek"  (U+02BB, turned comma)
+uz.normalize_apostrophes("o‘zbek")     # -> "oʻzbek"  (same result)
+uz.normalize_apostrophes("san'at")     # -> "sanʼat"  (U+02BC — different rule, not after o/g)
+
+# 2. "Isʼhoq" stays "Isʼhoq" -- sh is not wrongly merged into ш
+uz.latin_to_cyrillic("Isʼhoq").text        # -> "Исъҳоқ"  (not "Ишоқ")
+uz.latin_to_cyrillic("Ishoq").warnings     # non-empty: flags this as a known
+                                            # s+h boundary word even without
+                                            # the apostrophe
+
+# 3. ş vs ș
+uz.find_confusables("Kraiova munșasi")           # -> [Confusable(char='ș', codepoint='U+0219', ...)]
+uz.normalize_confusables("munşa") == uz.normalize_confusables("munșa")  # -> True
+
+# 4. Turkish I bug
+uz.uz_upper("olib")                        # -> "OLIB"  (never "OLİB")
+uz.find_turkish_i_corruption("OLİB")       # -> flags the İ as corruption
+uz.fix_turkish_i_corruption("OLİB")        # -> "OLIB"
+
+# 5. Ambiguous Cyrillic letters warn instead of silently guessing
+result = uz.cyrillic_to_latin("центр")
+result.text                                # -> "tsentr"  (best guess)
+result.warnings                            # -> non-empty, explains ц is ambiguous
+uz.cyrillic_to_latin("ёлғон", on_ambiguous="raise")  # -> raises instead
+
+# 6. The Sept 2026 alphabet reform
+uz.to_new_latin("Shahzoda Oʻzbekistonda choy ichdi").text
+# -> "Şahzoda Özbekistonda çoy içdi"
+uz.to_old_latin("Şahzoda Özbekistonda çoy içdi").text
+# -> "Shahzoda Oʻzbekistonda choy ichdi"
+
+# ... with brand names / URLs / code protected from conversion
+uz.to_new_latin("MyShop: sotib oling", protected_terms=["MyShop"]).text
+# -> "MyShop: sotib oling"   (MyShop untouched, rest still converts if applicable)
+uz.to_new_latin("See https://x.com/shahar for info").text
+# -> "See https://x.com/shahar for info"   (URL untouched)
+
+# 7. What script is this?
+uz.detect_alphabet("Shahzoda")   # -> AlphabetDetection(alphabet='old-latin', confidence=0.65)
+uz.detect_alphabet("Şahzoda")    # -> AlphabetDetection(alphabet='new-latin', confidence=0.7)
+uz.detect_alphabet("Шаҳзода")    # -> AlphabetDetection(alphabet='cyrillic', confidence=1.0)
+
+# 8. One search key regardless of script or apostrophe style
+uz.fold_search_key("o'zbek") == uz.fold_search_key("özbek") == uz.fold_search_key("ўзбек")
+# -> True (all fold to "özbek")
+```
+
+## Command line
+
+```bash
+echo "o'zbek" | alifbe normalize-apostrophes          # oʻzbek
+alifbe to-new-latin "Shahzoda choy ichdi"              # Şahzoda çoy içdi
+alifbe to-cyrillic "Ishoq"                              # Исъҳоқ  (+ warning on stderr)
+alifbe detect "Шаҳзода"                                 # cyrillic (confidence: 1.0)
+alifbe fold-key "oʻzbek"                                 # özbek
+alifbe check "OLİB"                                       # corruption: 'İ' (U+0130) at index 2
+```
+
+Every subcommand reads from the positional argument if given, or stdin
+otherwise — so it pipes cleanly. Conversion warnings go to stderr, so
+stdout stays clean for piping the result onward.
+
+## Design principle
+
+Every function that could plausibly get something wrong either:
+- makes the correct choice deterministically (apostrophes, casing,
+  alphabet-reform digraphs), or
+- tells you it's not sure, instead of guessing silently (Cyrillic
+  е/ц/ё, the sh/s+h boundary when no apostrophe survives).
+
+That second category is deliberate: a library that *always* looks
+confident is more dangerous than one that sometimes says "I'm not sure,
+here's my best guess and why." All conversion functions return the same
+`Result(text, warnings)` shape — `warnings` is empty when the library is
+confident, and populated (with a machine-readable `rule` plus a
+human-readable `message`) when it made a judgment call.
+
+## On the Sept 2026 alphabet reform
+
+Uzbekistan's Senate approved a bill on 10 September 2026 replacing the
+sh/ch/oʻ/gʻ digraphs with single letters ş/ç/ö/ğ. As of this writing the
+bill has been sent to the president and is **not yet in force** — school
+materials are expected to transition starting 2027. `to_new_latin()` is a
+forward-looking convenience, not a claim about which spelling is
+currently mandatory. Check lex.uz or the Ministry of Education for the
+authoritative status before treating conversion as required.
+
+## Known simplifications (read before relying on this for anything critical)
+
+- `SH_BOUNDARY_EXCEPTIONS` (in `exceptions_data.py`) is a small starting
+  list, not a linguistic corpus — extend it via `extra=` for your data,
+  and get a native speaker to review it before production use.
+- Cyrillic `ц` has no reliable positional rule (can be "s" or "ts"
+  depending on the word's origin); alifbe defaults to "ts" and always
+  warns. Cyrillic `е` uses a positional heuristic (ye at word start/after
+  a vowel, else e) which is usually right but isn't a certainty for
+  borrowed words.
+- `latin_to_cyrillic("c")` maps to "к" (since old-orthography Uzbek Latin
+  has no standalone "c" outside the "ch" digraph) — for text with
+  loanwords spelled with a bare "c", double-check the output.
+- `fold_search_key_loose()` is intentionally lossy (strips ö/ğ/ş/ç
+  diacritics and apostrophes) — never use it as a unique key, only for
+  fuzzy "did you mean" style matching.
+
+## Extending the sh-boundary word list
+
+```python
+uz.latin_to_cyrillic("Asʼhad", extra_sh_exceptions={"ashad"})
+```
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+102 tests across normalization, transliteration, the alphabet-reform
+converter, script detection, search-key folding, and the CLI.

alifbe-0.2.0/alifbe/__init__.py

@@ -0,0 +1,85 @@
+"""
+alifbe — robust text normalization utilities for the Uzbek language.
+
+Handles the sharp edges of real-world Uzbek text:
+
+  * Apostrophe chaos (oʻ / gʻ / tutuq belgisi typed with 8+ different glyphs)
+  * The "Isʼhoq" problem (s+h is not always the "sh" digraph)
+  * ş vs ș confusable Unicode lookalikes
+  * The "Turkish I" casing bug
+  * Ambiguous Cyrillic letters (е, ц, ё) that depend on context
+  * The pre-2026 vs. post-reform (Sept 2026 Senate-approved) Latin
+    orthographies -- sh/ch/oʻ/gʻ vs ş/ç/ö/ğ
+  * Cross-script/cross-orthography search-key folding, so "oʻzbek",
+    "özbek", and "ўзбек" are recognized as the same word
+
+Nothing here silently guesses when a guess would corrupt data — ambiguous
+cases are surfaced as warnings so the caller (or a human) can decide.
+"""
+
+from .apostrophe import normalize_apostrophes, TURNED_COMMA, APOSTROPHE
+from .confusables import normalize_confusables, find_confusables, Confusable
+from .case import (
+    uz_lower,
+    uz_upper,
+    uz_title,
+    find_turkish_i_corruption,
+    fix_turkish_i_corruption,
+    CorruptionHit,
+    TURKISH_DOTTED_I,
+    TURKISH_DOTLESS_I,
+)
+from .exceptions_data import SH_BOUNDARY_EXCEPTIONS, is_sh_boundary_word
+from .result import Result, Warning_
+from .transliterate import (
+    TransliterationResult,
+    AmbiguousTransliterationError,
+    latin_to_cyrillic,
+    cyrillic_to_latin,
+)
+from .oldnew import ConversionResult, to_new_latin, to_old_latin
+from .detect import detect_alphabet, AlphabetDetection
+from .searchkey import fold_search_key, fold_search_key_loose
+
+__all__ = [
+    # apostrophe
+    "normalize_apostrophes",
+    "TURNED_COMMA",
+    "APOSTROPHE",
+    # confusables
+    "normalize_confusables",
+    "find_confusables",
+    "Confusable",
+    # case
+    "uz_lower",
+    "uz_upper",
+    "uz_title",
+    "find_turkish_i_corruption",
+    "fix_turkish_i_corruption",
+    "CorruptionHit",
+    "TURKISH_DOTTED_I",
+    "TURKISH_DOTLESS_I",
+    # exceptions
+    "SH_BOUNDARY_EXCEPTIONS",
+    "is_sh_boundary_word",
+    # shared result type
+    "Result",
+    "Warning_",
+    # transliteration (Latin <-> Cyrillic)
+    "TransliterationResult",
+    "AmbiguousTransliterationError",
+    "latin_to_cyrillic",
+    "cyrillic_to_latin",
+    # 2026 reform (old Latin <-> new Latin)
+    "ConversionResult",
+    "to_new_latin",
+    "to_old_latin",
+    # detection
+    "detect_alphabet",
+    "AlphabetDetection",
+    # search keys
+    "fold_search_key",
+    "fold_search_key_loose",
+]
+
+__version__ = "0.2.0"

alifbe-0.2.0/alifbe/__main__.py

@@ -0,0 +1,95 @@
+"""
+Command-line interface: `python -m alifbe <command> [text]`.
+
+Reads TEXT from the positional argument if given, otherwise from stdin
+(so it works well in pipelines). Warnings, if any, go to stderr so stdout
+stays clean for piping the converted text onward.
+"""
+
+import argparse
+import sys
+
+from . import (
+    normalize_apostrophes,
+    normalize_confusables,
+    uz_upper,
+    uz_lower,
+    uz_title,
+    find_turkish_i_corruption,
+    latin_to_cyrillic,
+    cyrillic_to_latin,
+    to_new_latin,
+    to_old_latin,
+    detect_alphabet,
+    fold_search_key,
+)
+
+
+def _read_text(args) -> str:
+    if args.text is not None:
+        return args.text
+    return sys.stdin.read().rstrip("\n")
+
+
+def _emit(result_text: str, warnings=()) -> None:
+    print(result_text)
+    for w in warnings:
+        print(f"warning: {w.message}", file=sys.stderr)
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="alifbe",
+        description="Normalize and convert Uzbek text (apostrophes, ş/ș confusables, "
+                     "old/new Latin orthography, Cyrillic transliteration).",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    for name in ("normalize-apostrophes", "normalize-confusables",
+                 "upper", "lower", "title",
+                 "to-cyrillic", "to-latin", "to-new-latin", "to-old-latin",
+                 "detect", "fold-key", "check"):
+        p = sub.add_parser(name)
+        p.add_argument("text", nargs="?", help="Text to process (reads stdin if omitted)")
+
+    args = parser.parse_args(argv)
+    text = _read_text(args)
+
+    if args.command == "normalize-apostrophes":
+        _emit(normalize_apostrophes(text))
+    elif args.command == "normalize-confusables":
+        _emit(normalize_confusables(text))
+    elif args.command == "upper":
+        _emit(uz_upper(text))
+    elif args.command == "lower":
+        _emit(uz_lower(text))
+    elif args.command == "title":
+        _emit(uz_title(text))
+    elif args.command == "to-cyrillic":
+        result = latin_to_cyrillic(text)
+        _emit(result.text, result.warnings)
+    elif args.command == "to-latin":
+        result = cyrillic_to_latin(text)
+        _emit(result.text, result.warnings)
+    elif args.command == "to-new-latin":
+        _emit(to_new_latin(text).text)
+    elif args.command == "to-old-latin":
+        _emit(to_old_latin(text).text)
+    elif args.command == "detect":
+        detection = detect_alphabet(text)
+        print(f"{detection.alphabet} (confidence: {detection.confidence})")
+    elif args.command == "fold-key":
+        _emit(fold_search_key(text))
+    elif args.command == "check":
+        hits = find_turkish_i_corruption(text)
+        if hits:
+            for h in hits:
+                print(f"corruption: '{h.char}' ({h.codepoint}) at index {h.index}")
+            return 1
+        print("clean: no Turkish-I corruption found")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

alifbe-0.2.0/alifbe/apostrophe.py

@@ -0,0 +1,71 @@
+"""
+Normalize the ~8+ different characters people type for Uzbek apostrophes
+into the two *correct* Unicode code points:
+
+  ʻ  U+02BB  MODIFIER LETTER TURNED COMMA   -> used only inside oʻ / gʻ
+  ʼ  U+02BC  MODIFIER LETTER APOSTROPHE     -> the tutuq belgisi (glottal
+                                               stop) used everywhere else,
+                                               e.g. sanʼat, Isʼhoq, mashʼal
+
+People type either of these using: ' (U+0027), ‘ ’ (curly quotes),
+` ´ (grave/acute accent), ʹ (prime), ʻʼ themselves, and more — often
+inconsistently within the same document.
+
+The rule used to decide *which* canonical character a lookalike becomes:
+if it immediately follows o/O/g/G it is part of the oʻ/gʻ letter (turned
+comma); otherwise it is the tutuq belgisi (plain apostrophe).
+"""
+
+import re
+import unicodedata
+
+TURNED_COMMA = "\u02BB"  # ʻ — part of the letters oʻ / gʻ
+APOSTROPHE = "\u02BC"  # ʼ — tutuq belgisi / glottal stop marker
+
+# Every character that, in the wild, gets used to mean "apostrophe" in
+# Uzbek text. This list is intentionally generous.
+_APOSTROPHE_LOOKALIKES = {
+    "'",  # U+0027 APOSTROPHE (ascii)
+    "\u2018",  # ‘ LEFT SINGLE QUOTATION MARK
+    "\u2019",  # ’ RIGHT SINGLE QUOTATION MARK
+    "\u201A",  # ‚ SINGLE LOW-9 QUOTATION MARK
+    "\u201B",  # ‛ SINGLE HIGH-REVERSED-9 QUOTATION MARK
+    "`",  # U+0060 GRAVE ACCENT
+    "\u00B4",  # ´ ACUTE ACCENT
+    "\u02B9",  # ʹ MODIFIER LETTER PRIME
+    "\u02BA",  # ʺ MODIFIER LETTER DOUBLE PRIME
+    "\u02BB",  # ʻ MODIFIER LETTER TURNED COMMA (already canonical form #1)
+    "\u02BC",  # ʼ MODIFIER LETTER APOSTROPHE (already canonical form #2)
+    "\u02BD",  # ʽ MODIFIER LETTER REVERSED COMMA
+    "\u02BE",  # ʾ MODIFIER LETTER RIGHT HALF RING
+    "\u02BF",  # ʿ MODIFIER LETTER LEFT HALF RING
+    "\u02C8",  # ˈ MODIFIER LETTER VERTICAL LINE (stress mark, sometimes misused)
+    "\u02CA",  # ˊ MODIFIER LETTER ACUTE ACCENT
+    "\u02EE",  # ˮ MODIFIER LETTER DOUBLE APOSTROPHE
+    "\u2032",  # ′ PRIME
+    "\u0301",  # combining acute accent (when it ends up standalone)
+}
+
+_PATTERN = re.compile("[" + "".join(re.escape(c) for c in _APOSTROPHE_LOOKALIKES) + "]")
+
+_OG_TRIGGERS = set("oOgG")
+
+
+def normalize_apostrophes(text: str) -> str:
+    """
+    Rewrite every apostrophe-lookalike in `text` to the correct Uzbek
+    character: U+02BB (ʻ) right after o/g, otherwise U+02BC (ʼ).
+
+    NFC-normalizes first so precomposed vs. decomposed forms don't sneak
+    through as false negatives.
+    """
+    text = unicodedata.normalize("NFC", text)
+
+    def repl(match: "re.Match[str]") -> str:
+        idx = match.start()
+        prev_char = text[idx - 1] if idx > 0 else ""
+        if prev_char in _OG_TRIGGERS:
+            return TURNED_COMMA
+        return APOSTROPHE
+
+    return _PATTERN.sub(repl, text)