sindhinorm 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Arsalan Kaleem
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,13 @@
1
+ The bundled word-frequency list src/sindhinorm/data/sd_lexicon.json.gz is derived from
2
+ Sindhi Wikipedia (dump of 2023-11-01, distributed as the Hugging Face dataset
3
+ wikimedia/wikipedia, config 20231101.sd). Wikipedia text is licensed under the Creative
4
+ Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0),
5
+ https://creativecommons.org/licenses/by-sa/4.0/ . Attribution: Wikipedia contributors,
6
+ https://sd.wikipedia.org/ .
7
+
8
+ The list contains every word (after NFC and zero-width cleanup) that occurs at least twice
9
+ in 66,141 article lines, with its count. It is distributed under CC BY-SA 4.0. If you
10
+ redistribute the list or a modified version, keep this notice and use the same licence.
11
+
12
+ The native-speaker judgements encoded in NATIVE_KEEP (src/sindhinorm/core.py) are part of
13
+ the MIT-licensed code.
@@ -0,0 +1,102 @@
1
+ Metadata-Version: 2.4
2
+ Name: sindhinorm
3
+ Version: 0.1.0
4
+ Summary: Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)
5
+ Author: Arsalan Kaleem
6
+ License-Expression: MIT AND CC-BY-SA-4.0
7
+ Project-URL: Homepage, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm
8
+ Project-URL: Evaluation, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md
9
+ Keywords: sindhi,nlp,normalization,unicode,arabic-script,spelling
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Science/Research
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Topic :: Text Processing :: Linguistic
14
+ Requires-Python: >=3.10
15
+ Description-Content-Type: text/markdown
16
+ License-File: LICENSE
17
+ License-File: LICENSE-DATA
18
+ Dynamic: license-file
19
+
20
+ # sindhinorm
21
+
22
+ Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
23
+
24
+ Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
25
+ Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
26
+ **ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
27
+ same but are different characters to software, so search, counting and NLP models treat one
28
+ word as several.
29
+
30
+ A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
31
+ letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
32
+ using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
33
+ native Sindhi speaker judged to be different words.
34
+
35
+ ```bash
36
+ pip install sindhinorm
37
+ ```
38
+
39
+ ```python
40
+ from sindhinorm import normalize, explain
41
+
42
+ normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
43
+ normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
44
+
45
+ explain("ھن آهی کري")
46
+ # [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
47
+ # {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
48
+ # {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
49
+ ```
50
+
51
+ Command line:
52
+
53
+ ```bash
54
+ sindhinorm < input.txt > output.txt
55
+ sindhinorm input.txt -o output.txt
56
+ sindhinorm explain "ھن آهی کري"
57
+ sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
58
+ sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
59
+ ```
60
+
61
+ ## Rules
62
+
63
+ 1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
64
+ two letters (native review found those mark compound and ال-name joins).
65
+ 2. Candidate spellings: every combination of look-alike letters at the word's yeh
66
+ {ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
67
+ the rule allows the merge.
68
+ 3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
69
+ frequent.
70
+ 4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
71
+ rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
72
+ 5. 24 pairs judged different or unsure by a native reviewer are never merged.
73
+ 6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
74
+
75
+ `explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
76
+ `native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
77
+ `already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
78
+
79
+ ## How well it works
80
+
81
+ Measured with the bundled lexicon (details and scripts:
82
+ [MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
83
+
84
+ | Test | Result |
85
+ |---|---|
86
+ | Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
87
+ | Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
88
+ | Clean words changed | 1.35% |
89
+
90
+ **Limits.**
91
+ * Frequency cannot always tell a rare real word from a misspelling. The native keep-list
92
+ covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
93
+ with precision 0.87.
94
+ * The canonical spelling follows the Wikipedia majority, not a published standard. For
95
+ example ھن → هن; some writers prefer ھ.
96
+ * One native reviewer. Sindhi only (no Urdu mode).
97
+
98
+ ## Licences
99
+
100
+ * Code: MIT (`LICENSE`).
101
+ * Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
102
+ (2023-11-01): 61,718 words seen at least twice, with counts.
@@ -0,0 +1,83 @@
1
+ # sindhinorm
2
+
3
+ Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
4
+
5
+ Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
6
+ Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
7
+ **ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
8
+ same but are different characters to software, so search, counting and NLP models treat one
9
+ word as several.
10
+
11
+ A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
12
+ letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
13
+ using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
14
+ native Sindhi speaker judged to be different words.
15
+
16
+ ```bash
17
+ pip install sindhinorm
18
+ ```
19
+
20
+ ```python
21
+ from sindhinorm import normalize, explain
22
+
23
+ normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
24
+ normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
25
+
26
+ explain("ھن آهی کري")
27
+ # [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
28
+ # {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
29
+ # {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
30
+ ```
31
+
32
+ Command line:
33
+
34
+ ```bash
35
+ sindhinorm < input.txt > output.txt
36
+ sindhinorm input.txt -o output.txt
37
+ sindhinorm explain "ھن آهی کري"
38
+ sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
39
+ sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
40
+ ```
41
+
42
+ ## Rules
43
+
44
+ 1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
45
+ two letters (native review found those mark compound and ال-name joins).
46
+ 2. Candidate spellings: every combination of look-alike letters at the word's yeh
47
+ {ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
48
+ the rule allows the merge.
49
+ 3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
50
+ frequent.
51
+ 4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
52
+ rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
53
+ 5. 24 pairs judged different or unsure by a native reviewer are never merged.
54
+ 6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
55
+
56
+ `explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
57
+ `native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
58
+ `already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
59
+
60
+ ## How well it works
61
+
62
+ Measured with the bundled lexicon (details and scripts:
63
+ [MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
64
+
65
+ | Test | Result |
66
+ |---|---|
67
+ | Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
68
+ | Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
69
+ | Clean words changed | 1.35% |
70
+
71
+ **Limits.**
72
+ * Frequency cannot always tell a rare real word from a misspelling. The native keep-list
73
+ covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
74
+ with precision 0.87.
75
+ * The canonical spelling follows the Wikipedia majority, not a published standard. For
76
+ example ھن → هن; some writers prefer ھ.
77
+ * One native reviewer. Sindhi only (no Urdu mode).
78
+
79
+ ## Licences
80
+
81
+ * Code: MIT (`LICENSE`).
82
+ * Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
83
+ (2023-11-01): 61,718 words seen at least twice, with counts.
@@ -0,0 +1,34 @@
1
+ [build-system]
2
+ requires = ["setuptools>=77"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "sindhinorm"
7
+ version = "0.1.0"
8
+ description = "Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)"
9
+ readme = "README.md"
10
+ requires-python = ">=3.10"
11
+ license = "MIT AND CC-BY-SA-4.0"
12
+ license-files = ["LICENSE", "LICENSE-DATA"]
13
+ authors = [{ name = "Arsalan Kaleem" }]
14
+ keywords = ["sindhi", "nlp", "normalization", "unicode", "arabic-script", "spelling"]
15
+ classifiers = [
16
+ "Development Status :: 4 - Beta",
17
+ "Intended Audience :: Science/Research",
18
+ "Programming Language :: Python :: 3",
19
+ "Topic :: Text Processing :: Linguistic",
20
+ ]
21
+ dependencies = []
22
+
23
+ [project.urls]
24
+ Homepage = "https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm"
25
+ Evaluation = "https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md"
26
+
27
+ [project.scripts]
28
+ sindhinorm = "sindhinorm.cli:main"
29
+
30
+ [tool.setuptools.packages.find]
31
+ where = ["src"]
32
+
33
+ [tool.setuptools.package-data]
34
+ sindhinorm = ["data/*.json.gz", "py.typed"]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,44 @@
1
+ """sindhinorm: corpus-guided, native-validated spelling normalization for Sindhi.
2
+
3
+ >>> from sindhinorm import normalize
4
+ >>> normalize("هي ڪتاب آهی") # Farsi yeh in the last word
5
+ 'هي ڪتاب آهي'
6
+ """
7
+
8
+ from __future__ import annotations
9
+
10
+ from functools import lru_cache
11
+
12
+ from .core import (
13
+ CLASSES,
14
+ LEXICAL,
15
+ MEMBERS,
16
+ NATIVE_KEEP,
17
+ SAFE_OOV,
18
+ Normalizer,
19
+ build_lexicon,
20
+ clean_zero_width,
21
+ tokens,
22
+ )
23
+
24
+ __version__ = "0.1.0"
25
+ __all__ = [
26
+ "CLASSES", "LEXICAL", "MEMBERS", "NATIVE_KEEP", "SAFE_OOV", "Normalizer", "build_lexicon",
27
+ "clean_zero_width", "default_normalizer", "explain", "normalize", "tokens",
28
+ ] # fmt: skip
29
+
30
+
31
+ @lru_cache(maxsize=1)
32
+ def default_normalizer() -> Normalizer:
33
+ """The shared normalizer with the bundled lexicon (loaded once)."""
34
+ return Normalizer.default()
35
+
36
+
37
+ def normalize(text: str) -> str:
38
+ """Normalize Sindhi text with the bundled lexicon."""
39
+ return default_normalizer().normalize(text)
40
+
41
+
42
+ def explain(text: str) -> list[dict]:
43
+ """Per-word decisions for ``text`` with the bundled lexicon."""
44
+ return default_normalizer().explain(text)
@@ -0,0 +1,83 @@
1
+ """Command line: ``sindhinorm`` (normalize stdin or files), ``sindhinorm explain``, ``build-lexicon``.
2
+
3
+ sindhinorm < in.txt > out.txt
4
+ sindhinorm input.txt -o output.txt
5
+ sindhinorm explain "هي ڪتاب آهی"
6
+ sindhinorm build-lexicon corpus.txt -o my_lexicon.json.gz
7
+ sindhinorm -l my_lexicon.json.gz < in.txt
8
+ """
9
+
10
+ from __future__ import annotations
11
+
12
+ import argparse
13
+ import gzip
14
+ import json
15
+ import sys
16
+ from pathlib import Path
17
+
18
+ from . import __version__
19
+ from .core import Normalizer, build_lexicon
20
+
21
+
22
+ def _load(a: argparse.Namespace) -> Normalizer:
23
+ kw = {"lex_ratio": a.lex_ratio, "lex_max_count": a.lex_max_count}
24
+ return Normalizer.load(a.lexicon, **kw) if a.lexicon else Normalizer.default(**kw)
25
+
26
+
27
+ def main(argv: list[str] | None = None) -> int:
28
+ """Entry point."""
29
+ for s in (sys.stdin, sys.stdout):
30
+ if hasattr(s, "reconfigure"):
31
+ s.reconfigure(encoding="utf-8") # type: ignore[union-attr]
32
+ argv = list(sys.argv[1:] if argv is None else argv)
33
+ cmd = argv.pop(0) if argv and argv[0] in ("explain", "build-lexicon") else "normalize"
34
+ ap = argparse.ArgumentParser(prog=f"sindhinorm {cmd}".replace(" normalize", ""), description=__doc__,
35
+ formatter_class=argparse.RawDescriptionHelpFormatter) # fmt: skip
36
+ ap.add_argument("--version", action="version", version=f"sindhinorm {__version__}")
37
+ if cmd == "build-lexicon":
38
+ ap.add_argument("corpus", type=Path)
39
+ ap.add_argument("-o", "--out", type=Path, required=True)
40
+ ap.add_argument("--min-count", type=int, default=2)
41
+ a = ap.parse_args(argv)
42
+ with a.corpus.open(encoding="utf-8") as f:
43
+ lex = build_lexicon(f)
44
+ data = json.dumps({w: n for w, n in lex.most_common() if n >= a.min_count}, ensure_ascii=False)
45
+ raw = data.encode("utf-8")
46
+ a.out.write_bytes(gzip.compress(raw) if a.out.suffix == ".gz" else raw)
47
+ print(f"wrote {a.out} ({sum(n >= a.min_count for n in lex.values())} words)", file=sys.stderr)
48
+ return 0
49
+ ap.add_argument("-l", "--lexicon", type=Path, default=None, help="custom lexicon (default: bundled)")
50
+ ap.add_argument("--lex-ratio", type=float, default=50.0)
51
+ ap.add_argument("--lex-max-count", type=int, default=100)
52
+ if cmd == "explain":
53
+ ap.add_argument("text", nargs="+")
54
+ ap.add_argument("--json", action="store_true")
55
+ a = ap.parse_args(argv)
56
+ rows = _load(a).explain(" ".join(a.text))
57
+ if a.json:
58
+ print(json.dumps(rows, ensure_ascii=False, indent=1))
59
+ else:
60
+ for r in rows:
61
+ arrow = "->" if r["output"] != r["word"] else "=="
62
+ print(
63
+ f"{r['word']} ({r['count_in']}) {arrow} {r['output']} ({r['count_out']}) [{r['rule']}]"
64
+ )
65
+ return 0
66
+ ap.add_argument("inputs", nargs="*", type=Path, help="files (default: stdin)")
67
+ ap.add_argument("-o", "--out", type=Path, default=None)
68
+ a = ap.parse_args(argv)
69
+ norm = _load(a)
70
+ out = a.out.open("w", encoding="utf-8", newline="") if a.out else sys.stdout
71
+ try:
72
+ for src in a.inputs or [None]:
73
+ f = src.open(encoding="utf-8", newline="") if src else sys.stdin
74
+ for line in f:
75
+ out.write(norm.normalize(line))
76
+ finally:
77
+ if a.out:
78
+ out.close()
79
+ return 0
80
+
81
+
82
+ if __name__ == "__main__":
83
+ raise SystemExit(main())
@@ -0,0 +1,221 @@
1
+ """sindhinorm: a corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script).
2
+
3
+ Sindhi text typed on Urdu, Persian or Arabic keyboards mixes look-alike code points
4
+ (Farsi yeh ی for Sindhi ي, Urdu heh-goal ہ for ه, keheh ک or Arabic kaf ك for Sindhi ڪ) and
5
+ stray zero-width characters. Some of these are pure encoding variants; others are *different
6
+ Sindhi letters* (ک and ڪ are distinct letters: کي "to" vs ڪي "some"). A fixed character map
7
+ therefore either misses variants or corrupts real words.
8
+
9
+ sdnorm decides per **word**, using a frequency lexicon built from a Sindhi corpus:
10
+
11
+ * Letters are grouped into look-alike classes: yeh {ي ی ى ے}, heh {ه ھ ہ}, kaf {ڪ ک ك}.
12
+ * A word's candidate spellings are all combinations of class members at its class positions;
13
+ the canonical spelling is the candidate most frequent in the lexicon.
14
+ * **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
15
+ frequent. A native reviewer judged these SAME in nearly all reviewed pairs.
16
+ * **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only when the
17
+ rare form is at most ``lex_max_count`` times attested **and** the candidate is at least
18
+ ``lex_ratio`` times more frequent, so that two real words (کي / ڪي) are never merged.
19
+ * Pairs a native reviewer judged DIFFERENT or UNSURE (``NATIVE_KEEP``) are never merged.
20
+ * Words absent from the lexicon get only the safest encoding folds (ی ى → ي, ہ → ه).
21
+ * Zero-width non-joiners/joiners that cannot affect shaping (next to a space, punctuation,
22
+ a non-letter, a string edge, or another zero-width character) are removed; a ZWNJ between
23
+ two letters is kept, because native review found those mark compound and ال-name joins.
24
+
25
+ Thresholds were fixed before evaluation. The evaluation (agreement with native judgements,
26
+ restoration of keyboard-substituted text) lives in the MorphoTok-SD-UR repository:
27
+ https://github.com/ArsalanKaleem/MorphoTok-SD-UR (``docs/sdnorm.md``).
28
+ """
29
+
30
+ from __future__ import annotations
31
+
32
+ import gzip
33
+ import itertools
34
+ import json
35
+ import unicodedata
36
+ from collections import Counter
37
+ from collections.abc import Iterable, Iterator
38
+ from dataclasses import dataclass, field
39
+ from importlib import resources
40
+ from pathlib import Path
41
+
42
+ ZWNJ, ZWJ = "‌", "‍"
43
+ CLASSES: dict[str, str] = {
44
+ "ي": "yeh", "ی": "yeh", "ى": "yeh", "ے": "yeh",
45
+ "ه": "heh", "ھ": "heh", "ہ": "heh",
46
+ "ڪ": "kaf", "ک": "kaf", "ك": "kaf",
47
+ } # fmt: skip
48
+ MEMBERS: dict[str, tuple[str, ...]] = {
49
+ "yeh": ("ي", "ی", "ى", "ے"),
50
+ "heh": ("ه", "ھ", "ہ"),
51
+ "kaf": ("ڪ", "ک", "ك"),
52
+ }
53
+ LEXICAL = frozenset("ڪکكے") # letters whose substitution can change the word
54
+ SAFE_OOV = {"ی": "ي", "ى": "ي", "ہ": "ه"} # encoding folds applied to unknown words
55
+ MAX_CANDIDATES = 4096
56
+
57
+ # Pairs (variant -> would-be canonical) that a native Sindhi reviewer judged DIFFERENT or
58
+ # UNSURE (review of 150 Wikipedia pairs, 2026-11-09; docs/findings_human_review.md §4).
59
+ NATIVE_KEEP: frozenset[tuple[str, str]] = frozenset({
60
+ ("جے", "جي"), ("ڪي", "کي"), ("کے", "کي"), ("کی", "کي"), ("ڪان", "کان"), ("تہ", "ته"),
61
+ ("تھ", "ته"), ("کري", "ڪري"), ("كيو", "ڪيو"), ("کيو", "ڪيو"), ("ہے", "هي"), ("کئی", "ڪئي"),
62
+ ("كئي", "ڪئي"), ("کئي", "ڪئي"), ("ڪيس", "کيس"), ("شاھھ", "شاهه"), ("کن", "ڪن"), ("كن", "ڪن"),
63
+ ("ڪجھھ", "ڪجهه"), ("ڪجھہ", "ڪجهه"), ("ڪجهھ", "ڪجهه"), ("ڏڪڻ", "ڏکڻ"), ("کو", "ڪو"), ("کيل", "ڪيل"),
64
+ }) # fmt: skip
65
+
66
+
67
+ def is_word_char(c: str) -> bool:
68
+ """Letters and combining marks form words; ZWNJ/ZWJ are handled separately."""
69
+ return unicodedata.category(c)[0] in "LM"
70
+
71
+
72
+ def clean_zero_width(text: str) -> str:
73
+ """Drop ZWNJ/ZWJ unless it sits directly between two word characters (one kept per run)."""
74
+ out: list[str] = []
75
+ i, n = 0, len(text)
76
+ while i < n:
77
+ c = text[i]
78
+ if c not in (ZWNJ, ZWJ):
79
+ out.append(c)
80
+ i += 1
81
+ continue
82
+ j = i
83
+ while j < n and text[j] in (ZWNJ, ZWJ):
84
+ j += 1
85
+ run = text[i:j]
86
+ left = out[-1] if out else ""
87
+ right = text[j] if j < n else ""
88
+ if left and right and is_word_char(left) and is_word_char(right) and ZWNJ in run:
89
+ # a non-joiner between letters can change shaping; keep exactly one
90
+ if unicodedata.category(left) != "Mn" and unicodedata.category(right) != "Mn":
91
+ out.append(ZWNJ)
92
+ i = j
93
+ return "".join(out)
94
+
95
+
96
+ def tokens(text: str) -> Iterator[tuple[bool, str]]:
97
+ """Split into (is_word, chunk) pieces; a ZWNJ between letters stays inside the word."""
98
+ buf: list[str] = []
99
+ word = None
100
+ for i, c in enumerate(text):
101
+ inside = is_word_char(c) or (
102
+ c == ZWNJ and buf and word and i + 1 < len(text) and is_word_char(text[i + 1])
103
+ )
104
+ w = bool(inside)
105
+ if word is None or w == word:
106
+ buf.append(c)
107
+ else:
108
+ yield bool(word), "".join(buf)
109
+ buf = [c]
110
+ word = w
111
+ if buf:
112
+ yield bool(word), "".join(buf)
113
+
114
+
115
+ def build_lexicon(lines: Iterable[str]) -> Counter[str]:
116
+ """Word frequencies after zero-width cleanup."""
117
+ c: Counter[str] = Counter()
118
+ for ln in lines:
119
+ c.update(w for is_w, w in tokens(clean_zero_width(unicodedata.normalize("NFC", ln))) if is_w)
120
+ return c
121
+
122
+
123
+ @dataclass
124
+ class Normalizer:
125
+ """Corpus-guided word normalizer (see module docstring)."""
126
+
127
+ lexicon: Counter[str]
128
+ lex_ratio: float = 50.0
129
+ lex_max_count: int = 100
130
+ keep: frozenset[tuple[str, str]] = NATIVE_KEEP
131
+ _cache: dict[str, str] = field(default_factory=dict, repr=False)
132
+
133
+ def candidates(self, word: str) -> Iterator[str]:
134
+ """All spellings reachable by swapping look-alike letters (capped)."""
135
+ slots = [MEMBERS[CLASSES[c]] if c in CLASSES else (c,) for c in word]
136
+ if sum(len(s) > 1 for s in slots) > 8:
137
+ yield word
138
+ return
139
+ for k, combo in enumerate(itertools.product(*slots)):
140
+ if k >= MAX_CANDIDATES:
141
+ return
142
+ yield "".join(combo)
143
+
144
+ def allowed(self, word: str, cand: str) -> bool:
145
+ """Whether ``word`` may be rewritten as ``cand``."""
146
+ if cand == word or (word, cand) in self.keep:
147
+ return False
148
+ f_w, f_c = self.lexicon.get(word, 0), self.lexicon.get(cand, 0)
149
+ if f_c <= f_w:
150
+ return False
151
+ lexical = any(a != b and (a in LEXICAL or b in LEXICAL) for a, b in zip(word, cand, strict=True))
152
+ if lexical:
153
+ return f_w <= self.lex_max_count and f_c >= self.lex_ratio * max(f_w, 1)
154
+ return True
155
+
156
+ def decide(self, word: str) -> tuple[str, str]:
157
+ """Canonical spelling of one word and the rule that produced it.
158
+
159
+ Rules: ``no-variant-letters``, ``already-canonical`` (no more frequent variant),
160
+ ``encoding-merge``, ``lexical-merge``, ``native-keep`` (blocked by a native judgement),
161
+ ``lexical-guard`` (a more frequent variant exists but may be a different word),
162
+ ``unknown-fold`` and ``unknown-kept`` (word not in the lexicon).
163
+ """
164
+ if not any(c in CLASSES for c in word):
165
+ return word, "no-variant-letters"
166
+ ranked = sorted(self.candidates(word), key=lambda x: -self.lexicon.get(x, 0))
167
+ best = next((c for c in ranked if self.allowed(word, c)), word)
168
+ if best != word:
169
+ lexical = any(a != b and (a in LEXICAL or b in LEXICAL) for a, b in zip(word, best, strict=True))
170
+ return best, "lexical-merge" if lexical else "encoding-merge"
171
+ if self.lexicon.get(word, 0) == 0:
172
+ folded = "".join(SAFE_OOV.get(c, c) for c in word)
173
+ if folded != word and (word, folded) not in self.keep:
174
+ return folded, "unknown-fold"
175
+ top = next(
176
+ (c for c in ranked if c != word and self.lexicon.get(c, 0) > self.lexicon.get(word, 0)), None
177
+ )
178
+ if top is not None:
179
+ return word, "native-keep" if (word, top) in self.keep else "lexical-guard"
180
+ return word, "unknown-kept" if self.lexicon.get(word, 0) == 0 else "already-canonical"
181
+
182
+ def normalize_word(self, word: str) -> str:
183
+ """Canonical spelling of one word."""
184
+ if word not in self._cache:
185
+ self._cache[word] = self.decide(word)[0]
186
+ return self._cache[word]
187
+
188
+ def explain(self, text: str) -> list[dict]:
189
+ """Per-word decisions for ``text``: input, output, rule and lexicon counts."""
190
+ text = clean_zero_width(unicodedata.normalize("NFC", text))
191
+ out = []
192
+ for is_w, w in tokens(text):
193
+ if is_w:
194
+ o, rule = self.decide(w)
195
+ out.append({"word": w, "output": o, "rule": rule,
196
+ "count_in": self.lexicon.get(w, 0), "count_out": self.lexicon.get(o, 0)}) # fmt: skip
197
+ return out
198
+
199
+ def normalize(self, text: str) -> str:
200
+ """NFC, zero-width cleanup, then per-word normalization."""
201
+ text = clean_zero_width(unicodedata.normalize("NFC", text))
202
+ return "".join(self.normalize_word(w) if is_w else w for is_w, w in tokens(text))
203
+
204
+ def save(self, path: Path, min_count: int = 2) -> None:
205
+ """Write the lexicon (counts ≥ ``min_count``) as JSON."""
206
+ data = {w: n for w, n in self.lexicon.items() if n >= min_count}
207
+ path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
208
+
209
+ @classmethod
210
+ def load(cls, path: Path, **kw: float) -> Normalizer:
211
+ """Load a lexicon written by :meth:`save` (plain or ``.gz``)."""
212
+ raw = path.read_bytes()
213
+ if raw[:2] == b"\x1f\x8b":
214
+ raw = gzip.decompress(raw)
215
+ return cls(Counter(json.loads(raw.decode("utf-8"))), **kw) # type: ignore[arg-type]
216
+
217
+ @classmethod
218
+ def default(cls, **kw: float) -> Normalizer:
219
+ """Normalizer with the bundled lexicon (Sindhi Wikipedia 2023-11-01, words seen ≥ 2 times)."""
220
+ raw = resources.files("sindhinorm").joinpath("data/sd_lexicon.json.gz").read_bytes()
221
+ return cls(Counter(json.loads(gzip.decompress(raw).decode("utf-8"))), **kw) # type: ignore[arg-type]
File without changes
@@ -0,0 +1,102 @@
1
+ Metadata-Version: 2.4
2
+ Name: sindhinorm
3
+ Version: 0.1.0
4
+ Summary: Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)
5
+ Author: Arsalan Kaleem
6
+ License-Expression: MIT AND CC-BY-SA-4.0
7
+ Project-URL: Homepage, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm
8
+ Project-URL: Evaluation, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md
9
+ Keywords: sindhi,nlp,normalization,unicode,arabic-script,spelling
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Science/Research
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Topic :: Text Processing :: Linguistic
14
+ Requires-Python: >=3.10
15
+ Description-Content-Type: text/markdown
16
+ License-File: LICENSE
17
+ License-File: LICENSE-DATA
18
+ Dynamic: license-file
19
+
20
+ # sindhinorm
21
+
22
+ Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
23
+
24
+ Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
25
+ Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
26
+ **ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
27
+ same but are different characters to software, so search, counting and NLP models treat one
28
+ word as several.
29
+
30
+ A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
31
+ letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
32
+ using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
33
+ native Sindhi speaker judged to be different words.
34
+
35
+ ```bash
36
+ pip install sindhinorm
37
+ ```
38
+
39
+ ```python
40
+ from sindhinorm import normalize, explain
41
+
42
+ normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
43
+ normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
44
+
45
+ explain("ھن آهی کري")
46
+ # [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
47
+ # {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
48
+ # {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
49
+ ```
50
+
51
+ Command line:
52
+
53
+ ```bash
54
+ sindhinorm < input.txt > output.txt
55
+ sindhinorm input.txt -o output.txt
56
+ sindhinorm explain "ھن آهی کري"
57
+ sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
58
+ sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
59
+ ```
60
+
61
+ ## Rules
62
+
63
+ 1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
64
+ two letters (native review found those mark compound and ال-name joins).
65
+ 2. Candidate spellings: every combination of look-alike letters at the word's yeh
66
+ {ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
67
+ the rule allows the merge.
68
+ 3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
69
+ frequent.
70
+ 4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
71
+ rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
72
+ 5. 24 pairs judged different or unsure by a native reviewer are never merged.
73
+ 6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
74
+
75
+ `explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
76
+ `native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
77
+ `already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
78
+
79
+ ## How well it works
80
+
81
+ Measured with the bundled lexicon (details and scripts:
82
+ [MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
83
+
84
+ | Test | Result |
85
+ |---|---|
86
+ | Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
87
+ | Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
88
+ | Clean words changed | 1.35% |
89
+
90
+ **Limits.**
91
+ * Frequency cannot always tell a rare real word from a misspelling. The native keep-list
92
+ covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
93
+ with precision 0.87.
94
+ * The canonical spelling follows the Wikipedia majority, not a published standard. For
95
+ example ھن → هن; some writers prefer ھ.
96
+ * One native reviewer. Sindhi only (no Urdu mode).
97
+
98
+ ## Licences
99
+
100
+ * Code: MIT (`LICENSE`).
101
+ * Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
102
+ (2023-11-01): 61,718 words seen at least twice, with counts.
@@ -0,0 +1,14 @@
1
+ LICENSE
2
+ LICENSE-DATA
3
+ README.md
4
+ pyproject.toml
5
+ src/sindhinorm/__init__.py
6
+ src/sindhinorm/cli.py
7
+ src/sindhinorm/core.py
8
+ src/sindhinorm/py.typed
9
+ src/sindhinorm.egg-info/PKG-INFO
10
+ src/sindhinorm.egg-info/SOURCES.txt
11
+ src/sindhinorm.egg-info/dependency_links.txt
12
+ src/sindhinorm.egg-info/entry_points.txt
13
+ src/sindhinorm.egg-info/top_level.txt
14
+ src/sindhinorm/data/sd_lexicon.json.gz
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ sindhinorm = sindhinorm.cli:main
@@ -0,0 +1 @@
1
+ sindhinorm