sindhinorm 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- sindhinorm-0.1.0/LICENSE +21 -0
- sindhinorm-0.1.0/LICENSE-DATA +13 -0
- sindhinorm-0.1.0/PKG-INFO +102 -0
- sindhinorm-0.1.0/README.md +83 -0
- sindhinorm-0.1.0/pyproject.toml +34 -0
- sindhinorm-0.1.0/setup.cfg +4 -0
- sindhinorm-0.1.0/src/sindhinorm/__init__.py +44 -0
- sindhinorm-0.1.0/src/sindhinorm/cli.py +83 -0
- sindhinorm-0.1.0/src/sindhinorm/core.py +221 -0
- sindhinorm-0.1.0/src/sindhinorm/data/sd_lexicon.json.gz +0 -0
- sindhinorm-0.1.0/src/sindhinorm/py.typed +0 -0
- sindhinorm-0.1.0/src/sindhinorm.egg-info/PKG-INFO +102 -0
- sindhinorm-0.1.0/src/sindhinorm.egg-info/SOURCES.txt +14 -0
- sindhinorm-0.1.0/src/sindhinorm.egg-info/dependency_links.txt +1 -0
- sindhinorm-0.1.0/src/sindhinorm.egg-info/entry_points.txt +2 -0
- sindhinorm-0.1.0/src/sindhinorm.egg-info/top_level.txt +1 -0
sindhinorm-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Arsalan Kaleem
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
The bundled word-frequency list src/sindhinorm/data/sd_lexicon.json.gz is derived from
|
|
2
|
+
Sindhi Wikipedia (dump of 2023-11-01, distributed as the Hugging Face dataset
|
|
3
|
+
wikimedia/wikipedia, config 20231101.sd). Wikipedia text is licensed under the Creative
|
|
4
|
+
Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0),
|
|
5
|
+
https://creativecommons.org/licenses/by-sa/4.0/ . Attribution: Wikipedia contributors,
|
|
6
|
+
https://sd.wikipedia.org/ .
|
|
7
|
+
|
|
8
|
+
The list contains every word (after NFC and zero-width cleanup) that occurs at least twice
|
|
9
|
+
in 66,141 article lines, with its count. It is distributed under CC BY-SA 4.0. If you
|
|
10
|
+
redistribute the list or a modified version, keep this notice and use the same licence.
|
|
11
|
+
|
|
12
|
+
The native-speaker judgements encoded in NATIVE_KEEP (src/sindhinorm/core.py) are part of
|
|
13
|
+
the MIT-licensed code.
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sindhinorm
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)
|
|
5
|
+
Author: Arsalan Kaleem
|
|
6
|
+
License-Expression: MIT AND CC-BY-SA-4.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm
|
|
8
|
+
Project-URL: Evaluation, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md
|
|
9
|
+
Keywords: sindhi,nlp,normalization,unicode,arabic-script,spelling
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
14
|
+
Requires-Python: >=3.10
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
License-File: LICENSE
|
|
17
|
+
License-File: LICENSE-DATA
|
|
18
|
+
Dynamic: license-file
|
|
19
|
+
|
|
20
|
+
# sindhinorm
|
|
21
|
+
|
|
22
|
+
Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
|
|
23
|
+
|
|
24
|
+
Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
|
|
25
|
+
Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
|
|
26
|
+
**ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
|
|
27
|
+
same but are different characters to software, so search, counting and NLP models treat one
|
|
28
|
+
word as several.
|
|
29
|
+
|
|
30
|
+
A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
|
|
31
|
+
letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
|
|
32
|
+
using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
|
|
33
|
+
native Sindhi speaker judged to be different words.
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pip install sindhinorm
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
```python
|
|
40
|
+
from sindhinorm import normalize, explain
|
|
41
|
+
|
|
42
|
+
normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
|
|
43
|
+
normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
|
|
44
|
+
|
|
45
|
+
explain("ھن آهی کري")
|
|
46
|
+
# [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
|
|
47
|
+
# {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
|
|
48
|
+
# {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Command line:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
sindhinorm < input.txt > output.txt
|
|
55
|
+
sindhinorm input.txt -o output.txt
|
|
56
|
+
sindhinorm explain "ھن آهی کري"
|
|
57
|
+
sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
|
|
58
|
+
sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Rules
|
|
62
|
+
|
|
63
|
+
1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
|
|
64
|
+
two letters (native review found those mark compound and ال-name joins).
|
|
65
|
+
2. Candidate spellings: every combination of look-alike letters at the word's yeh
|
|
66
|
+
{ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
|
|
67
|
+
the rule allows the merge.
|
|
68
|
+
3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
|
|
69
|
+
frequent.
|
|
70
|
+
4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
|
|
71
|
+
rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
|
|
72
|
+
5. 24 pairs judged different or unsure by a native reviewer are never merged.
|
|
73
|
+
6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
|
|
74
|
+
|
|
75
|
+
`explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
|
|
76
|
+
`native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
|
|
77
|
+
`already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
|
|
78
|
+
|
|
79
|
+
## How well it works
|
|
80
|
+
|
|
81
|
+
Measured with the bundled lexicon (details and scripts:
|
|
82
|
+
[MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
|
|
83
|
+
|
|
84
|
+
| Test | Result |
|
|
85
|
+
|---|---|
|
|
86
|
+
| Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
|
|
87
|
+
| Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
|
|
88
|
+
| Clean words changed | 1.35% |
|
|
89
|
+
|
|
90
|
+
**Limits.**
|
|
91
|
+
* Frequency cannot always tell a rare real word from a misspelling. The native keep-list
|
|
92
|
+
covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
|
|
93
|
+
with precision 0.87.
|
|
94
|
+
* The canonical spelling follows the Wikipedia majority, not a published standard. For
|
|
95
|
+
example ھن → هن; some writers prefer ھ.
|
|
96
|
+
* One native reviewer. Sindhi only (no Urdu mode).
|
|
97
|
+
|
|
98
|
+
## Licences
|
|
99
|
+
|
|
100
|
+
* Code: MIT (`LICENSE`).
|
|
101
|
+
* Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
|
|
102
|
+
(2023-11-01): 61,718 words seen at least twice, with counts.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# sindhinorm
|
|
2
|
+
|
|
3
|
+
Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
|
|
4
|
+
|
|
5
|
+
Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
|
|
6
|
+
Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
|
|
7
|
+
**ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
|
|
8
|
+
same but are different characters to software, so search, counting and NLP models treat one
|
|
9
|
+
word as several.
|
|
10
|
+
|
|
11
|
+
A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
|
|
12
|
+
letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
|
|
13
|
+
using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
|
|
14
|
+
native Sindhi speaker judged to be different words.
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
pip install sindhinorm
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
```python
|
|
21
|
+
from sindhinorm import normalize, explain
|
|
22
|
+
|
|
23
|
+
normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
|
|
24
|
+
normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
|
|
25
|
+
|
|
26
|
+
explain("ھن آهی کري")
|
|
27
|
+
# [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
|
|
28
|
+
# {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
|
|
29
|
+
# {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Command line:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
sindhinorm < input.txt > output.txt
|
|
36
|
+
sindhinorm input.txt -o output.txt
|
|
37
|
+
sindhinorm explain "ھن آهی کري"
|
|
38
|
+
sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
|
|
39
|
+
sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Rules
|
|
43
|
+
|
|
44
|
+
1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
|
|
45
|
+
two letters (native review found those mark compound and ال-name joins).
|
|
46
|
+
2. Candidate spellings: every combination of look-alike letters at the word's yeh
|
|
47
|
+
{ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
|
|
48
|
+
the rule allows the merge.
|
|
49
|
+
3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
|
|
50
|
+
frequent.
|
|
51
|
+
4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
|
|
52
|
+
rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
|
|
53
|
+
5. 24 pairs judged different or unsure by a native reviewer are never merged.
|
|
54
|
+
6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
|
|
55
|
+
|
|
56
|
+
`explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
|
|
57
|
+
`native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
|
|
58
|
+
`already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
|
|
59
|
+
|
|
60
|
+
## How well it works
|
|
61
|
+
|
|
62
|
+
Measured with the bundled lexicon (details and scripts:
|
|
63
|
+
[MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
|
|
64
|
+
|
|
65
|
+
| Test | Result |
|
|
66
|
+
|---|---|
|
|
67
|
+
| Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
|
|
68
|
+
| Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
|
|
69
|
+
| Clean words changed | 1.35% |
|
|
70
|
+
|
|
71
|
+
**Limits.**
|
|
72
|
+
* Frequency cannot always tell a rare real word from a misspelling. The native keep-list
|
|
73
|
+
covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
|
|
74
|
+
with precision 0.87.
|
|
75
|
+
* The canonical spelling follows the Wikipedia majority, not a published standard. For
|
|
76
|
+
example ھن → هن; some writers prefer ھ.
|
|
77
|
+
* One native reviewer. Sindhi only (no Urdu mode).
|
|
78
|
+
|
|
79
|
+
## Licences
|
|
80
|
+
|
|
81
|
+
* Code: MIT (`LICENSE`).
|
|
82
|
+
* Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
|
|
83
|
+
(2023-11-01): 61,718 words seen at least twice, with counts.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=77"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "sindhinorm"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.10"
|
|
11
|
+
license = "MIT AND CC-BY-SA-4.0"
|
|
12
|
+
license-files = ["LICENSE", "LICENSE-DATA"]
|
|
13
|
+
authors = [{ name = "Arsalan Kaleem" }]
|
|
14
|
+
keywords = ["sindhi", "nlp", "normalization", "unicode", "arabic-script", "spelling"]
|
|
15
|
+
classifiers = [
|
|
16
|
+
"Development Status :: 4 - Beta",
|
|
17
|
+
"Intended Audience :: Science/Research",
|
|
18
|
+
"Programming Language :: Python :: 3",
|
|
19
|
+
"Topic :: Text Processing :: Linguistic",
|
|
20
|
+
]
|
|
21
|
+
dependencies = []
|
|
22
|
+
|
|
23
|
+
[project.urls]
|
|
24
|
+
Homepage = "https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm"
|
|
25
|
+
Evaluation = "https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md"
|
|
26
|
+
|
|
27
|
+
[project.scripts]
|
|
28
|
+
sindhinorm = "sindhinorm.cli:main"
|
|
29
|
+
|
|
30
|
+
[tool.setuptools.packages.find]
|
|
31
|
+
where = ["src"]
|
|
32
|
+
|
|
33
|
+
[tool.setuptools.package-data]
|
|
34
|
+
sindhinorm = ["data/*.json.gz", "py.typed"]
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
"""sindhinorm: corpus-guided, native-validated spelling normalization for Sindhi.
|
|
2
|
+
|
|
3
|
+
>>> from sindhinorm import normalize
|
|
4
|
+
>>> normalize("هي ڪتاب آهی") # Farsi yeh in the last word
|
|
5
|
+
'هي ڪتاب آهي'
|
|
6
|
+
"""
|
|
7
|
+
|
|
8
|
+
from __future__ import annotations
|
|
9
|
+
|
|
10
|
+
from functools import lru_cache
|
|
11
|
+
|
|
12
|
+
from .core import (
|
|
13
|
+
CLASSES,
|
|
14
|
+
LEXICAL,
|
|
15
|
+
MEMBERS,
|
|
16
|
+
NATIVE_KEEP,
|
|
17
|
+
SAFE_OOV,
|
|
18
|
+
Normalizer,
|
|
19
|
+
build_lexicon,
|
|
20
|
+
clean_zero_width,
|
|
21
|
+
tokens,
|
|
22
|
+
)
|
|
23
|
+
|
|
24
|
+
__version__ = "0.1.0"
|
|
25
|
+
__all__ = [
|
|
26
|
+
"CLASSES", "LEXICAL", "MEMBERS", "NATIVE_KEEP", "SAFE_OOV", "Normalizer", "build_lexicon",
|
|
27
|
+
"clean_zero_width", "default_normalizer", "explain", "normalize", "tokens",
|
|
28
|
+
] # fmt: skip
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
@lru_cache(maxsize=1)
|
|
32
|
+
def default_normalizer() -> Normalizer:
|
|
33
|
+
"""The shared normalizer with the bundled lexicon (loaded once)."""
|
|
34
|
+
return Normalizer.default()
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
def normalize(text: str) -> str:
|
|
38
|
+
"""Normalize Sindhi text with the bundled lexicon."""
|
|
39
|
+
return default_normalizer().normalize(text)
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def explain(text: str) -> list[dict]:
|
|
43
|
+
"""Per-word decisions for ``text`` with the bundled lexicon."""
|
|
44
|
+
return default_normalizer().explain(text)
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
"""Command line: ``sindhinorm`` (normalize stdin or files), ``sindhinorm explain``, ``build-lexicon``.
|
|
2
|
+
|
|
3
|
+
sindhinorm < in.txt > out.txt
|
|
4
|
+
sindhinorm input.txt -o output.txt
|
|
5
|
+
sindhinorm explain "هي ڪتاب آهی"
|
|
6
|
+
sindhinorm build-lexicon corpus.txt -o my_lexicon.json.gz
|
|
7
|
+
sindhinorm -l my_lexicon.json.gz < in.txt
|
|
8
|
+
"""
|
|
9
|
+
|
|
10
|
+
from __future__ import annotations
|
|
11
|
+
|
|
12
|
+
import argparse
|
|
13
|
+
import gzip
|
|
14
|
+
import json
|
|
15
|
+
import sys
|
|
16
|
+
from pathlib import Path
|
|
17
|
+
|
|
18
|
+
from . import __version__
|
|
19
|
+
from .core import Normalizer, build_lexicon
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def _load(a: argparse.Namespace) -> Normalizer:
|
|
23
|
+
kw = {"lex_ratio": a.lex_ratio, "lex_max_count": a.lex_max_count}
|
|
24
|
+
return Normalizer.load(a.lexicon, **kw) if a.lexicon else Normalizer.default(**kw)
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def main(argv: list[str] | None = None) -> int:
|
|
28
|
+
"""Entry point."""
|
|
29
|
+
for s in (sys.stdin, sys.stdout):
|
|
30
|
+
if hasattr(s, "reconfigure"):
|
|
31
|
+
s.reconfigure(encoding="utf-8") # type: ignore[union-attr]
|
|
32
|
+
argv = list(sys.argv[1:] if argv is None else argv)
|
|
33
|
+
cmd = argv.pop(0) if argv and argv[0] in ("explain", "build-lexicon") else "normalize"
|
|
34
|
+
ap = argparse.ArgumentParser(prog=f"sindhinorm {cmd}".replace(" normalize", ""), description=__doc__,
|
|
35
|
+
formatter_class=argparse.RawDescriptionHelpFormatter) # fmt: skip
|
|
36
|
+
ap.add_argument("--version", action="version", version=f"sindhinorm {__version__}")
|
|
37
|
+
if cmd == "build-lexicon":
|
|
38
|
+
ap.add_argument("corpus", type=Path)
|
|
39
|
+
ap.add_argument("-o", "--out", type=Path, required=True)
|
|
40
|
+
ap.add_argument("--min-count", type=int, default=2)
|
|
41
|
+
a = ap.parse_args(argv)
|
|
42
|
+
with a.corpus.open(encoding="utf-8") as f:
|
|
43
|
+
lex = build_lexicon(f)
|
|
44
|
+
data = json.dumps({w: n for w, n in lex.most_common() if n >= a.min_count}, ensure_ascii=False)
|
|
45
|
+
raw = data.encode("utf-8")
|
|
46
|
+
a.out.write_bytes(gzip.compress(raw) if a.out.suffix == ".gz" else raw)
|
|
47
|
+
print(f"wrote {a.out} ({sum(n >= a.min_count for n in lex.values())} words)", file=sys.stderr)
|
|
48
|
+
return 0
|
|
49
|
+
ap.add_argument("-l", "--lexicon", type=Path, default=None, help="custom lexicon (default: bundled)")
|
|
50
|
+
ap.add_argument("--lex-ratio", type=float, default=50.0)
|
|
51
|
+
ap.add_argument("--lex-max-count", type=int, default=100)
|
|
52
|
+
if cmd == "explain":
|
|
53
|
+
ap.add_argument("text", nargs="+")
|
|
54
|
+
ap.add_argument("--json", action="store_true")
|
|
55
|
+
a = ap.parse_args(argv)
|
|
56
|
+
rows = _load(a).explain(" ".join(a.text))
|
|
57
|
+
if a.json:
|
|
58
|
+
print(json.dumps(rows, ensure_ascii=False, indent=1))
|
|
59
|
+
else:
|
|
60
|
+
for r in rows:
|
|
61
|
+
arrow = "->" if r["output"] != r["word"] else "=="
|
|
62
|
+
print(
|
|
63
|
+
f"{r['word']} ({r['count_in']}) {arrow} {r['output']} ({r['count_out']}) [{r['rule']}]"
|
|
64
|
+
)
|
|
65
|
+
return 0
|
|
66
|
+
ap.add_argument("inputs", nargs="*", type=Path, help="files (default: stdin)")
|
|
67
|
+
ap.add_argument("-o", "--out", type=Path, default=None)
|
|
68
|
+
a = ap.parse_args(argv)
|
|
69
|
+
norm = _load(a)
|
|
70
|
+
out = a.out.open("w", encoding="utf-8", newline="") if a.out else sys.stdout
|
|
71
|
+
try:
|
|
72
|
+
for src in a.inputs or [None]:
|
|
73
|
+
f = src.open(encoding="utf-8", newline="") if src else sys.stdin
|
|
74
|
+
for line in f:
|
|
75
|
+
out.write(norm.normalize(line))
|
|
76
|
+
finally:
|
|
77
|
+
if a.out:
|
|
78
|
+
out.close()
|
|
79
|
+
return 0
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
if __name__ == "__main__":
|
|
83
|
+
raise SystemExit(main())
|
|
@@ -0,0 +1,221 @@
|
|
|
1
|
+
"""sindhinorm: a corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script).
|
|
2
|
+
|
|
3
|
+
Sindhi text typed on Urdu, Persian or Arabic keyboards mixes look-alike code points
|
|
4
|
+
(Farsi yeh ی for Sindhi ي, Urdu heh-goal ہ for ه, keheh ک or Arabic kaf ك for Sindhi ڪ) and
|
|
5
|
+
stray zero-width characters. Some of these are pure encoding variants; others are *different
|
|
6
|
+
Sindhi letters* (ک and ڪ are distinct letters: کي "to" vs ڪي "some"). A fixed character map
|
|
7
|
+
therefore either misses variants or corrupts real words.
|
|
8
|
+
|
|
9
|
+
sdnorm decides per **word**, using a frequency lexicon built from a Sindhi corpus:
|
|
10
|
+
|
|
11
|
+
* Letters are grouped into look-alike classes: yeh {ي ی ى ے}, heh {ه ھ ہ}, kaf {ڪ ک ك}.
|
|
12
|
+
* A word's candidate spellings are all combinations of class members at its class positions;
|
|
13
|
+
the canonical spelling is the candidate most frequent in the lexicon.
|
|
14
|
+
* **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
|
|
15
|
+
frequent. A native reviewer judged these SAME in nearly all reviewed pairs.
|
|
16
|
+
* **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only when the
|
|
17
|
+
rare form is at most ``lex_max_count`` times attested **and** the candidate is at least
|
|
18
|
+
``lex_ratio`` times more frequent, so that two real words (کي / ڪي) are never merged.
|
|
19
|
+
* Pairs a native reviewer judged DIFFERENT or UNSURE (``NATIVE_KEEP``) are never merged.
|
|
20
|
+
* Words absent from the lexicon get only the safest encoding folds (ی ى → ي, ہ → ه).
|
|
21
|
+
* Zero-width non-joiners/joiners that cannot affect shaping (next to a space, punctuation,
|
|
22
|
+
a non-letter, a string edge, or another zero-width character) are removed; a ZWNJ between
|
|
23
|
+
two letters is kept, because native review found those mark compound and ال-name joins.
|
|
24
|
+
|
|
25
|
+
Thresholds were fixed before evaluation. The evaluation (agreement with native judgements,
|
|
26
|
+
restoration of keyboard-substituted text) lives in the MorphoTok-SD-UR repository:
|
|
27
|
+
https://github.com/ArsalanKaleem/MorphoTok-SD-UR (``docs/sdnorm.md``).
|
|
28
|
+
"""
|
|
29
|
+
|
|
30
|
+
from __future__ import annotations
|
|
31
|
+
|
|
32
|
+
import gzip
|
|
33
|
+
import itertools
|
|
34
|
+
import json
|
|
35
|
+
import unicodedata
|
|
36
|
+
from collections import Counter
|
|
37
|
+
from collections.abc import Iterable, Iterator
|
|
38
|
+
from dataclasses import dataclass, field
|
|
39
|
+
from importlib import resources
|
|
40
|
+
from pathlib import Path
|
|
41
|
+
|
|
42
|
+
ZWNJ, ZWJ = "", ""
|
|
43
|
+
CLASSES: dict[str, str] = {
|
|
44
|
+
"ي": "yeh", "ی": "yeh", "ى": "yeh", "ے": "yeh",
|
|
45
|
+
"ه": "heh", "ھ": "heh", "ہ": "heh",
|
|
46
|
+
"ڪ": "kaf", "ک": "kaf", "ك": "kaf",
|
|
47
|
+
} # fmt: skip
|
|
48
|
+
MEMBERS: dict[str, tuple[str, ...]] = {
|
|
49
|
+
"yeh": ("ي", "ی", "ى", "ے"),
|
|
50
|
+
"heh": ("ه", "ھ", "ہ"),
|
|
51
|
+
"kaf": ("ڪ", "ک", "ك"),
|
|
52
|
+
}
|
|
53
|
+
LEXICAL = frozenset("ڪکكے") # letters whose substitution can change the word
|
|
54
|
+
SAFE_OOV = {"ی": "ي", "ى": "ي", "ہ": "ه"} # encoding folds applied to unknown words
|
|
55
|
+
MAX_CANDIDATES = 4096
|
|
56
|
+
|
|
57
|
+
# Pairs (variant -> would-be canonical) that a native Sindhi reviewer judged DIFFERENT or
|
|
58
|
+
# UNSURE (review of 150 Wikipedia pairs, 2026-11-09; docs/findings_human_review.md §4).
|
|
59
|
+
NATIVE_KEEP: frozenset[tuple[str, str]] = frozenset({
|
|
60
|
+
("جے", "جي"), ("ڪي", "کي"), ("کے", "کي"), ("کی", "کي"), ("ڪان", "کان"), ("تہ", "ته"),
|
|
61
|
+
("تھ", "ته"), ("کري", "ڪري"), ("كيو", "ڪيو"), ("کيو", "ڪيو"), ("ہے", "هي"), ("کئی", "ڪئي"),
|
|
62
|
+
("كئي", "ڪئي"), ("کئي", "ڪئي"), ("ڪيس", "کيس"), ("شاھھ", "شاهه"), ("کن", "ڪن"), ("كن", "ڪن"),
|
|
63
|
+
("ڪجھھ", "ڪجهه"), ("ڪجھہ", "ڪجهه"), ("ڪجهھ", "ڪجهه"), ("ڏڪڻ", "ڏکڻ"), ("کو", "ڪو"), ("کيل", "ڪيل"),
|
|
64
|
+
}) # fmt: skip
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def is_word_char(c: str) -> bool:
|
|
68
|
+
"""Letters and combining marks form words; ZWNJ/ZWJ are handled separately."""
|
|
69
|
+
return unicodedata.category(c)[0] in "LM"
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def clean_zero_width(text: str) -> str:
|
|
73
|
+
"""Drop ZWNJ/ZWJ unless it sits directly between two word characters (one kept per run)."""
|
|
74
|
+
out: list[str] = []
|
|
75
|
+
i, n = 0, len(text)
|
|
76
|
+
while i < n:
|
|
77
|
+
c = text[i]
|
|
78
|
+
if c not in (ZWNJ, ZWJ):
|
|
79
|
+
out.append(c)
|
|
80
|
+
i += 1
|
|
81
|
+
continue
|
|
82
|
+
j = i
|
|
83
|
+
while j < n and text[j] in (ZWNJ, ZWJ):
|
|
84
|
+
j += 1
|
|
85
|
+
run = text[i:j]
|
|
86
|
+
left = out[-1] if out else ""
|
|
87
|
+
right = text[j] if j < n else ""
|
|
88
|
+
if left and right and is_word_char(left) and is_word_char(right) and ZWNJ in run:
|
|
89
|
+
# a non-joiner between letters can change shaping; keep exactly one
|
|
90
|
+
if unicodedata.category(left) != "Mn" and unicodedata.category(right) != "Mn":
|
|
91
|
+
out.append(ZWNJ)
|
|
92
|
+
i = j
|
|
93
|
+
return "".join(out)
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def tokens(text: str) -> Iterator[tuple[bool, str]]:
|
|
97
|
+
"""Split into (is_word, chunk) pieces; a ZWNJ between letters stays inside the word."""
|
|
98
|
+
buf: list[str] = []
|
|
99
|
+
word = None
|
|
100
|
+
for i, c in enumerate(text):
|
|
101
|
+
inside = is_word_char(c) or (
|
|
102
|
+
c == ZWNJ and buf and word and i + 1 < len(text) and is_word_char(text[i + 1])
|
|
103
|
+
)
|
|
104
|
+
w = bool(inside)
|
|
105
|
+
if word is None or w == word:
|
|
106
|
+
buf.append(c)
|
|
107
|
+
else:
|
|
108
|
+
yield bool(word), "".join(buf)
|
|
109
|
+
buf = [c]
|
|
110
|
+
word = w
|
|
111
|
+
if buf:
|
|
112
|
+
yield bool(word), "".join(buf)
|
|
113
|
+
|
|
114
|
+
|
|
115
|
+
def build_lexicon(lines: Iterable[str]) -> Counter[str]:
|
|
116
|
+
"""Word frequencies after zero-width cleanup."""
|
|
117
|
+
c: Counter[str] = Counter()
|
|
118
|
+
for ln in lines:
|
|
119
|
+
c.update(w for is_w, w in tokens(clean_zero_width(unicodedata.normalize("NFC", ln))) if is_w)
|
|
120
|
+
return c
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
@dataclass
|
|
124
|
+
class Normalizer:
|
|
125
|
+
"""Corpus-guided word normalizer (see module docstring)."""
|
|
126
|
+
|
|
127
|
+
lexicon: Counter[str]
|
|
128
|
+
lex_ratio: float = 50.0
|
|
129
|
+
lex_max_count: int = 100
|
|
130
|
+
keep: frozenset[tuple[str, str]] = NATIVE_KEEP
|
|
131
|
+
_cache: dict[str, str] = field(default_factory=dict, repr=False)
|
|
132
|
+
|
|
133
|
+
def candidates(self, word: str) -> Iterator[str]:
|
|
134
|
+
"""All spellings reachable by swapping look-alike letters (capped)."""
|
|
135
|
+
slots = [MEMBERS[CLASSES[c]] if c in CLASSES else (c,) for c in word]
|
|
136
|
+
if sum(len(s) > 1 for s in slots) > 8:
|
|
137
|
+
yield word
|
|
138
|
+
return
|
|
139
|
+
for k, combo in enumerate(itertools.product(*slots)):
|
|
140
|
+
if k >= MAX_CANDIDATES:
|
|
141
|
+
return
|
|
142
|
+
yield "".join(combo)
|
|
143
|
+
|
|
144
|
+
def allowed(self, word: str, cand: str) -> bool:
|
|
145
|
+
"""Whether ``word`` may be rewritten as ``cand``."""
|
|
146
|
+
if cand == word or (word, cand) in self.keep:
|
|
147
|
+
return False
|
|
148
|
+
f_w, f_c = self.lexicon.get(word, 0), self.lexicon.get(cand, 0)
|
|
149
|
+
if f_c <= f_w:
|
|
150
|
+
return False
|
|
151
|
+
lexical = any(a != b and (a in LEXICAL or b in LEXICAL) for a, b in zip(word, cand, strict=True))
|
|
152
|
+
if lexical:
|
|
153
|
+
return f_w <= self.lex_max_count and f_c >= self.lex_ratio * max(f_w, 1)
|
|
154
|
+
return True
|
|
155
|
+
|
|
156
|
+
def decide(self, word: str) -> tuple[str, str]:
|
|
157
|
+
"""Canonical spelling of one word and the rule that produced it.
|
|
158
|
+
|
|
159
|
+
Rules: ``no-variant-letters``, ``already-canonical`` (no more frequent variant),
|
|
160
|
+
``encoding-merge``, ``lexical-merge``, ``native-keep`` (blocked by a native judgement),
|
|
161
|
+
``lexical-guard`` (a more frequent variant exists but may be a different word),
|
|
162
|
+
``unknown-fold`` and ``unknown-kept`` (word not in the lexicon).
|
|
163
|
+
"""
|
|
164
|
+
if not any(c in CLASSES for c in word):
|
|
165
|
+
return word, "no-variant-letters"
|
|
166
|
+
ranked = sorted(self.candidates(word), key=lambda x: -self.lexicon.get(x, 0))
|
|
167
|
+
best = next((c for c in ranked if self.allowed(word, c)), word)
|
|
168
|
+
if best != word:
|
|
169
|
+
lexical = any(a != b and (a in LEXICAL or b in LEXICAL) for a, b in zip(word, best, strict=True))
|
|
170
|
+
return best, "lexical-merge" if lexical else "encoding-merge"
|
|
171
|
+
if self.lexicon.get(word, 0) == 0:
|
|
172
|
+
folded = "".join(SAFE_OOV.get(c, c) for c in word)
|
|
173
|
+
if folded != word and (word, folded) not in self.keep:
|
|
174
|
+
return folded, "unknown-fold"
|
|
175
|
+
top = next(
|
|
176
|
+
(c for c in ranked if c != word and self.lexicon.get(c, 0) > self.lexicon.get(word, 0)), None
|
|
177
|
+
)
|
|
178
|
+
if top is not None:
|
|
179
|
+
return word, "native-keep" if (word, top) in self.keep else "lexical-guard"
|
|
180
|
+
return word, "unknown-kept" if self.lexicon.get(word, 0) == 0 else "already-canonical"
|
|
181
|
+
|
|
182
|
+
def normalize_word(self, word: str) -> str:
|
|
183
|
+
"""Canonical spelling of one word."""
|
|
184
|
+
if word not in self._cache:
|
|
185
|
+
self._cache[word] = self.decide(word)[0]
|
|
186
|
+
return self._cache[word]
|
|
187
|
+
|
|
188
|
+
def explain(self, text: str) -> list[dict]:
|
|
189
|
+
"""Per-word decisions for ``text``: input, output, rule and lexicon counts."""
|
|
190
|
+
text = clean_zero_width(unicodedata.normalize("NFC", text))
|
|
191
|
+
out = []
|
|
192
|
+
for is_w, w in tokens(text):
|
|
193
|
+
if is_w:
|
|
194
|
+
o, rule = self.decide(w)
|
|
195
|
+
out.append({"word": w, "output": o, "rule": rule,
|
|
196
|
+
"count_in": self.lexicon.get(w, 0), "count_out": self.lexicon.get(o, 0)}) # fmt: skip
|
|
197
|
+
return out
|
|
198
|
+
|
|
199
|
+
def normalize(self, text: str) -> str:
|
|
200
|
+
"""NFC, zero-width cleanup, then per-word normalization."""
|
|
201
|
+
text = clean_zero_width(unicodedata.normalize("NFC", text))
|
|
202
|
+
return "".join(self.normalize_word(w) if is_w else w for is_w, w in tokens(text))
|
|
203
|
+
|
|
204
|
+
def save(self, path: Path, min_count: int = 2) -> None:
|
|
205
|
+
"""Write the lexicon (counts ≥ ``min_count``) as JSON."""
|
|
206
|
+
data = {w: n for w, n in self.lexicon.items() if n >= min_count}
|
|
207
|
+
path.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")
|
|
208
|
+
|
|
209
|
+
@classmethod
|
|
210
|
+
def load(cls, path: Path, **kw: float) -> Normalizer:
|
|
211
|
+
"""Load a lexicon written by :meth:`save` (plain or ``.gz``)."""
|
|
212
|
+
raw = path.read_bytes()
|
|
213
|
+
if raw[:2] == b"\x1f\x8b":
|
|
214
|
+
raw = gzip.decompress(raw)
|
|
215
|
+
return cls(Counter(json.loads(raw.decode("utf-8"))), **kw) # type: ignore[arg-type]
|
|
216
|
+
|
|
217
|
+
@classmethod
|
|
218
|
+
def default(cls, **kw: float) -> Normalizer:
|
|
219
|
+
"""Normalizer with the bundled lexicon (Sindhi Wikipedia 2023-11-01, words seen ≥ 2 times)."""
|
|
220
|
+
raw = resources.files("sindhinorm").joinpath("data/sd_lexicon.json.gz").read_bytes()
|
|
221
|
+
return cls(Counter(json.loads(gzip.decompress(raw).decode("utf-8"))), **kw) # type: ignore[arg-type]
|
|
Binary file
|
|
File without changes
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sindhinorm
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Corpus-guided, native-validated spelling normalizer for Sindhi (Arabic script)
|
|
5
|
+
Author: Arsalan Kaleem
|
|
6
|
+
License-Expression: MIT AND CC-BY-SA-4.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/tree/main/packages/sindhinorm
|
|
8
|
+
Project-URL: Evaluation, https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md
|
|
9
|
+
Keywords: sindhi,nlp,normalization,unicode,arabic-script,spelling
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
14
|
+
Requires-Python: >=3.10
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
License-File: LICENSE
|
|
17
|
+
License-File: LICENSE-DATA
|
|
18
|
+
Dynamic: license-file
|
|
19
|
+
|
|
20
|
+
# sindhinorm
|
|
21
|
+
|
|
22
|
+
Spelling normalization for Sindhi text typed on mixed keyboards. No dependencies, Python ≥ 3.10.
|
|
23
|
+
|
|
24
|
+
Sindhi text often mixes look-alike Unicode letters from Urdu, Persian and Arabic keyboards:
|
|
25
|
+
Farsi yeh **ی** or alef maksura **ى** for Sindhi **ي**, Urdu heh-goal **ہ** or do-chashmi **ھ** for
|
|
26
|
+
**ه**, keheh **ک** or Arabic kaf **ك** for **ڪ**, plus stray zero-width characters. They look the
|
|
27
|
+
same but are different characters to software, so search, counting and NLP models treat one
|
|
28
|
+
word as several.
|
|
29
|
+
|
|
30
|
+
A fixed character map does not work for Sindhi, because **ک and ڪ are both real Sindhi
|
|
31
|
+
letters**: کي ("to") and ڪي ("some") are different words. sindhinorm decides **per word**,
|
|
32
|
+
using how often each spelling occurs in Sindhi Wikipedia, and never merges a pair that a
|
|
33
|
+
native Sindhi speaker judged to be different words.
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pip install sindhinorm
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
```python
|
|
40
|
+
from sindhinorm import normalize, explain
|
|
41
|
+
|
|
42
|
+
normalize("هي ڪتاب آهی") # 'هي ڪتاب آهي' (Farsi yeh fixed)
|
|
43
|
+
normalize("ڪي کي") # 'ڪي کي' (two different words: left alone)
|
|
44
|
+
|
|
45
|
+
explain("ھن آهی کري")
|
|
46
|
+
# [{'word': 'ھن', 'output': 'هن', 'rule': 'encoding-merge', 'count_in': 12528, 'count_out': 22047},
|
|
47
|
+
# {'word': 'آهی', 'output': 'آهي', 'rule': 'encoding-merge', 'count_in': 0, 'count_out': 62425},
|
|
48
|
+
# {'word': 'کري', 'output': 'کري', 'rule': 'native-keep', 'count_in': 14, 'count_out': 14}]
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Command line:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
sindhinorm < input.txt > output.txt
|
|
55
|
+
sindhinorm input.txt -o output.txt
|
|
56
|
+
sindhinorm explain "ھن آهی کري"
|
|
57
|
+
sindhinorm build-lexicon my_corpus.txt -o my_lexicon.json.gz # your own word counts
|
|
58
|
+
sindhinorm -l my_lexicon.json.gz < input.txt > output.txt
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Rules
|
|
62
|
+
|
|
63
|
+
1. Unicode NFC. Zero-width non-joiners and joiners are removed unless a ZWNJ sits between
|
|
64
|
+
two letters (native review found those mark compound and ال-name joins).
|
|
65
|
+
2. Candidate spellings: every combination of look-alike letters at the word's yeh
|
|
66
|
+
{ي ی ى ے}, heh {ه ھ ہ} and kaf {ڪ ک ك} positions. The most frequent candidate wins if
|
|
67
|
+
the rule allows the merge.
|
|
68
|
+
3. **Encoding-only** differences (ي/ی/ى, ه/ھ/ہ) are merged whenever the candidate is more
|
|
69
|
+
frequent.
|
|
70
|
+
4. **Lexical** differences (any position involving ک, ڪ, ك or ے) are merged only if the
|
|
71
|
+
rare form occurs ≤ 100 times **and** the candidate is ≥ 50× more frequent.
|
|
72
|
+
5. 24 pairs judged different or unsure by a native reviewer are never merged.
|
|
73
|
+
6. Words not in the lexicon only get the safest folds (ی ى → ي, ہ → ه).
|
|
74
|
+
|
|
75
|
+
`explain` reports which rule decided each word: `encoding-merge`, `lexical-merge`,
|
|
76
|
+
`native-keep`, `lexical-guard` (a more frequent variant exists but may be another word),
|
|
77
|
+
`already-canonical`, `no-variant-letters`, `unknown-fold` or `unknown-kept`.
|
|
78
|
+
|
|
79
|
+
## How well it works
|
|
80
|
+
|
|
81
|
+
Measured with the bundled lexicon (details and scripts:
|
|
82
|
+
[MorphoTok-SD-UR, docs/sdnorm.md](https://github.com/ArsalanKaleem/MorphoTok-SD-UR/blob/main/docs/sdnorm.md)):
|
|
83
|
+
|
|
84
|
+
| Test | Result |
|
|
85
|
+
|---|---|
|
|
86
|
+
| Text with ي replaced by ی (UD Sindhi test set, news) | **94.0%** of 4,511 altered words restored |
|
|
87
|
+
| Random-letter edits, same words (control) | 0.4% "restored": it does not undo arbitrary typos |
|
|
88
|
+
| Clean words changed | 1.35% |
|
|
89
|
+
|
|
90
|
+
**Limits.**
|
|
91
|
+
* Frequency cannot always tell a rare real word from a misspelling. The native keep-list
|
|
92
|
+
covers the 24 reviewed cases. On the 150 reviewed pairs, the frequency rule alone merged
|
|
93
|
+
with precision 0.87.
|
|
94
|
+
* The canonical spelling follows the Wikipedia majority, not a published standard. For
|
|
95
|
+
example ھن → هن; some writers prefer ھ.
|
|
96
|
+
* One native reviewer. Sindhi only (no Urdu mode).
|
|
97
|
+
|
|
98
|
+
## Licences
|
|
99
|
+
|
|
100
|
+
* Code: MIT (`LICENSE`).
|
|
101
|
+
* Bundled word list: CC BY-SA 4.0 (`LICENSE-DATA`). It is derived from Sindhi Wikipedia
|
|
102
|
+
(2023-11-01): 61,718 words seen at least twice, with counts.
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
LICENSE-DATA
|
|
3
|
+
README.md
|
|
4
|
+
pyproject.toml
|
|
5
|
+
src/sindhinorm/__init__.py
|
|
6
|
+
src/sindhinorm/cli.py
|
|
7
|
+
src/sindhinorm/core.py
|
|
8
|
+
src/sindhinorm/py.typed
|
|
9
|
+
src/sindhinorm.egg-info/PKG-INFO
|
|
10
|
+
src/sindhinorm.egg-info/SOURCES.txt
|
|
11
|
+
src/sindhinorm.egg-info/dependency_links.txt
|
|
12
|
+
src/sindhinorm.egg-info/entry_points.txt
|
|
13
|
+
src/sindhinorm.egg-info/top_level.txt
|
|
14
|
+
src/sindhinorm/data/sd_lexicon.json.gz
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
sindhinorm
|