persian-text-guard 1.4.0__tar.gz

persian_text_guard-1.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Amirehsan Kohannasab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

persian_text_guard-1.4.0/PKG-INFO

@@ -0,0 +1,454 @@
+Metadata-Version: 2.5
+Name: persian-text-guard
+Version: 1.4.0
+Summary: Persian text normalization and evasion-resistant profanity filtering for Python. Folds Arabic/Persian look-alike letters, zero-width and bidi characters, and all three digit ranges, and catches the ways people get around a word list: spaced or dotted letters, held keys, leetspeak, filler symbols, Cyrillic look-alikes and Finglish. Finds every match with its position and censors messages with a fixed mask. Ships an optional curated Persian/Finglish/English word list of about 1,250 categorised entries.
+Project-URL: Homepage, https://github.com/AmirehsanK/PersianTextGuard/tree/main/python#readme
+Project-URL: Source, https://github.com/AmirehsanK/PersianTextGuard
+Project-URL: Issues, https://github.com/AmirehsanK/PersianTextGuard/issues
+Author: Amirehsan Kohannasab
+License-Expression: MIT
+License-File: LICENSE
+License-File: THIRD-PARTY-NOTICES.md
+Keywords: arabic,farsi,finglish,moderation,normalization,persian,profanity,profanity-filter,python,rtl,text
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Natural Language :: English
+Classifier: Natural Language :: Persian
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: Free Threading :: 3 - Stable
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# persian-text-guard
+
+Persian text normalization and evasion-resistant profanity filtering for Python.
+
+[فارسی](#فارسی) · [Full project README](https://github.com/AmirehsanK/PersianTextGuard#readme) · [.NET package](https://www.nuget.org/packages/PersianTextGuard) · [npm package](https://www.npmjs.com/package/persian-text-guard)
+
+A word list is easy to get around. Type an Arabic `ي` instead of `ی`, slip a zero-width character
+inside a word, write `f u c k`, `sh1t`, `کیییییر` or `k0s`, and a plain `word in text` check sees
+nothing. `persian-text-guard` reads through those tricks, and just as importantly, it doesn't flag
+ordinary messages: `هر کس`, `تخم مرغ`, `class`, `push it` and `Scunthorpe` all pass.
+
+It is the Python port of [PersianTextGuard](https://github.com/AmirehsanK/PersianTextGuard), and gives
+the same answers as the .NET and JavaScript packages for every case in the shared
+[conformance corpus](https://github.com/AmirehsanK/PersianTextGuard/tree/main/conformance).
+
+- Pure Python, with full type hints (`py.typed`), for CPython 3.11 and later, free-threaded 3.14
+  included.
+- No dependencies.
+- Safe to share across threads.
+
+## Installation
+
+```bash
+pip install persian-text-guard
+```
+
+or, in a project managed by uv:
+
+```bash
+uv add persian-text-guard
+```
+
+## Quick start
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+# Build once and share it: a filter never changes after it is built.
+filter = ProfanityFilter(WordList.persian_default())
+
+assert not filter.contains_profanity("سلام، سفارشم کی میرسه؟")
+assert filter.contains_profanity("ک.ی.ر")
+assert filter.contains_profanity("f u c k")
+```
+
+### What matched, and why
+
+```python
+from persian_text_guard import EvasionKind, ProfanityFilter, WordCategory, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+match = filter.find_match("sh1iiit")
+
+assert match is not None
+assert match.word.text == "shit"
+assert match.word.category == WordCategory.PROFANITY
+assert match.evasion == (EvasionKind.REPEATED_LETTERS, EvasionKind.LOOKALIKE_CHARACTERS)
+assert filter.find_match("hello") is None
+```
+
+### Every match, with its position
+
+Positions are Python string indexes (code points), so slicing gives the matched text:
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+message = "sh1t and f u c k"
+
+found = [
+    (match.word.text, match.index, match.length, message[match.index : match.index + match.length])
+    for match in filter.find_matches(message)
+]
+assert found == [("shit", 0, 4, "sh1t"), ("fuck", 9, 7, "f u c k")]
+```
+
+This holds for emoji and every other character outside the Basic Multilingual Plane too: each counts
+as one position, as it does everywhere else in Python.
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+message = "\N{GRINNING FACE} کیر"
+
+match = filter.find_match(message)
+assert match is not None
+assert (match.index, match.length) == (2, 3)
+assert message[match.index : match.index + match.length] == "کیر"
+```
+
+### Censoring
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+
+assert filter.censor("kir and motherfucker") == "**** and ****"
+assert filter.censor("جنده\N{ZERO WIDTH NON-JOINER}ها رو ببین") == "**** رو ببین"
+assert filter.censor("this is kir", "#") == "this is ####"
+assert filter.censor("hello") == "hello"
+```
+
+Whole words are hidden, every mask is four characters, and the output is always clean. Clean text
+comes back unchanged, and everything outside a hidden word is returned exactly as given. The mask can
+be any symbol or punctuation; a letter, digit, whitespace or control character raises `ValueError`.
+
+### Categories
+
+Every bundled entry has a category: `profanity`, `sexual`, `insult`, `slur`, `harassment` or `mild`.
+
+```python
+from persian_text_guard import ProfanityFilter, WordCategory, WordList
+
+everyday = ProfanityFilter(WordList.persian_default())  # everything except mild
+strict = ProfanityFilter(WordList.all())  # mild too
+slurs_and_abuse = ProfanityFilter(WordList.bundled(WordCategory.SLUR, "harassment"))
+
+assert not everyday.contains_profanity("این فیلم آشغال بود")
+assert strict.contains_profanity("این فیلم آشغال بود")
+assert slurs_and_abuse.contains_profanity("kys")
+assert not slurs_and_abuse.contains_profanity("کیر")
+```
+
+A category, like a match mode or a normalization step, can be given as an enum member or as its name
+(`"slur"`); a type checker rejects a name that does not exist, and so does the code at run time.
+
+### Your own words
+
+```python
+from persian_text_guard import BannedWord, ProfanityFilter, WordList, WordMatchMode
+
+filter = ProfanityFilter(
+    [
+        BannedWord("اسپم"),  # whole word (the default)
+        BannedWord("casino", WordMatchMode.ANYWHERE),  # inside longer words too
+    ]
+)
+assert filter.contains_profanity("این پیام اسپم است")
+assert filter.contains_profanity("onlinecasino.example")
+
+# Or combine your entries with the bundled list:
+combined = ProfanityFilter([*WordList.persian_default(), BannedWord("casino", "anywhere", "mild")])
+assert combined.contains_profanity("casino night")
+```
+
+### Word-list files
+
+Word lists can also be kept in a text file, one entry per line. `WordList.load` reads a path or an open
+file as UTF-8, and `WordList.parse` reads text you already have:
+
+```python
+import tempfile
+from pathlib import Path
+
+from persian_text_guard import ProfanityFilter, WordCategory, WordList, WordMatchMode
+
+text = "# comments start with #\n[insult]\nidiot\n~scam\n"
+
+words = WordList.parse(text)
+assert [(w.text, w.mode, w.category) for w in words] == [
+    ("idiot", WordMatchMode.WHOLE_WORD, WordCategory.INSULT),
+    ("scam", WordMatchMode.ANYWHERE, WordCategory.INSULT),
+]
+assert ProfanityFilter(words).contains_profanity("scammers everywhere")
+
+with tempfile.TemporaryDirectory() as folder:
+    path = Path(folder) / "words.txt"
+    path.write_text(text, encoding="utf-8")
+    assert WordList.load(path) == words
+    with path.open("rb") as file:
+        assert WordList.load(file) == words  # an open file is read, and left open
+```
+
+A heading such as `[insult]` names a category in any letter case. An unknown heading raises a
+`WordListFormatError`, a `ValueError` with the heading's `line`. A UTF-8 byte order mark is ignored.
+
+### Options
+
+Every evasion the filter reads through can be turned off:
+
+```python
+from persian_text_guard import ProfanityFilter, ProfanityFilterOptions, WordList
+
+literal = ProfanityFilter(WordList.persian_default(), ProfanityFilterOptions(join_spaced_letters=False))
+
+assert not literal.contains_profanity("f u c k")
+assert literal.contains_profanity("fuuuck")
+```
+
+### Normalization
+
+```python
+from persian_text_guard import NormalizationStep, normalize, to_ascii_digits, to_persian_digits, tokenize
+
+text = "كتاب\N{ZERO WIDTH NON-JOINER}هاي  ۱۲ ABC"
+
+# "comparison" (the default): everything folded, for searching and matching.
+assert normalize(text) == "کتابهای 12 abc"
+# "standard": safe to store and show; keeps the zero-width non-joiner, Persian digits and case.
+assert normalize(text, "standard") == "کتاب\N{ZERO WIDTH NON-JOINER}های ۱۲ ABC"
+# Or any combination of steps, as members or names.
+assert normalize("ABC ي", [NormalizationStep.LOWER_CASE, "unifyLetters"]) == "abc ی"
+
+assert tokenize("سلام، دنیا! خوبی؟") == ["سلام", "دنیا", "خوبی"]
+assert to_persian_digits("2 ساعت پیش") == "۲ ساعت پیش"
+assert to_ascii_digits("۱۴۰۴/۰۵/۱۴") == "1404/05/14"
+```
+
+## Thread safety
+
+A filter is immutable after it is built, and the bundled lists are parsed once, under a lock. Build one
+filter and share it between threads, including on free-threaded CPython 3.14 with the GIL disabled:
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+from persian_text_guard import ProfanityFilter, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+messages = ["سلام", "ک.ی.ر", "f u c k", "hello"] * 50
+
+with ThreadPoolExecutor(max_workers=8) as pool:
+    results = list(pool.map(filter.contains_profanity, messages))
+
+assert results == [filter.contains_profanity(message) for message in messages]
+```
+
+## Validating input
+
+Checking, finding matches, censoring, normalizing and tokenizing accept any `str`, including empty
+text, broken emoji, lone surrogates and very long messages, and treat `None` as an empty message. They
+never raise for text.
+
+Anything else, such as `bytes`, a number or a list, raises `TypeError`, so a missing check shows up at
+once instead of an unchecked value passing as clean. Values from a request are untrusted, so check that
+the message is a string first:
+
+```python
+import json
+
+from persian_text_guard import ProfanityFilter, WordList
+
+filter = ProfanityFilter(WordList.persian_default())
+body = json.loads('{"message": ["kir"]}')
+
+message = body.get("message")
+if not isinstance(message, str):
+    outcome = "rejected: message must be text"
+else:
+    outcome = filter.censor(message)
+assert outcome == "rejected: message must be text"
+```
+
+| Error | Raised by | When |
+| --- | --- | --- |
+| `TypeError` | every text function, `censor`, `WordList.parse`, the constructors | A value that is not a `str` (or `None`, where `None` is allowed), an option that is not a `bool`, or an entry that is not a `BannedWord` |
+| `ValueError` | `censor`, enum parameters, `normalize` | An invalid mask, or an unknown category, mode, step or preset name |
+| `WordListFormatError` (a `ValueError`) | `WordList.parse`, `WordList.load` | An unknown section heading; `.line` is 1-based |
+| `OSError`, `UnicodeDecodeError` | `WordList.load` | The file cannot be read, or is not UTF-8 |
+
+## .NET, JavaScript and Python
+
+| .NET | JavaScript/TypeScript | Python |
+| --- | --- | --- |
+| `new ProfanityFilter(words, options)` | `new ProfanityFilter(words, options)` | `ProfanityFilter(words, options)` |
+| `filter.Count` | `filter.count` | `filter.count` |
+| `filter.ContainsProfanity(text)` | `filter.containsProfanity(text)` | `filter.contains_profanity(text)` |
+| `filter.FindMatch(text)` | `filter.findMatch(text)` | `filter.find_match(text)` (`None` when there is no match) |
+| `filter.FindMatches(text)` | `filter.findMatches(text)` | `filter.find_matches(text)` (a tuple) |
+| `filter.Censor(text, '#')` | `filter.censor(text, '#')` | `filter.censor(text, "#")` |
+| `new BannedWord("x", WordMatchMode.Anywhere) { Category = WordCategory.Slur }` | `{ text: 'x', mode: 'anywhere', category: 'slur' }` | `BannedWord("x", WordMatchMode.ANYWHERE, WordCategory.SLUR)` |
+| `ProfanityFilterOptions { SqueezeRepeatedLetters = false }` | `{ squeezeRepeatedLetters: false }` | `ProfanityFilterOptions(squeeze_repeated_letters=False)` |
+| `match.Word`, `match.Evasion`, `match.Index`, `match.Length` | `match.word`, `.evasion`, `.index`, `.length` | `match.word`, `.evasion`, `.index`, `.length` (code points) |
+| `EvasionKind.LookalikeCharacters \| EvasionKind.RepeatedLetters` | `['repeatedLetters', 'lookalikeCharacters']` | `(EvasionKind.REPEATED_LETTERS, EvasionKind.LOOKALIKE_CHARACTERS)` |
+| `WordList.All`, `WordList.PersianDefault` | `WordList.all`, `WordList.persianDefault` | `WordList.all()`, `WordList.persian_default()` |
+| `WordList.Bundled(WordCategory.Slur)` | `WordList.bundled('slur')` | `WordList.bundled(WordCategory.SLUR)` or `WordList.bundled("slur")` |
+| `WordList.Parse(text)` | `WordList.parse(text)` | `WordList.parse(text)` |
+| `WordList.Load(stream)` | read the file, then `WordList.parse(text)` | `WordList.load(path_or_file)` |
+| `PersianNormalizer.Normalize(text, PersianNormalization.Standard)` | `normalize(text, 'standard')` | `normalize(text, "standard")` |
+| `PersianNormalizer.Tokenize(text)` | `tokenize(text)` | `tokenize(text)` |
+| `PersianNormalizer.ToPersianDigits(text)` | `toPersianDigits(text)` | `to_persian_digits(text)` |
+| `FormatException` | `WordListFormatError` (`.line`) | `WordListFormatError` (`.line`), a `ValueError` |
+| `ArgumentException` (mask) | `RangeError` | `ValueError` |
+
+## Performance
+
+Against `WordList.persian_default()` (about 1,000 entries), on CPython 3.14.6 (Intel Core i7-9700K,
+pyperf with `--rigorous`):
+
+| Operation | Mean | Operations/s |
+| --- | ---: | ---: |
+| Short clean message (5 words) | 91.4 µs | 10,937 |
+| Long clean message (60 words) | 909 µs | 1,100 |
+| Message with evasions | 50.4 µs | 19,850 |
+| Normalize a long message | 35.1 µs | 28,454 |
+| Build a filter from the bundled list | 13.3 ms | 75 |
+| `find_matches`, clean short message | 90.4 µs | 11,059 |
+| `find_matches`, message with three banned words | 240 µs | 4,169 |
+| `censor`, short message with one banned word | 118 µs | 8,461 |
+| `censor`, 60-word message with three banned words | 3.2 ms | 311 |
+| A 132,000-character message | 210 ms | 5 |
+
+Build the filter once and reuse it. The messages are the same as the .NET and JavaScript benchmarks';
+run them with `uv run python bench/bench_filter.py -o .bench/results.json` in `python/`, and print the
+table with `uv run python scripts/bench_table.py .bench/results.json`.
+
+## Limitations
+
+- **Unicode versions.** The filter uses the running Python's own Unicode data (`unicodedata`),
+  through a layer that reproduces .NET's rules for whitespace, case and categories. CPython 3.14 and
+  .NET 10 agree on the category of every code point. Older versions differ only on characters assigned
+  in Unicode 15 or later, and Python 3.11 also does not fold the 62 Cyrillic modifier letters
+  U+1E030–U+1E06D in compatibility normalization. The conformance corpus contains none of these
+  characters; if a difference matters to you, please open an issue.
+- **Values that are not text.** `bytes`, numbers and other objects raise `TypeError` instead of being
+  read as an empty or converted message; see [Validating input](#validating-input).
+- **Interpreters.** CPython 3.11 to 3.14, including free-threaded 3.14, is tested. Other interpreters,
+  such as PyPy, may work but are not tested.
+- The evasions the filter reads through, and what it deliberately does not catch, are listed in the
+  [project README](https://github.com/AmirehsanK/PersianTextGuard#what-it-reads-through).
+
+## Links
+
+- [Project README](https://github.com/AmirehsanK/PersianTextGuard#readme)
+- [Conformance corpus](https://github.com/AmirehsanK/PersianTextGuard/tree/main/conformance)
+- [Word lists](https://github.com/AmirehsanK/PersianTextGuard/tree/main/wordlists)
+- [.NET package on NuGet](https://www.nuget.org/packages/PersianTextGuard) and
+  [JavaScript package on npm](https://www.npmjs.com/package/persian-text-guard)
+- [MIT licence](https://github.com/AmirehsanK/PersianTextGuard/blob/main/LICENSE)
+
+---
+
+<div dir="rtl">
+
+## فارسی
+
+</div>
+
+<div dir="rtl">
+
+این بسته نسخهٔ پایتونِ پروژهٔ PersianTextGuard است: یکسان‌سازی متن فارسی و فیلتر کلمات نامناسب که ترفندهای دور زدن فهرست کلمات را می‌شناسد، مثل «ک.ی.ر»، حروف فاصله‌دار، حروف عربی به‌جای فارسی و نویسه‌های نامرئی، و در عین حال پیام‌های عادی مثل «هر کس» و «تخم مرغ» را رد نمی‌کند.
+
+</div>
+
+<div dir="rtl">
+
+برای هر پیام همان پاسخی را می‌دهد که بسته‌های دات‌نت و جاوااسکریپت می‌دهند، و جایگاه‌ها را مثل خودِ پایتون می‌شمارد.
+
+</div>
+
+<div dir="rtl">
+
+### نصب
+
+</div>
+
+```bash
+pip install persian-text-guard
+```
+
+<div dir="rtl">
+
+### شروع سریع
+
+</div>
+
+<div dir="rtl">
+
+فیلتر را یک بار بسازید و همه‌جا از همان استفاده کنید؛ فیلتر بعد از ساخته شدن هرگز تغییر نمی‌کند و استفادهٔ هم‌زمان از چند نخ هم امن است.
+
+</div>
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+# فیلتر را یک بار بسازید و به اشتراک بگذارید.
+# Build once and share it: a filter never changes after it is built.
+filter = ProfanityFilter(WordList.persian_default())
+
+assert not filter.contains_profanity("سلام، سفارشم کی میرسه؟")
+assert filter.contains_profanity("ک.ی.ر")
+```
+
+<div dir="rtl">
+
+### سانسور
+
+</div>
+
+<div dir="rtl">
+
+هر کلمهٔ نامناسب، با پسوندها و فاصله‌هایش، با چهار نویسهٔ ثابت پوشانده می‌شود و بقیهٔ متن دست نمی‌خورد.
+
+</div>
+
+```python
+from persian_text_guard import ProfanityFilter, WordList
+
+# کلمات نامناسب با چهار نویسه پوشانده می‌شوند.
+# Every banned word is hidden behind a four-character mask.
+filter = ProfanityFilter(WordList.persian_default())
+
+assert filter.censor("جنده\N{ZERO WIDTH NON-JOINER}ها رو ببین") == "**** رو ببین"
+assert filter.censor("this is kir", "#") == "this is ####"
+```
+
+<div dir="rtl">
+
+### بررسی ورودی
+
+</div>
+
+<div dir="rtl">
+
+هر رشته‌ای پذیرفته می‌شود و مقدار خالیِ پایتون مثل پیام خالی است، اما هر مقدار دیگری، مثل عدد یا بایت یا فهرست، خطای نوع می‌دهد. پیش از فیلتر کردنِ داده‌های درخواست، بررسی کنید که پیام از نوع رشته باشد.
+
+</div>
+
+<div dir="rtl">
+
+مستندات کامل، فهرست ترفندهایی که شناخته می‌شوند و محدودیت‌ها در [راهنمای اصلی پروژه](https://github.com/AmirehsanK/PersianTextGuard#readme) آمده است.
+
+</div>