iraqi-dialect-bad-words 1.2.0 → 1.3.0

package/README.md

@@ -65,15 +65,44 @@ Creates a filter with the built-in word list.
 new IraqiBadWordsFilter();                               // profanity only
 new IraqiBadWordsFilter({ includeInsults: true });       // also milder insults
 new IraqiBadWordsFilter({ stripSuffixes: true });        // also unlisted suffix forms
+new IraqiBadWordsFilter({ dialects: ['mosul'] });        // also Mosul ر → غ forms
 new IraqiBadWordsFilter({ customWords: ['word1', 'word2'] });
+new IraqiBadWordsFilter({ safeWords: ['word3'] });        // clean words not to flag
 new IraqiBadWordsFilter(['word1', 'word2']);             // same as customWords
 ```
 
 | Option | Default | Description |
 |---|---|---|
-| `customWords` | `[]` | Extra words to filter. |
+| `customWords` | `[]` | Extra words to filter, on top of the built-in list. Can come from your own JSON file (see below). |
+| `safeWords` | `[]` | Extra safe stems, on top of the built-in [`safe-stems.json`](src/safe-stems.json): clean words that would otherwise match once a prefix or suffix is stripped, like the built-in `بعير` ("camel" = `ب` + `عير`). Use it for clean words your app sees flagged, especially ones your own `customWords` cause. Their prefixed and suffixed forms are safe too (`وبعيرك`). Can come from your own JSON file (see below). |
 | `includeInsults` | `false` | Also filter milder insults like كلب, حمار, قندرة, نعال. Off by default because many of them are everyday words, so turning it on flags sentences like `عندي كلب` ("I have a dog"). |
 | `stripSuffixes` | `false` | Also match words with a pronoun suffix (ي ك ج ه ها نا كم هم) that the list doesn't spell out, e.g. `طيزهم` → `طيز`. Off by default because it widens matching: a clean word that ends like a bad word plus a suffix gets flagged too. |
+| `dialects` | `[]` | Regional pronunciations to also match. `'mosul'`: ر said as غ, so every entry with ر also matches its غ form (`شرموطة` → `شغموطة`); the ر forms keep matching. Safe stems get their غ form too. |
+
+#### Using your own word files
+
+Keep your app's extra words, and clean words you don't want flagged, in JSON files: arrays of single words, the same format as [`src/words.json`](src/words.json) and [`src/safe-stems.json`](src/safe-stems.json).
+
+```json
+["زفت", "نذل"]
+```
+
+Import them and pass them as `customWords` and `safeWords`. The built-in lists stay on, and your words are added to them:
+
+```ts
+import IraqiBadWordsFilter from 'iraqi-dialect-bad-words';
+import myWords from './my-words.json'; // TypeScript needs "resolveJsonModule": true
+import mySafeWords from './my-safe-words.json';
+// CommonJS: const myWords = require('./my-words.json');
+
+const filter = new IraqiBadWordsFilter({ customWords: myWords, safeWords: mySafeWords });
+```
+
+Your words go through the same normalization as the built-in lists, so you don't need to list diacritic, ة/ه, stretched or prefixed spellings. Whitespace around a word is trimmed and blank entries are skipped. `inspect()` reports a match from your word file with `source: 'custom'`, and a match cleared by your safe words with `suppressedBy` set to that safe word.
+
+A safe word only clears a match that needed a prefix or suffix stripped, like the built-in safe stems. To stop a listed word itself from being flagged, use `removeWords()`.
+
+React Native, Expo, Vite and Next.js bundle the JSON files into your app, so this works the same in each. The filter doesn't read files at runtime. If your lists come from a server, fetch them and pass the arrays to `addWords()` and `addSafeWords()`.
 
 ### `isProfane(text: string): boolean`
 
@@ -89,34 +118,55 @@ Lists each bad word in the text and explains why it matched. Use it to highlight
 
 ```ts
 filter.inspect('روح للكواد!');
-// [{ token: 'للكواد', start: 4, end: 10, normalized: 'لكواد',
-//    entry: 'كواد', source: 'words', prefix: 'ل' }]
+// [{ token: 'للكواد', start: 4, end: 10, normalized: 'لكواد', entry: 'كواد',
+//    source: 'words', prefix: 'ل', suffix: '', rules: ['stretched', 'prefix'] }]
 ```
 
 | Field | Description |
 |---|---|
 | `token` | The word exactly as written. |
 | `start`, `end` | Offsets in the **original** text, so `text.slice(start, end) === token`. Counted in JavaScript string units (UTF-16); convert them if another platform counts characters differently. |
-| `normalized` | The word after normalization (Arabizi is folded too, so `Qa7ba` shows as `gahba`). |
-| `entry` | The list entry it matched, spelled as in the list. |
+| `normalized` | The word after normalization (Arabizi is folded too, so `Qa7ba` shows as `gahba`). For a word with گ, the reading that matched (ق or ك). |
+| `entry` | The list entry it matched, spelled as in the list. For a Mosul form, the listed ر spelling. |
 | `source` | `'words'`, `'insults'` or `'custom'`. A word on more than one list keeps the first: built-in, then insults, then custom. |
 | `prefix` | The attached prefix stripped before matching (after normalization, so `لل` shows as `ل`), or `''` for a direct match. |
 | `suffix` | The suffix stripped before matching (only with `stripSuffixes`), or `''`. |
+| `rules` | The rules the word needed to match the entry (see below). Empty when it's spelled exactly like the entry. |
 | `suppressedBy` | Only with `includeSuppressed`: the safe stem that cleared this match. |
 
 Pass `{ includeSuppressed: true }` to also see words that matched but were cleared by a safe stem. They are not profane (`isProfane` and `clean` ignore them), but they show why a word *wasn't* flagged:
 
 ```ts
 filter.inspect('وبعيرك', { includeSuppressed: true });
-// [{ token: 'وبعيرك', …, entry: 'عيرك', prefix: 'وب', suppressedBy: 'بعير' }]
+// [{ token: 'وبعيرك', …, entry: 'عيرك', prefix: 'وب', rules: ['prefix'], suppressedBy: 'بعير' }]
 ```
 
-Including the output of `inspect` in a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml) shows exactly which entry and prefix rule fired.
+`rules` lists only the rules that made a difference between the typed word and the entry, so a report shows straight away why something matched:
+
+| Rule | Example |
+|---|---|
+| `diacritics` | `عَارَة`, `عـــارة` |
+| `invisible` | `عا‌رة` (zero-width character inside) |
+| `spelling` | `عاره` (ة/ه, ى/ي, أ/ا) |
+| `keyboard` | `عیر`, `کس` (Persian/Kurdish letters) |
+| `arabic-digits` | `ga٧ba` |
+| `stretched` | `عاااارة`, `للكواد` |
+| `glued-digits` | `عارة3` |
+| `arabizi` | `qa7ba` → `gahba` |
+| `gaf-as-qaf` / `gaf-as-kaf` | `گحبة` → قحبة, `گسها` → كسها |
+| `prefix` / `suffix` | `بطيزك`, `طيزهم` (with `stripSuffixes`) |
+| `mosul` | `شغموطة` → شرموطة (with `dialects: ['mosul']`) |
+
+Including the output of `inspect` in a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml) shows exactly which entry matched and which rules it needed; the issue form has a field for it.
 
 ### `addWords(words: string[]): void`
 
 Adds words to this filter.
 
+### `addSafeWords(words: string[]): void`
+
+Adds safe stems to this filter, like the `safeWords` option.
+
 ### `removeWords(words: string[]): void`
 
 Removes words from this filter, for example a built-in word you don't want flagged in your app.
@@ -131,10 +181,14 @@ Matching is done per whole word, after normalizing both the text and the word li
 | Stretched letters | `عاااارة` |
 | Tatweel | `عـــارة` |
 | ة / ه, ى / ي, أ / إ / آ / ا | `عاره` |
+| Persian/Kurdish keyboard letters ی ک ە | `عیر`, `کس` |
+| گ read as ق (Iraqi) and as ك (Kurdish typists) | `گحبة` = `قحبة`, `گس` = `كس` |
+| Invisible characters hidden inside a word (zero-width, direction marks, soft hyphen) | `عا‌رة` |
 | Attached prefixes و ف ب ل ك ال | `والعارة`, `للعارة` |
 | Pronoun suffixes (with `stripSuffixes`) | `طيزهم`, `وطيزكم` |
 | Punctuation next to the word | `عارة!` `عارة،` |
-| Arabizi (Latin letters and numbers), including spelling variants | `ga7ba` = `qahba` = `gahba`, `kosomak` = `kusumak`, `3ayri` = `ayri` |
+| Digits glued to or inside an Arabic word | `عارة3`, `عا٣رة` |
+| Arabizi (Latin letters and numbers, Western or Arabic-Indic), including spelling variants | `ga7ba` = `ga٧ba` = `qahba` = `gahba`, `kosomak` = `kusumak`, `3ayri` = `ayri` |
 
 Because matching is whole-word, a clean word that merely contains a bad word is not flagged.
 
@@ -151,6 +205,7 @@ The word list is the heart of this package, and native speakers are the best jud
 - **A word is missing:** open a [“Suggest a word” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=add-word.yml), or add it to [`src/words.json`](src/words.json) in a pull request.
 - **A clean word is flagged:** open a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml).
   If it's flagged only because of a prefix (e.g. `بعير` "camel" = `ب` + a bad word), the fix is adding it to [`src/safe-stems.json`](src/safe-stems.json).
+  A safe stem written with گ protects only words spelled with گ, so `گلب` ("heart", Iraqi spelling of قلب) can be safe while `كلب` is not.
 
 - **A sentence is judged wrong:** add it to [`test/fixtures/sentences.json`](test/fixtures/sentences.json) with the verdict a native speaker expects (`"profane": true` or `false`, plus `"options"` if it needs `includeInsults` or `stripSuffixes`). Every sentence there runs as a test, so later word-list changes can't silently flip it.
 

package/dist/index.d.mts

@@ -1,6 +1,22 @@
+/**
+ * A rule a word needed to match its list entry, reported in ProfanityMatch.rules.
+ * Only rules that made a difference are reported: عارة matching عارة has none.
+ */
+type MatchRule = "diacritics" | "invisible" | "spelling" | "keyboard" | "arabic-digits" | "stretched" | "glued-digits" | "arabizi" | "gaf-as-qaf" | "gaf-as-kaf" | "prefix" | "suffix" | "mosul";
 interface IraqiBadWordsFilterOptions {
-    /** Extra words to filter, on top of the built-in list. */
+    /**
+     * Extra words to filter, on top of the built-in list. Can be your own JSON
+     * word file: `customWords: require('./my-words.json')`.
+     */
     customWords?: string[];
+    /**
+     * Extra safe stems, on top of the built-in safe-stems.json: clean words that
+     * would otherwise match once a prefix or suffix is stripped, like the
+     * built-in بعير ("camel" = ب + عير). Their prefixed and suffixed forms are
+     * safe too (وبعيرك).
+     * Can be your own JSON file: `safeWords: require('./my-safe-words.json')`.
+     */
+    safeWords?: string[];
     /**
      * Also filter milder insults (كلب, حمار, قندرة, …). Off by default because
      * many of them are everyday words ("عندي كلب", "جلب لي هدية").
@@ -12,7 +28,15 @@ interface IraqiBadWordsFilterOptions {
      * clean words that happen to end like a bad word plus a suffix.
      */
     stripSuffixes?: boolean;
+    /**
+     * Regional pronunciations to also match. "mosul": ر said as غ, so every entry
+     * and safe stem with ر also matches its غ form (عير -> عيغ, بعير -> بعيغ).
+     * The ر forms keep matching. Off by default.
+     */
+    dialects?: Dialect[];
 }
+/** A regional pronunciation the filter can also match. */
+type Dialect = "mosul";
 /** Which list a dictionary entry came from. */
 type WordSource = "words" | "insults" | "custom";
 interface InspectOptions {
@@ -33,9 +57,15 @@ interface ProfanityMatch {
     start: number;
     /** Offset just past the end of the word in the input. */
     end: number;
-    /** The word after normalization (diacritics, stretched letters, … removed). */
+    /**
+     * The word after normalization (diacritics, stretched letters, … removed).
+     * For a word with گ, the reading that matched: گ as ق or as ك.
+     */
     normalized: string;
-    /** The dictionary entry it matched, spelled as in its list. */
+    /**
+     * The dictionary entry it matched, spelled as in its list. For a Mosul form
+     * (rule "mosul"), the listed ر spelling: الشرموطة for الشغموطة.
+     */
     entry: string;
     /** The list the entry came from. */
     source: WordSource;
@@ -43,8 +73,10 @@ interface ProfanityMatch {
     prefix: string;
     /** The suffix stripped before matching (only with stripSuffixes), or "". */
     suffix: string;
+    /** The rules the word needed to match the entry, e.g. ["prefix", "arabizi"]. */
+    rules: MatchRule[];
     /**
-     * The safe stem that cleared this match, spelled as in safe-stems.json.
+     * The safe stem that cleared this match, spelled as in safe-stems.json or safeWords.
      * Only set on matches returned with `includeSuppressed`; these are not profane.
      */
     suppressedBy?: string;
@@ -53,22 +85,35 @@ declare class IraqiBadWordsFilter {
     private words;
     private safeStems;
     private stripSuffixes;
+    private mosul;
     constructor(options?: string[] | IraqiBadWordsFilterOptions);
     /**
-     * Normalizes Arabic text so spelling variants compare equal:
-     * removes diacritics (Tashkeel) and tatweel, unifies Alef/Teh Marbuta/Alef Maksura
-     * forms, and collapses stretched letters (عاااارة -> عارة).
+     * Normalizes a word or token for comparison: removes diacritics, tatweel and
+     * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
+     * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
+     * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
+     */
+    private normalize;
+    /**
+     * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
+     * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
      */
-    private normalizeArabic;
+    private readings;
+    /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
+    private dialectForms;
+    /** The key a list word is stored under. A گ in a list entry means ق. */
+    private entryKey;
     /**
-     * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
-     * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
-     * teezak / tizak. Only words made purely of Latin letters and digits are
-     * folded; expects text already passed through normalizeArabic.
+     * A word list with surrounding whitespace trimmed and entries that are empty
+     * after normalization skipped. A stray "" in a user's file would otherwise
+     * match a lone tatweel as a word, or clear every match as a safe stem.
      */
-    private foldArabizi;
-    /** Normalizes a word or token for comparison. */
-    private normalize;
+    private cleanList;
+    /**
+     * Adds safe stems from one list. A safe stem written with گ protects only
+     * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
+     */
+    private addSafeStems;
     /**
      * Adds words from one list. A word already present keeps its first source.
      */
@@ -80,8 +125,26 @@ declare class IraqiBadWordsFilter {
      * A match cleared by a safe stem is returned with `suppressedBy` set.
      */
     private matchWord;
-    /** True when the word is profane, i.e. matched and not cleared by a safe stem. */
-    private isProfaneWord;
+    /**
+     * Works out which rules a match needed, by comparing the typed word with the
+     * listed spelling (plus any stripped prefix and suffix). A normalization rule
+     * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
+     * "stretched" but not "spelling", since both have ة.
+     */
+    private explain;
+    /** The word plus each form of it with an attached prefix removed. */
+    private prefixCandidates;
+    /** The first safe stem one of the candidates starts with, as listed. */
+    private findSafe;
+    /** Matches one normalized reading of a word. */
+    private matchReading;
+    /**
+     * Finds every match in the text, including ones cleared by a safe stem.
+     * A token with invisible characters is matched whole first (عا‌رة -> عارة);
+     * if that finds nothing profane, each visible piece is tried on its own, so an
+     * invisible character between two words can't hide a bad word either.
+     */
+    private findMatches;
     /**
      * Checks if a given text contains profane words.
      */
@@ -99,10 +162,14 @@ declare class IraqiBadWordsFilter {
      * Dynamically add words to the active dictionary.
      */
     addWords(newWords: string[]): void;
+    /**
+     * Add safe stems to this filter, like the safeWords option.
+     */
+    addSafeWords(newWords: string[]): void;
     /**
      * Remove words from the active filter dictionary.
      */
     removeWords(wordsToRemove: string[]): void;
 }
 
-export { type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
+export { type Dialect, type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type MatchRule, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };

package/dist/index.d.ts

@@ -1,6 +1,22 @@
+/**
+ * A rule a word needed to match its list entry, reported in ProfanityMatch.rules.
+ * Only rules that made a difference are reported: عارة matching عارة has none.
+ */
+type MatchRule = "diacritics" | "invisible" | "spelling" | "keyboard" | "arabic-digits" | "stretched" | "glued-digits" | "arabizi" | "gaf-as-qaf" | "gaf-as-kaf" | "prefix" | "suffix" | "mosul";
 interface IraqiBadWordsFilterOptions {
-    /** Extra words to filter, on top of the built-in list. */
+    /**
+     * Extra words to filter, on top of the built-in list. Can be your own JSON
+     * word file: `customWords: require('./my-words.json')`.
+     */
     customWords?: string[];
+    /**
+     * Extra safe stems, on top of the built-in safe-stems.json: clean words that
+     * would otherwise match once a prefix or suffix is stripped, like the
+     * built-in بعير ("camel" = ب + عير). Their prefixed and suffixed forms are
+     * safe too (وبعيرك).
+     * Can be your own JSON file: `safeWords: require('./my-safe-words.json')`.
+     */
+    safeWords?: string[];
     /**
      * Also filter milder insults (كلب, حمار, قندرة, …). Off by default because
      * many of them are everyday words ("عندي كلب", "جلب لي هدية").
@@ -12,7 +28,15 @@ interface IraqiBadWordsFilterOptions {
      * clean words that happen to end like a bad word plus a suffix.
      */
     stripSuffixes?: boolean;
+    /**
+     * Regional pronunciations to also match. "mosul": ر said as غ, so every entry
+     * and safe stem with ر also matches its غ form (عير -> عيغ, بعير -> بعيغ).
+     * The ر forms keep matching. Off by default.
+     */
+    dialects?: Dialect[];
 }
+/** A regional pronunciation the filter can also match. */
+type Dialect = "mosul";
 /** Which list a dictionary entry came from. */
 type WordSource = "words" | "insults" | "custom";
 interface InspectOptions {
@@ -33,9 +57,15 @@ interface ProfanityMatch {
     start: number;
     /** Offset just past the end of the word in the input. */
     end: number;
-    /** The word after normalization (diacritics, stretched letters, … removed). */
+    /**
+     * The word after normalization (diacritics, stretched letters, … removed).
+     * For a word with گ, the reading that matched: گ as ق or as ك.
+     */
     normalized: string;
-    /** The dictionary entry it matched, spelled as in its list. */
+    /**
+     * The dictionary entry it matched, spelled as in its list. For a Mosul form
+     * (rule "mosul"), the listed ر spelling: الشرموطة for الشغموطة.
+     */
     entry: string;
     /** The list the entry came from. */
     source: WordSource;
@@ -43,8 +73,10 @@ interface ProfanityMatch {
     prefix: string;
     /** The suffix stripped before matching (only with stripSuffixes), or "". */
     suffix: string;
+    /** The rules the word needed to match the entry, e.g. ["prefix", "arabizi"]. */
+    rules: MatchRule[];
     /**
-     * The safe stem that cleared this match, spelled as in safe-stems.json.
+     * The safe stem that cleared this match, spelled as in safe-stems.json or safeWords.
      * Only set on matches returned with `includeSuppressed`; these are not profane.
      */
     suppressedBy?: string;
@@ -53,22 +85,35 @@ declare class IraqiBadWordsFilter {
     private words;
     private safeStems;
     private stripSuffixes;
+    private mosul;
     constructor(options?: string[] | IraqiBadWordsFilterOptions);
     /**
-     * Normalizes Arabic text so spelling variants compare equal:
-     * removes diacritics (Tashkeel) and tatweel, unifies Alef/Teh Marbuta/Alef Maksura
-     * forms, and collapses stretched letters (عاااارة -> عارة).
+     * Normalizes a word or token for comparison: removes diacritics, tatweel and
+     * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
+     * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
+     * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
+     */
+    private normalize;
+    /**
+     * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
+     * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
      */
-    private normalizeArabic;
+    private readings;
+    /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
+    private dialectForms;
+    /** The key a list word is stored under. A گ in a list entry means ق. */
+    private entryKey;
     /**
-     * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
-     * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
-     * teezak / tizak. Only words made purely of Latin letters and digits are
-     * folded; expects text already passed through normalizeArabic.
+     * A word list with surrounding whitespace trimmed and entries that are empty
+     * after normalization skipped. A stray "" in a user's file would otherwise
+     * match a lone tatweel as a word, or clear every match as a safe stem.
      */
-    private foldArabizi;
-    /** Normalizes a word or token for comparison. */
-    private normalize;
+    private cleanList;
+    /**
+     * Adds safe stems from one list. A safe stem written with گ protects only
+     * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
+     */
+    private addSafeStems;
     /**
      * Adds words from one list. A word already present keeps its first source.
      */
@@ -80,8 +125,26 @@ declare class IraqiBadWordsFilter {
      * A match cleared by a safe stem is returned with `suppressedBy` set.
      */
     private matchWord;
-    /** True when the word is profane, i.e. matched and not cleared by a safe stem. */
-    private isProfaneWord;
+    /**
+     * Works out which rules a match needed, by comparing the typed word with the
+     * listed spelling (plus any stripped prefix and suffix). A normalization rule
+     * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
+     * "stretched" but not "spelling", since both have ة.
+     */
+    private explain;
+    /** The word plus each form of it with an attached prefix removed. */
+    private prefixCandidates;
+    /** The first safe stem one of the candidates starts with, as listed. */
+    private findSafe;
+    /** Matches one normalized reading of a word. */
+    private matchReading;
+    /**
+     * Finds every match in the text, including ones cleared by a safe stem.
+     * A token with invisible characters is matched whole first (عا‌رة -> عارة);
+     * if that finds nothing profane, each visible piece is tried on its own, so an
+     * invisible character between two words can't hide a bad word either.
+     */
+    private findMatches;
     /**
      * Checks if a given text contains profane words.
      */
@@ -99,10 +162,14 @@ declare class IraqiBadWordsFilter {
      * Dynamically add words to the active dictionary.
      */
     addWords(newWords: string[]): void;
+    /**
+     * Add safe stems to this filter, like the safeWords option.
+     */
+    addSafeWords(newWords: string[]): void;
     /**
      * Remove words from the active filter dictionary.
      */
     removeWords(wordsToRemove: string[]): void;
 }
 
-export { type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
+export { type Dialect, type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type MatchRule, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };

package/dist/index.js

@@ -233,56 +233,134 @@ var insults_default = [
 var safe_stems_default = [
   "\u0628\u0639\u064A\u0631",
   "\u0628\u0639\u064A\u063A",
+  "\u06AF\u0644\u0628",
+  "\u06AF\u0644\u0627\u0628",
+  "\u0637\u06AF\u0639\u0629",
   "\u0628\u0643\u0633",
   "\u0648\u0643\u0633",
-  "\u0641\u0643\u0633"
+  "\u0641\u0643\u0633",
+  "\u0628\u062B\u0648\u0631",
+  "\u062B\u0648\u0631\u0629",
+  "\u062B\u0648\u0631\u064A",
+  "\u0628\u0627\u0644\u0639\u062C\u0644"
 ];
 
 // src/index.ts
-var WORD_PATTERN = /[A-Za-z0-9_À-ɏ؀-؋؍-ؚ؜-؞ؠ-٩ٮ-ۓە-ۿݐ-ݿࢠ-ࣿﭐ-﷿ﹰ-ﻼ]+/g;
+var INVISIBLE = "\\u00AD\\u061C\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u2069\\uFEFF";
+var INVISIBLE_CHARS = new RegExp(`[${INVISIBLE}]`, "g");
+var HAS_INVISIBLE = new RegExp(`[${INVISIBLE}]`);
+var VISIBLE_RUN = new RegExp(`[^${INVISIBLE}]+`, "g");
+var WORD_PATTERN = new RegExp(
+  `[A-Za-z0-9_\xC0-\u024F\u0600-\u060B\u060D-\u061A\u061C-\u061E\u0620-\u0669\u066E-\u06D3\u06D5-\u06FF\u0750-\u077F\u08A0-\u08FF\uFB50-\uFDFF\uFE70-\uFEFC${INVISIBLE}]+`,
+  "g"
+);
 var CONJUNCTIONS = ["", "\u0648", "\u0641"];
 var PREPOSITIONS = ["", "\u0628", "\u0644", "\u0643"];
 var ARTICLES = ["", "\u0627\u0644"];
 var SUFFIXES = ["\u064A", "\u0643", "\u062C", "\u0647", "\u0647\u0627", "\u0646\u0627", "\u0643\u0645", "\u0647\u0645"];
+var HAS_ARABIC = /[\u0600-\uFEFF]/;
+function foldArabizi(word) {
+  if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
+  return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
+}
+var NORMALIZE_STEPS = [
+  { rule: "diacritics", apply: (t) => t.replace(/[\u064B-\u065F\u0670\u0640]/g, "") },
+  { rule: "invisible", apply: (t) => t.replace(INVISIBLE_CHARS, "") },
+  { rule: "spelling", apply: (t) => t.replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A") },
+  { rule: "keyboard", apply: (t) => t.replace(/[ەہۀ]/g, "\u0647").replace(/ی/g, "\u064A").replace(/[کڪ]/g, "\u0643") },
+  {
+    rule: "arabic-digits",
+    apply: (t) => t.replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776))
+  },
+  { rule: "stretched", apply: (t) => t.replace(/(.)\1+/gu, "$1") },
+  { rule: null, apply: (t) => t.toLowerCase() },
+  // Digits glued to an Arabic word (عارة3, عا٣رة) are padding; in Arabizi they're letters
+  {
+    rule: "glued-digits",
+    apply: (t) => HAS_ARABIC.test(t) && /[0-9]/.test(t) ? t.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : t
+  },
+  { rule: "arabizi", apply: (t) => HAS_ARABIC.test(t) ? t : foldArabizi(t) }
+];
+function normalizeSteps(text, skip = -1) {
+  return NORMALIZE_STEPS.reduce((t, step, i) => i === skip ? t : step.apply(t), text);
+}
 var IraqiBadWordsFilter = class {
   constructor(options = {}) {
-    const { customWords = [], includeInsults = false, stripSuffixes = false } = Array.isArray(options) ? { customWords: options } : options;
+    // Clean words that would otherwise match after a prefix is stripped
+    // (e.g. بعير "camel" = ب + عير). Also covers their prefixed and suffixed
+    // forms (وبعيرك). Listed in safe-stems.json, plus the safeWords option.
+    this.safeStems = [];
+    const {
+      customWords = [],
+      safeWords = [],
+      includeInsults = false,
+      stripSuffixes = false,
+      dialects = []
+    } = Array.isArray(options) ? { customWords: options } : options;
     this.words = /* @__PURE__ */ new Map();
     this.stripSuffixes = stripSuffixes;
-    this.safeStems = safe_stems_default.map((word) => ({ normalized: this.normalize(word), word }));
+    this.mosul = dialects.includes("mosul");
+    this.addSafeStems(safe_stems_default);
+    this.addSafeStems(safeWords);
     this.addEntries(words_default, "words");
     if (includeInsults) this.addEntries(insults_default, "insults");
     this.addEntries(customWords, "custom");
   }
   /**
-   * Normalizes Arabic text so spelling variants compare equal:
-   * removes diacritics (Tashkeel) and tatweel, unifies Alef/Teh Marbuta/Alef Maksura
-   * forms, and collapses stretched letters (عاااارة -> عارة).
+   * Normalizes a word or token for comparison: removes diacritics, tatweel and
+   * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
+   * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
+   * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
    */
-  normalizeArabic(text) {
-    return text.replace(/[ً-ٰٟـ]/g, "").replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A").replace(/(.)\1+/gu, "$1").toLowerCase();
+  normalize(text) {
+    return normalizeSteps(text);
   }
   /**
-   * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
-   * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
-   * teezak / tizak. Only words made purely of Latin letters and digits are
-   * folded; expects text already passed through normalizeArabic.
+   * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
+   * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
    */
-  foldArabizi(word) {
-    if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
-    return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
+  readings(word) {
+    return word.includes("\u06AF") ? [word.replace(/گ/g, "\u0642"), word.replace(/گ/g, "\u0643")] : [word];
   }
-  /** Normalizes a word or token for comparison. */
-  normalize(text) {
-    return this.foldArabizi(this.normalizeArabic(text));
+  /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
+  dialectForms(word) {
+    return this.mosul && word.includes("\u0631") ? [word, word.replace(/ر/g, "\u063A")] : [word];
+  }
+  /** The key a list word is stored under. A گ in a list entry means ق. */
+  entryKey(word) {
+    return this.normalize(word).replace(/گ/g, "\u0642");
+  }
+  /**
+   * A word list with surrounding whitespace trimmed and entries that are empty
+   * after normalization skipped. A stray "" in a user's file would otherwise
+   * match a lone tatweel as a word, or clear every match as a safe stem.
+   */
+  cleanList(list) {
+    return list.filter((word) => typeof word === "string").map((word) => word.trim()).filter((word) => this.entryKey(word) !== "");
+  }
+  /**
+   * Adds safe stems from one list. A safe stem written with گ protects only
+   * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
+   */
+  addSafeStems(stems) {
+    for (const stem of this.cleanList(stems)) {
+      for (const word of this.dialectForms(stem)) {
+        this.safeStems.push({ normalized: this.normalize(word), word });
+      }
+    }
   }
   /**
    * Adds words from one list. A word already present keeps its first source.
    */
-  addEntries(newWords, source) {
-    for (const word of newWords) {
-      const normalized = this.normalize(word);
-      if (!this.words.has(normalized)) this.words.set(normalized, { word, source });
+  addEntries(listWords, source) {
+    const newWords = this.cleanList(listWords);
+    const forms = [
+      ...newWords.map((word) => ({ word, form: word })),
+      ...newWords.flatMap((word) => this.dialectForms(word).slice(1).map((form) => ({ word, form })))
+    ];
+    for (const { word, form } of forms) {
+      const key = this.entryKey(form);
+      if (!this.words.has(key)) this.words.set(key, { word, form, source });
     }
   }
   /**
@@ -292,11 +370,44 @@ var IraqiBadWordsFilter = class {
    * A match cleared by a safe stem is returned with `suppressedBy` set.
    */
   matchWord(token) {
-    const word = this.normalize(token);
-    const direct = this.words.get(word);
-    if (direct) {
-      return { normalized: word, entry: direct.word, source: direct.source, prefix: "", suffix: "" };
+    const spelled = this.normalize(token);
+    const spelledSafe = spelled.includes("\u06AF") ? this.findSafe(this.prefixCandidates(spelled)) : void 0;
+    let suppressed = null;
+    for (const reading of this.readings(spelled)) {
+      const found = this.matchReading(reading);
+      if (!found) continue;
+      const { form, ...rest } = found;
+      const match = { ...rest, rules: this.explain(token, found, spelled, reading) };
+      if (match.suppressedBy === void 0 && spelledSafe) match.suppressedBy = spelledSafe;
+      if (match.suppressedBy === void 0) return match;
+      suppressed = suppressed ?? match;
+    }
+    return suppressed;
+  }
+  /**
+   * Works out which rules a match needed, by comparing the typed word with the
+   * listed spelling (plus any stripped prefix and suffix). A normalization rule
+   * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
+   * "stretched" but not "spelling", since both have ة.
+   */
+  explain(token, found, spelled, reading) {
+    const listed = found.prefix + found.form + found.suffix;
+    const gaf = reading === spelled ? "" : reading === spelled.replace(/گ/g, "\u0642") ? "\u0642" : "\u0643";
+    const same = (typed, list) => typed.replace(/گ/g, gaf || "\u06AF") === list.replace(/گ/g, "\u0642");
+    const rules = [];
+    NORMALIZE_STEPS.forEach(({ rule }, i) => {
+      if (rule && !same(normalizeSteps(token, i), normalizeSteps(listed, i))) rules.push(rule);
+    });
+    if (gaf && normalizeSteps(token) !== normalizeSteps(listed)) {
+      rules.push(gaf === "\u0642" ? "gaf-as-qaf" : "gaf-as-kaf");
     }
+    if (found.prefix) rules.push("prefix");
+    if (found.suffix) rules.push("suffix");
+    if (found.form !== found.entry) rules.push("mosul");
+    return rules;
+  }
+  /** The word plus each form of it with an attached prefix removed. */
+  prefixCandidates(word) {
     const candidates = [{ prefix: "", stem: word, suffix: "" }];
     for (const conjunction of CONJUNCTIONS) {
       for (const preposition of PREPOSITIONS) {
@@ -308,6 +419,30 @@ var IraqiBadWordsFilter = class {
         }
       }
     }
+    return candidates;
+  }
+  /** The first safe stem one of the candidates starts with, as listed. */
+  findSafe(candidates) {
+    for (const { stem } of candidates) {
+      const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
+      if (safe) return safe.word;
+    }
+    return void 0;
+  }
+  /** Matches one normalized reading of a word. */
+  matchReading(word) {
+    const direct = this.words.get(word);
+    if (direct) {
+      return {
+        normalized: word,
+        entry: direct.word,
+        form: direct.form,
+        source: direct.source,
+        prefix: "",
+        suffix: ""
+      };
+    }
+    const candidates = this.prefixCandidates(word);
     if (this.stripSuffixes) {
       for (const { prefix, stem } of [...candidates]) {
         for (const suffix of SUFFIXES) {
@@ -319,34 +454,55 @@ var IraqiBadWordsFilter = class {
     }
     const match = candidates.find(({ stem }) => this.words.has(stem));
     if (!match) return null;
-    const { word: entry, source } = this.words.get(match.stem);
+    const { word: entry, form, source } = this.words.get(match.stem);
     const result = {
       normalized: word,
       entry,
+      form,
       source,
       prefix: match.prefix,
       suffix: match.suffix
     };
-    for (const { stem } of candidates) {
-      const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
-      if (safe) return { ...result, suppressedBy: safe.word };
-    }
-    return result;
+    const safe = this.findSafe(candidates);
+    return safe ? { ...result, suppressedBy: safe } : result;
   }
-  /** True when the word is profane, i.e. matched and not cleared by a safe stem. */
-  isProfaneWord(token) {
-    const match = this.matchWord(token);
-    return match !== null && match.suppressedBy === void 0;
+  /**
+   * Finds every match in the text, including ones cleared by a safe stem.
+   * A token with invisible characters is matched whole first (عا‌رة -> عارة);
+   * if that finds nothing profane, each visible piece is tried on its own, so an
+   * invisible character between two words can't hide a bad word either.
+   */
+  *findMatches(text) {
+    const pattern = new RegExp(WORD_PATTERN.source, "g");
+    let found;
+    while (found = pattern.exec(text)) {
+      const token = found[0];
+      const start = found.index;
+      const whole = this.matchWord(token);
+      const wholeMatch = whole && { token, start, end: start + token.length, ...whole };
+      if (whole && whole.suppressedBy === void 0 || !HAS_INVISIBLE.test(token)) {
+        if (wholeMatch) yield wholeMatch;
+        continue;
+      }
+      const pieces = [];
+      const piecePattern = new RegExp(VISIBLE_RUN.source, "g");
+      let piece;
+      while (piece = piecePattern.exec(token)) {
+        const match = this.matchWord(piece[0]);
+        const pieceStart = start + piece.index;
+        if (match) pieces.push({ token: piece[0], start: pieceStart, end: pieceStart + piece[0].length, ...match });
+      }
+      if (pieces.length) yield* pieces;
+      else if (wholeMatch) yield wholeMatch;
+    }
   }
   /**
    * Checks if a given text contains profane words.
    */
   isProfane(text) {
     if (!text) return false;
-    for (const token of text.match(WORD_PATTERN) ?? []) {
-      if (this.isProfaneWord(token)) {
-        return true;
-      }
+    for (const match of this.findMatches(text)) {
+      if (match.suppressedBy === void 0) return true;
     }
     return false;
   }
@@ -357,14 +513,8 @@ var IraqiBadWordsFilter = class {
   inspect(text, options = {}) {
     if (!text) return [];
     const matches = [];
-    const pattern = new RegExp(WORD_PATTERN.source, "g");
-    let found;
-    while (found = pattern.exec(text)) {
-      const token = found[0];
-      const match = this.matchWord(token);
-      if (match && (match.suppressedBy === void 0 || options.includeSuppressed)) {
-        matches.push({ token, start: found.index, end: found.index + token.length, ...match });
-      }
+    for (const match of this.findMatches(text)) {
+      if (match.suppressedBy === void 0 || options.includeSuppressed) matches.push(match);
     }
     return matches;
   }
@@ -387,11 +537,19 @@ var IraqiBadWordsFilter = class {
   addWords(newWords) {
     this.addEntries(newWords, "custom");
   }
+  /**
+   * Add safe stems to this filter, like the safeWords option.
+   */
+  addSafeWords(newWords) {
+    this.addSafeStems(newWords);
+  }
   /**
    * Remove words from the active filter dictionary.
    */
   removeWords(wordsToRemove) {
-    wordsToRemove.forEach((word) => this.words.delete(this.normalize(word)));
+    for (const word of wordsToRemove.flatMap((w) => this.dialectForms(w))) {
+      this.words.delete(this.entryKey(word));
+    }
   }
 };
 var index_default = IraqiBadWordsFilter;

package/dist/index.mjs

@@ -206,56 +206,134 @@ var insults_default = [
 var safe_stems_default = [
   "\u0628\u0639\u064A\u0631",
   "\u0628\u0639\u064A\u063A",
+  "\u06AF\u0644\u0628",
+  "\u06AF\u0644\u0627\u0628",
+  "\u0637\u06AF\u0639\u0629",
   "\u0628\u0643\u0633",
   "\u0648\u0643\u0633",
-  "\u0641\u0643\u0633"
+  "\u0641\u0643\u0633",
+  "\u0628\u062B\u0648\u0631",
+  "\u062B\u0648\u0631\u0629",
+  "\u062B\u0648\u0631\u064A",
+  "\u0628\u0627\u0644\u0639\u062C\u0644"
 ];
 
 // src/index.ts
-var WORD_PATTERN = /[A-Za-z0-9_À-ɏ؀-؋؍-ؚ؜-؞ؠ-٩ٮ-ۓە-ۿݐ-ݿࢠ-ࣿﭐ-﷿ﹰ-ﻼ]+/g;
+var INVISIBLE = "\\u00AD\\u061C\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u2069\\uFEFF";
+var INVISIBLE_CHARS = new RegExp(`[${INVISIBLE}]`, "g");
+var HAS_INVISIBLE = new RegExp(`[${INVISIBLE}]`);
+var VISIBLE_RUN = new RegExp(`[^${INVISIBLE}]+`, "g");
+var WORD_PATTERN = new RegExp(
+  `[A-Za-z0-9_\xC0-\u024F\u0600-\u060B\u060D-\u061A\u061C-\u061E\u0620-\u0669\u066E-\u06D3\u06D5-\u06FF\u0750-\u077F\u08A0-\u08FF\uFB50-\uFDFF\uFE70-\uFEFC${INVISIBLE}]+`,
+  "g"
+);
 var CONJUNCTIONS = ["", "\u0648", "\u0641"];
 var PREPOSITIONS = ["", "\u0628", "\u0644", "\u0643"];
 var ARTICLES = ["", "\u0627\u0644"];
 var SUFFIXES = ["\u064A", "\u0643", "\u062C", "\u0647", "\u0647\u0627", "\u0646\u0627", "\u0643\u0645", "\u0647\u0645"];
+var HAS_ARABIC = /[\u0600-\uFEFF]/;
+function foldArabizi(word) {
+  if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
+  return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
+}
+var NORMALIZE_STEPS = [
+  { rule: "diacritics", apply: (t) => t.replace(/[\u064B-\u065F\u0670\u0640]/g, "") },
+  { rule: "invisible", apply: (t) => t.replace(INVISIBLE_CHARS, "") },
+  { rule: "spelling", apply: (t) => t.replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A") },
+  { rule: "keyboard", apply: (t) => t.replace(/[ەہۀ]/g, "\u0647").replace(/ی/g, "\u064A").replace(/[کڪ]/g, "\u0643") },
+  {
+    rule: "arabic-digits",
+    apply: (t) => t.replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776))
+  },
+  { rule: "stretched", apply: (t) => t.replace(/(.)\1+/gu, "$1") },
+  { rule: null, apply: (t) => t.toLowerCase() },
+  // Digits glued to an Arabic word (عارة3, عا٣رة) are padding; in Arabizi they're letters
+  {
+    rule: "glued-digits",
+    apply: (t) => HAS_ARABIC.test(t) && /[0-9]/.test(t) ? t.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : t
+  },
+  { rule: "arabizi", apply: (t) => HAS_ARABIC.test(t) ? t : foldArabizi(t) }
+];
+function normalizeSteps(text, skip = -1) {
+  return NORMALIZE_STEPS.reduce((t, step, i) => i === skip ? t : step.apply(t), text);
+}
 var IraqiBadWordsFilter = class {
   constructor(options = {}) {
-    const { customWords = [], includeInsults = false, stripSuffixes = false } = Array.isArray(options) ? { customWords: options } : options;
+    // Clean words that would otherwise match after a prefix is stripped
+    // (e.g. بعير "camel" = ب + عير). Also covers their prefixed and suffixed
+    // forms (وبعيرك). Listed in safe-stems.json, plus the safeWords option.
+    this.safeStems = [];
+    const {
+      customWords = [],
+      safeWords = [],
+      includeInsults = false,
+      stripSuffixes = false,
+      dialects = []
+    } = Array.isArray(options) ? { customWords: options } : options;
     this.words = /* @__PURE__ */ new Map();
     this.stripSuffixes = stripSuffixes;
-    this.safeStems = safe_stems_default.map((word) => ({ normalized: this.normalize(word), word }));
+    this.mosul = dialects.includes("mosul");
+    this.addSafeStems(safe_stems_default);
+    this.addSafeStems(safeWords);
     this.addEntries(words_default, "words");
     if (includeInsults) this.addEntries(insults_default, "insults");
     this.addEntries(customWords, "custom");
   }
   /**
-   * Normalizes Arabic text so spelling variants compare equal:
-   * removes diacritics (Tashkeel) and tatweel, unifies Alef/Teh Marbuta/Alef Maksura
-   * forms, and collapses stretched letters (عاااارة -> عارة).
+   * Normalizes a word or token for comparison: removes diacritics, tatweel and
+   * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
+   * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
+   * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
    */
-  normalizeArabic(text) {
-    return text.replace(/[ً-ٰٟـ]/g, "").replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A").replace(/(.)\1+/gu, "$1").toLowerCase();
+  normalize(text) {
+    return normalizeSteps(text);
   }
   /**
-   * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
-   * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
-   * teezak / tizak. Only words made purely of Latin letters and digits are
-   * folded; expects text already passed through normalizeArabic.
+   * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
+   * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
    */
-  foldArabizi(word) {
-    if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
-    return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
+  readings(word) {
+    return word.includes("\u06AF") ? [word.replace(/گ/g, "\u0642"), word.replace(/گ/g, "\u0643")] : [word];
   }
-  /** Normalizes a word or token for comparison. */
-  normalize(text) {
-    return this.foldArabizi(this.normalizeArabic(text));
+  /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
+  dialectForms(word) {
+    return this.mosul && word.includes("\u0631") ? [word, word.replace(/ر/g, "\u063A")] : [word];
+  }
+  /** The key a list word is stored under. A گ in a list entry means ق. */
+  entryKey(word) {
+    return this.normalize(word).replace(/گ/g, "\u0642");
+  }
+  /**
+   * A word list with surrounding whitespace trimmed and entries that are empty
+   * after normalization skipped. A stray "" in a user's file would otherwise
+   * match a lone tatweel as a word, or clear every match as a safe stem.
+   */
+  cleanList(list) {
+    return list.filter((word) => typeof word === "string").map((word) => word.trim()).filter((word) => this.entryKey(word) !== "");
+  }
+  /**
+   * Adds safe stems from one list. A safe stem written with گ protects only
+   * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
+   */
+  addSafeStems(stems) {
+    for (const stem of this.cleanList(stems)) {
+      for (const word of this.dialectForms(stem)) {
+        this.safeStems.push({ normalized: this.normalize(word), word });
+      }
+    }
   }
   /**
    * Adds words from one list. A word already present keeps its first source.
    */
-  addEntries(newWords, source) {
-    for (const word of newWords) {
-      const normalized = this.normalize(word);
-      if (!this.words.has(normalized)) this.words.set(normalized, { word, source });
+  addEntries(listWords, source) {
+    const newWords = this.cleanList(listWords);
+    const forms = [
+      ...newWords.map((word) => ({ word, form: word })),
+      ...newWords.flatMap((word) => this.dialectForms(word).slice(1).map((form) => ({ word, form })))
+    ];
+    for (const { word, form } of forms) {
+      const key = this.entryKey(form);
+      if (!this.words.has(key)) this.words.set(key, { word, form, source });
     }
   }
   /**
@@ -265,11 +343,44 @@ var IraqiBadWordsFilter = class {
    * A match cleared by a safe stem is returned with `suppressedBy` set.
    */
   matchWord(token) {
-    const word = this.normalize(token);
-    const direct = this.words.get(word);
-    if (direct) {
-      return { normalized: word, entry: direct.word, source: direct.source, prefix: "", suffix: "" };
+    const spelled = this.normalize(token);
+    const spelledSafe = spelled.includes("\u06AF") ? this.findSafe(this.prefixCandidates(spelled)) : void 0;
+    let suppressed = null;
+    for (const reading of this.readings(spelled)) {
+      const found = this.matchReading(reading);
+      if (!found) continue;
+      const { form, ...rest } = found;
+      const match = { ...rest, rules: this.explain(token, found, spelled, reading) };
+      if (match.suppressedBy === void 0 && spelledSafe) match.suppressedBy = spelledSafe;
+      if (match.suppressedBy === void 0) return match;
+      suppressed = suppressed ?? match;
+    }
+    return suppressed;
+  }
+  /**
+   * Works out which rules a match needed, by comparing the typed word with the
+   * listed spelling (plus any stripped prefix and suffix). A normalization rule
+   * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
+   * "stretched" but not "spelling", since both have ة.
+   */
+  explain(token, found, spelled, reading) {
+    const listed = found.prefix + found.form + found.suffix;
+    const gaf = reading === spelled ? "" : reading === spelled.replace(/گ/g, "\u0642") ? "\u0642" : "\u0643";
+    const same = (typed, list) => typed.replace(/گ/g, gaf || "\u06AF") === list.replace(/گ/g, "\u0642");
+    const rules = [];
+    NORMALIZE_STEPS.forEach(({ rule }, i) => {
+      if (rule && !same(normalizeSteps(token, i), normalizeSteps(listed, i))) rules.push(rule);
+    });
+    if (gaf && normalizeSteps(token) !== normalizeSteps(listed)) {
+      rules.push(gaf === "\u0642" ? "gaf-as-qaf" : "gaf-as-kaf");
     }
+    if (found.prefix) rules.push("prefix");
+    if (found.suffix) rules.push("suffix");
+    if (found.form !== found.entry) rules.push("mosul");
+    return rules;
+  }
+  /** The word plus each form of it with an attached prefix removed. */
+  prefixCandidates(word) {
     const candidates = [{ prefix: "", stem: word, suffix: "" }];
     for (const conjunction of CONJUNCTIONS) {
       for (const preposition of PREPOSITIONS) {
@@ -281,6 +392,30 @@ var IraqiBadWordsFilter = class {
         }
       }
     }
+    return candidates;
+  }
+  /** The first safe stem one of the candidates starts with, as listed. */
+  findSafe(candidates) {
+    for (const { stem } of candidates) {
+      const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
+      if (safe) return safe.word;
+    }
+    return void 0;
+  }
+  /** Matches one normalized reading of a word. */
+  matchReading(word) {
+    const direct = this.words.get(word);
+    if (direct) {
+      return {
+        normalized: word,
+        entry: direct.word,
+        form: direct.form,
+        source: direct.source,
+        prefix: "",
+        suffix: ""
+      };
+    }
+    const candidates = this.prefixCandidates(word);
     if (this.stripSuffixes) {
       for (const { prefix, stem } of [...candidates]) {
         for (const suffix of SUFFIXES) {
@@ -292,34 +427,55 @@ var IraqiBadWordsFilter = class {
     }
     const match = candidates.find(({ stem }) => this.words.has(stem));
     if (!match) return null;
-    const { word: entry, source } = this.words.get(match.stem);
+    const { word: entry, form, source } = this.words.get(match.stem);
     const result = {
       normalized: word,
       entry,
+      form,
       source,
       prefix: match.prefix,
       suffix: match.suffix
     };
-    for (const { stem } of candidates) {
-      const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
-      if (safe) return { ...result, suppressedBy: safe.word };
-    }
-    return result;
+    const safe = this.findSafe(candidates);
+    return safe ? { ...result, suppressedBy: safe } : result;
   }
-  /** True when the word is profane, i.e. matched and not cleared by a safe stem. */
-  isProfaneWord(token) {
-    const match = this.matchWord(token);
-    return match !== null && match.suppressedBy === void 0;
+  /**
+   * Finds every match in the text, including ones cleared by a safe stem.
+   * A token with invisible characters is matched whole first (عا‌رة -> عارة);
+   * if that finds nothing profane, each visible piece is tried on its own, so an
+   * invisible character between two words can't hide a bad word either.
+   */
+  *findMatches(text) {
+    const pattern = new RegExp(WORD_PATTERN.source, "g");
+    let found;
+    while (found = pattern.exec(text)) {
+      const token = found[0];
+      const start = found.index;
+      const whole = this.matchWord(token);
+      const wholeMatch = whole && { token, start, end: start + token.length, ...whole };
+      if (whole && whole.suppressedBy === void 0 || !HAS_INVISIBLE.test(token)) {
+        if (wholeMatch) yield wholeMatch;
+        continue;
+      }
+      const pieces = [];
+      const piecePattern = new RegExp(VISIBLE_RUN.source, "g");
+      let piece;
+      while (piece = piecePattern.exec(token)) {
+        const match = this.matchWord(piece[0]);
+        const pieceStart = start + piece.index;
+        if (match) pieces.push({ token: piece[0], start: pieceStart, end: pieceStart + piece[0].length, ...match });
+      }
+      if (pieces.length) yield* pieces;
+      else if (wholeMatch) yield wholeMatch;
+    }
   }
   /**
    * Checks if a given text contains profane words.
    */
   isProfane(text) {
     if (!text) return false;
-    for (const token of text.match(WORD_PATTERN) ?? []) {
-      if (this.isProfaneWord(token)) {
-        return true;
-      }
+    for (const match of this.findMatches(text)) {
+      if (match.suppressedBy === void 0) return true;
     }
     return false;
   }
@@ -330,14 +486,8 @@ var IraqiBadWordsFilter = class {
   inspect(text, options = {}) {
     if (!text) return [];
     const matches = [];
-    const pattern = new RegExp(WORD_PATTERN.source, "g");
-    let found;
-    while (found = pattern.exec(text)) {
-      const token = found[0];
-      const match = this.matchWord(token);
-      if (match && (match.suppressedBy === void 0 || options.includeSuppressed)) {
-        matches.push({ token, start: found.index, end: found.index + token.length, ...match });
-      }
+    for (const match of this.findMatches(text)) {
+      if (match.suppressedBy === void 0 || options.includeSuppressed) matches.push(match);
     }
     return matches;
   }
@@ -360,11 +510,19 @@ var IraqiBadWordsFilter = class {
   addWords(newWords) {
     this.addEntries(newWords, "custom");
   }
+  /**
+   * Add safe stems to this filter, like the safeWords option.
+   */
+  addSafeWords(newWords) {
+    this.addSafeStems(newWords);
+  }
   /**
    * Remove words from the active filter dictionary.
    */
   removeWords(wordsToRemove) {
-    wordsToRemove.forEach((word) => this.words.delete(this.normalize(word)));
+    for (const word of wordsToRemove.flatMap((w) => this.dialectForms(w))) {
+      this.words.delete(this.entryKey(word));
+    }
   }
 };
 var index_default = IraqiBadWordsFilter;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "iraqi-dialect-bad-words",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "An Iraqi dialect profanity filter for React, React Native Expo, and Node.js",
   "main": "./dist/index.js",
   "module": "./dist/index.mjs",