iraqi-dialect-bad-words 1.2.1 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -65,15 +65,44 @@ Creates a filter with the built-in word list.
65
65
  new IraqiBadWordsFilter(); // profanity only
66
66
  new IraqiBadWordsFilter({ includeInsults: true }); // also milder insults
67
67
  new IraqiBadWordsFilter({ stripSuffixes: true }); // also unlisted suffix forms
68
+ new IraqiBadWordsFilter({ dialects: ['mosul'] }); // also Mosul ر → غ forms
68
69
  new IraqiBadWordsFilter({ customWords: ['word1', 'word2'] });
70
+ new IraqiBadWordsFilter({ safeWords: ['word3'] }); // clean words not to flag
69
71
  new IraqiBadWordsFilter(['word1', 'word2']); // same as customWords
70
72
  ```
71
73
 
72
74
  | Option | Default | Description |
73
75
  |---|---|---|
74
- | `customWords` | `[]` | Extra words to filter. |
76
+ | `customWords` | `[]` | Extra words to filter, on top of the built-in list. Can come from your own JSON file (see below). |
77
+ | `safeWords` | `[]` | Extra safe stems, on top of the built-in [`safe-stems.json`](src/safe-stems.json): clean words that would otherwise match once a prefix or suffix is stripped, like the built-in `بعير` ("camel" = `ب` + `عير`). Use it for clean words your app sees flagged, especially ones your own `customWords` cause. Their prefixed and suffixed forms are safe too (`وبعيرك`). Can come from your own JSON file (see below). |
75
78
  | `includeInsults` | `false` | Also filter milder insults like كلب, حمار, قندرة, نعال. Off by default because many of them are everyday words, so turning it on flags sentences like `عندي كلب` ("I have a dog"). |
76
79
  | `stripSuffixes` | `false` | Also match words with a pronoun suffix (ي ك ج ه ها نا كم هم) that the list doesn't spell out, e.g. `طيزهم` → `طيز`. Off by default because it widens matching: a clean word that ends like a bad word plus a suffix gets flagged too. |
80
+ | `dialects` | `[]` | Regional pronunciations to also match. `'mosul'`: ر said as غ, so every entry with ر also matches its غ form (`شرموطة` → `شغموطة`); the ر forms keep matching. Safe stems get their غ form too. |
81
+
82
+ #### Using your own word files
83
+
84
+ Keep your app's extra words, and clean words you don't want flagged, in JSON files: arrays of single words, the same format as [`src/words.json`](src/words.json) and [`src/safe-stems.json`](src/safe-stems.json).
85
+
86
+ ```json
87
+ ["زفت", "نذل"]
88
+ ```
89
+
90
+ Import them and pass them as `customWords` and `safeWords`. The built-in lists stay on, and your words are added to them:
91
+
92
+ ```ts
93
+ import IraqiBadWordsFilter from 'iraqi-dialect-bad-words';
94
+ import myWords from './my-words.json'; // TypeScript needs "resolveJsonModule": true
95
+ import mySafeWords from './my-safe-words.json';
96
+ // CommonJS: const myWords = require('./my-words.json');
97
+
98
+ const filter = new IraqiBadWordsFilter({ customWords: myWords, safeWords: mySafeWords });
99
+ ```
100
+
101
+ Your words go through the same normalization as the built-in lists, so you don't need to list diacritic, ة/ه, stretched or prefixed spellings. Whitespace around a word is trimmed and blank entries are skipped. `inspect()` reports a match from your word file with `source: 'custom'`, and a match cleared by your safe words with `suppressedBy` set to that safe word.
102
+
103
+ A safe word only clears a match that needed a prefix or suffix stripped, like the built-in safe stems. To stop a listed word itself from being flagged, use `removeWords()`.
104
+
105
+ React Native, Expo, Vite and Next.js bundle the JSON files into your app, so this works the same in each. The filter doesn't read files at runtime. If your lists come from a server, fetch them and pass the arrays to `addWords()` and `addSafeWords()`.
77
106
 
78
107
  ### `isProfane(text: string): boolean`
79
108
 
@@ -89,34 +118,55 @@ Lists each bad word in the text and explains why it matched. Use it to highlight
89
118
 
90
119
  ```ts
91
120
  filter.inspect('روح للكواد!');
92
- // [{ token: 'للكواد', start: 4, end: 10, normalized: 'لكواد',
93
- // entry: 'كواد', source: 'words', prefix: 'ل' }]
121
+ // [{ token: 'للكواد', start: 4, end: 10, normalized: 'لكواد', entry: 'كواد',
122
+ // source: 'words', prefix: 'ل', suffix: '', rules: ['stretched', 'prefix'] }]
94
123
  ```
95
124
 
96
125
  | Field | Description |
97
126
  |---|---|
98
127
  | `token` | The word exactly as written. |
99
128
  | `start`, `end` | Offsets in the **original** text, so `text.slice(start, end) === token`. Counted in JavaScript string units (UTF-16); convert them if another platform counts characters differently. |
100
- | `normalized` | The word after normalization (Arabizi is folded too, so `Qa7ba` shows as `gahba`). |
101
- | `entry` | The list entry it matched, spelled as in the list. |
129
+ | `normalized` | The word after normalization (Arabizi is folded too, so `Qa7ba` shows as `gahba`). For a word with گ, the reading that matched (ق or ك). |
130
+ | `entry` | The list entry it matched, spelled as in the list. For a Mosul form, the listed ر spelling. |
102
131
  | `source` | `'words'`, `'insults'` or `'custom'`. A word on more than one list keeps the first: built-in, then insults, then custom. |
103
132
  | `prefix` | The attached prefix stripped before matching (after normalization, so `لل` shows as `ل`), or `''` for a direct match. |
104
133
  | `suffix` | The suffix stripped before matching (only with `stripSuffixes`), or `''`. |
134
+ | `rules` | The rules the word needed to match the entry (see below). Empty when it's spelled exactly like the entry. |
105
135
  | `suppressedBy` | Only with `includeSuppressed`: the safe stem that cleared this match. |
106
136
 
107
137
  Pass `{ includeSuppressed: true }` to also see words that matched but were cleared by a safe stem. They are not profane (`isProfane` and `clean` ignore them), but they show why a word *wasn't* flagged:
108
138
 
109
139
  ```ts
110
140
  filter.inspect('وبعيرك', { includeSuppressed: true });
111
- // [{ token: 'وبعيرك', …, entry: 'عيرك', prefix: 'وب', suppressedBy: 'بعير' }]
141
+ // [{ token: 'وبعيرك', …, entry: 'عيرك', prefix: 'وب', rules: ['prefix'], suppressedBy: 'بعير' }]
112
142
  ```
113
143
 
114
- Including the output of `inspect` in a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml) shows exactly which entry and prefix rule fired.
144
+ `rules` lists only the rules that made a difference between the typed word and the entry, so a report shows straight away why something matched:
145
+
146
+ | Rule | Example |
147
+ |---|---|
148
+ | `diacritics` | `عَارَة`, `عـــارة` |
149
+ | `invisible` | `عا‌رة` (zero-width character inside) |
150
+ | `spelling` | `عاره` (ة/ه, ى/ي, أ/ا) |
151
+ | `keyboard` | `عیر`, `کس` (Persian/Kurdish letters) |
152
+ | `arabic-digits` | `ga٧ba` |
153
+ | `stretched` | `عاااارة`, `للكواد` |
154
+ | `glued-digits` | `عارة3` |
155
+ | `arabizi` | `qa7ba` → `gahba` |
156
+ | `gaf-as-qaf` / `gaf-as-kaf` | `گحبة` → قحبة, `گسها` → كسها |
157
+ | `prefix` / `suffix` | `بطيزك`, `طيزهم` (with `stripSuffixes`) |
158
+ | `mosul` | `شغموطة` → شرموطة (with `dialects: ['mosul']`) |
159
+
160
+ Including the output of `inspect` in a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml) shows exactly which entry matched and which rules it needed; the issue form has a field for it.
115
161
 
116
162
  ### `addWords(words: string[]): void`
117
163
 
118
164
  Adds words to this filter.
119
165
 
166
+ ### `addSafeWords(words: string[]): void`
167
+
168
+ Adds safe stems to this filter, like the `safeWords` option.
169
+
120
170
  ### `removeWords(words: string[]): void`
121
171
 
122
172
  Removes words from this filter, for example a built-in word you don't want flagged in your app.
@@ -132,6 +182,7 @@ Matching is done per whole word, after normalizing both the text and the word li
132
182
  | Tatweel | `عـــارة` |
133
183
  | ة / ه, ى / ي, أ / إ / آ / ا | `عاره` |
134
184
  | Persian/Kurdish keyboard letters ی ک ە | `عیر`, `کس` |
185
+ | گ read as ق (Iraqi) and as ك (Kurdish typists) | `گحبة` = `قحبة`, `گس` = `كس` |
135
186
  | Invisible characters hidden inside a word (zero-width, direction marks, soft hyphen) | `عا‌رة` |
136
187
  | Attached prefixes و ف ب ل ك ال | `والعارة`, `للعارة` |
137
188
  | Pronoun suffixes (with `stripSuffixes`) | `طيزهم`, `وطيزكم` |
@@ -154,6 +205,7 @@ The word list is the heart of this package, and native speakers are the best jud
154
205
  - **A word is missing:** open a [“Suggest a word” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=add-word.yml), or add it to [`src/words.json`](src/words.json) in a pull request.
155
206
  - **A clean word is flagged:** open a [“Clean word flagged” issue](https://github.com/karam-fikre/iraqi-dialect-bad-words/issues/new?template=false-positive.yml).
156
207
  If it's flagged only because of a prefix (e.g. `بعير` "camel" = `ب` + a bad word), the fix is adding it to [`src/safe-stems.json`](src/safe-stems.json).
208
+ A safe stem written with گ protects only words spelled with گ, so `گلب` ("heart", Iraqi spelling of قلب) can be safe while `كلب` is not.
157
209
 
158
210
  - **A sentence is judged wrong:** add it to [`test/fixtures/sentences.json`](test/fixtures/sentences.json) with the verdict a native speaker expects (`"profane": true` or `false`, plus `"options"` if it needs `includeInsults` or `stripSuffixes`). Every sentence there runs as a test, so later word-list changes can't silently flip it.
159
211
 
package/dist/index.d.mts CHANGED
@@ -1,6 +1,22 @@
1
+ /**
2
+ * A rule a word needed to match its list entry, reported in ProfanityMatch.rules.
3
+ * Only rules that made a difference are reported: عارة matching عارة has none.
4
+ */
5
+ type MatchRule = "diacritics" | "invisible" | "spelling" | "keyboard" | "arabic-digits" | "stretched" | "glued-digits" | "arabizi" | "gaf-as-qaf" | "gaf-as-kaf" | "prefix" | "suffix" | "mosul";
1
6
  interface IraqiBadWordsFilterOptions {
2
- /** Extra words to filter, on top of the built-in list. */
7
+ /**
8
+ * Extra words to filter, on top of the built-in list. Can be your own JSON
9
+ * word file: `customWords: require('./my-words.json')`.
10
+ */
3
11
  customWords?: string[];
12
+ /**
13
+ * Extra safe stems, on top of the built-in safe-stems.json: clean words that
14
+ * would otherwise match once a prefix or suffix is stripped, like the
15
+ * built-in بعير ("camel" = ب + عير). Their prefixed and suffixed forms are
16
+ * safe too (وبعيرك).
17
+ * Can be your own JSON file: `safeWords: require('./my-safe-words.json')`.
18
+ */
19
+ safeWords?: string[];
4
20
  /**
5
21
  * Also filter milder insults (كلب, حمار, قندرة, …). Off by default because
6
22
  * many of them are everyday words ("عندي كلب", "جلب لي هدية").
@@ -12,7 +28,15 @@ interface IraqiBadWordsFilterOptions {
12
28
  * clean words that happen to end like a bad word plus a suffix.
13
29
  */
14
30
  stripSuffixes?: boolean;
31
+ /**
32
+ * Regional pronunciations to also match. "mosul": ر said as غ, so every entry
33
+ * and safe stem with ر also matches its غ form (عير -> عيغ, بعير -> بعيغ).
34
+ * The ر forms keep matching. Off by default.
35
+ */
36
+ dialects?: Dialect[];
15
37
  }
38
+ /** A regional pronunciation the filter can also match. */
39
+ type Dialect = "mosul";
16
40
  /** Which list a dictionary entry came from. */
17
41
  type WordSource = "words" | "insults" | "custom";
18
42
  interface InspectOptions {
@@ -33,9 +57,15 @@ interface ProfanityMatch {
33
57
  start: number;
34
58
  /** Offset just past the end of the word in the input. */
35
59
  end: number;
36
- /** The word after normalization (diacritics, stretched letters, … removed). */
60
+ /**
61
+ * The word after normalization (diacritics, stretched letters, … removed).
62
+ * For a word with گ, the reading that matched: گ as ق or as ك.
63
+ */
37
64
  normalized: string;
38
- /** The dictionary entry it matched, spelled as in its list. */
65
+ /**
66
+ * The dictionary entry it matched, spelled as in its list. For a Mosul form
67
+ * (rule "mosul"), the listed ر spelling: الشرموطة for الشغموطة.
68
+ */
39
69
  entry: string;
40
70
  /** The list the entry came from. */
41
71
  source: WordSource;
@@ -43,8 +73,10 @@ interface ProfanityMatch {
43
73
  prefix: string;
44
74
  /** The suffix stripped before matching (only with stripSuffixes), or "". */
45
75
  suffix: string;
76
+ /** The rules the word needed to match the entry, e.g. ["prefix", "arabizi"]. */
77
+ rules: MatchRule[];
46
78
  /**
47
- * The safe stem that cleared this match, spelled as in safe-stems.json.
79
+ * The safe stem that cleared this match, spelled as in safe-stems.json or safeWords.
48
80
  * Only set on matches returned with `includeSuppressed`; these are not profane.
49
81
  */
50
82
  suppressedBy?: string;
@@ -53,24 +85,35 @@ declare class IraqiBadWordsFilter {
53
85
  private words;
54
86
  private safeStems;
55
87
  private stripSuffixes;
88
+ private mosul;
56
89
  constructor(options?: string[] | IraqiBadWordsFilterOptions);
57
90
  /**
58
- * Normalizes Arabic text so spelling variants compare equal:
59
- * removes diacritics (Tashkeel), tatweel and invisible characters, unifies
60
- * Alef/Teh Marbuta/Alef Maksura forms and the Persian/Kurdish keyboard letters
61
- * ی ک ە, turns Arabic-Indic digits into 0-9, and collapses stretched letters
62
- * (عاااارة -> عارة).
91
+ * Normalizes a word or token for comparison: removes diacritics, tatweel and
92
+ * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
93
+ * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
94
+ * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
63
95
  */
64
- private normalizeArabic;
96
+ private normalize;
65
97
  /**
66
- * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
67
- * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
68
- * teezak / tizak. Only words made purely of Latin letters and digits are
69
- * folded; expects text already passed through normalizeArabic.
98
+ * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
99
+ * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
70
100
  */
71
- private foldArabizi;
72
- /** Normalizes a word or token for comparison. */
73
- private normalize;
101
+ private readings;
102
+ /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
103
+ private dialectForms;
104
+ /** The key a list word is stored under. A گ in a list entry means ق. */
105
+ private entryKey;
106
+ /**
107
+ * A word list with surrounding whitespace trimmed and entries that are empty
108
+ * after normalization skipped. A stray "" in a user's file would otherwise
109
+ * match a lone tatweel as a word, or clear every match as a safe stem.
110
+ */
111
+ private cleanList;
112
+ /**
113
+ * Adds safe stems from one list. A safe stem written with گ protects only
114
+ * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
115
+ */
116
+ private addSafeStems;
74
117
  /**
75
118
  * Adds words from one list. A word already present keeps its first source.
76
119
  */
@@ -82,6 +125,19 @@ declare class IraqiBadWordsFilter {
82
125
  * A match cleared by a safe stem is returned with `suppressedBy` set.
83
126
  */
84
127
  private matchWord;
128
+ /**
129
+ * Works out which rules a match needed, by comparing the typed word with the
130
+ * listed spelling (plus any stripped prefix and suffix). A normalization rule
131
+ * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
132
+ * "stretched" but not "spelling", since both have ة.
133
+ */
134
+ private explain;
135
+ /** The word plus each form of it with an attached prefix removed. */
136
+ private prefixCandidates;
137
+ /** The first safe stem one of the candidates starts with, as listed. */
138
+ private findSafe;
139
+ /** Matches one normalized reading of a word. */
140
+ private matchReading;
85
141
  /**
86
142
  * Finds every match in the text, including ones cleared by a safe stem.
87
143
  * A token with invisible characters is matched whole first (عا‌رة -> عارة);
@@ -106,10 +162,14 @@ declare class IraqiBadWordsFilter {
106
162
  * Dynamically add words to the active dictionary.
107
163
  */
108
164
  addWords(newWords: string[]): void;
165
+ /**
166
+ * Add safe stems to this filter, like the safeWords option.
167
+ */
168
+ addSafeWords(newWords: string[]): void;
109
169
  /**
110
170
  * Remove words from the active filter dictionary.
111
171
  */
112
172
  removeWords(wordsToRemove: string[]): void;
113
173
  }
114
174
 
115
- export { type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
175
+ export { type Dialect, type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type MatchRule, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
package/dist/index.d.ts CHANGED
@@ -1,6 +1,22 @@
1
+ /**
2
+ * A rule a word needed to match its list entry, reported in ProfanityMatch.rules.
3
+ * Only rules that made a difference are reported: عارة matching عارة has none.
4
+ */
5
+ type MatchRule = "diacritics" | "invisible" | "spelling" | "keyboard" | "arabic-digits" | "stretched" | "glued-digits" | "arabizi" | "gaf-as-qaf" | "gaf-as-kaf" | "prefix" | "suffix" | "mosul";
1
6
  interface IraqiBadWordsFilterOptions {
2
- /** Extra words to filter, on top of the built-in list. */
7
+ /**
8
+ * Extra words to filter, on top of the built-in list. Can be your own JSON
9
+ * word file: `customWords: require('./my-words.json')`.
10
+ */
3
11
  customWords?: string[];
12
+ /**
13
+ * Extra safe stems, on top of the built-in safe-stems.json: clean words that
14
+ * would otherwise match once a prefix or suffix is stripped, like the
15
+ * built-in بعير ("camel" = ب + عير). Their prefixed and suffixed forms are
16
+ * safe too (وبعيرك).
17
+ * Can be your own JSON file: `safeWords: require('./my-safe-words.json')`.
18
+ */
19
+ safeWords?: string[];
4
20
  /**
5
21
  * Also filter milder insults (كلب, حمار, قندرة, …). Off by default because
6
22
  * many of them are everyday words ("عندي كلب", "جلب لي هدية").
@@ -12,7 +28,15 @@ interface IraqiBadWordsFilterOptions {
12
28
  * clean words that happen to end like a bad word plus a suffix.
13
29
  */
14
30
  stripSuffixes?: boolean;
31
+ /**
32
+ * Regional pronunciations to also match. "mosul": ر said as غ, so every entry
33
+ * and safe stem with ر also matches its غ form (عير -> عيغ, بعير -> بعيغ).
34
+ * The ر forms keep matching. Off by default.
35
+ */
36
+ dialects?: Dialect[];
15
37
  }
38
+ /** A regional pronunciation the filter can also match. */
39
+ type Dialect = "mosul";
16
40
  /** Which list a dictionary entry came from. */
17
41
  type WordSource = "words" | "insults" | "custom";
18
42
  interface InspectOptions {
@@ -33,9 +57,15 @@ interface ProfanityMatch {
33
57
  start: number;
34
58
  /** Offset just past the end of the word in the input. */
35
59
  end: number;
36
- /** The word after normalization (diacritics, stretched letters, … removed). */
60
+ /**
61
+ * The word after normalization (diacritics, stretched letters, … removed).
62
+ * For a word with گ, the reading that matched: گ as ق or as ك.
63
+ */
37
64
  normalized: string;
38
- /** The dictionary entry it matched, spelled as in its list. */
65
+ /**
66
+ * The dictionary entry it matched, spelled as in its list. For a Mosul form
67
+ * (rule "mosul"), the listed ر spelling: الشرموطة for الشغموطة.
68
+ */
39
69
  entry: string;
40
70
  /** The list the entry came from. */
41
71
  source: WordSource;
@@ -43,8 +73,10 @@ interface ProfanityMatch {
43
73
  prefix: string;
44
74
  /** The suffix stripped before matching (only with stripSuffixes), or "". */
45
75
  suffix: string;
76
+ /** The rules the word needed to match the entry, e.g. ["prefix", "arabizi"]. */
77
+ rules: MatchRule[];
46
78
  /**
47
- * The safe stem that cleared this match, spelled as in safe-stems.json.
79
+ * The safe stem that cleared this match, spelled as in safe-stems.json or safeWords.
48
80
  * Only set on matches returned with `includeSuppressed`; these are not profane.
49
81
  */
50
82
  suppressedBy?: string;
@@ -53,24 +85,35 @@ declare class IraqiBadWordsFilter {
53
85
  private words;
54
86
  private safeStems;
55
87
  private stripSuffixes;
88
+ private mosul;
56
89
  constructor(options?: string[] | IraqiBadWordsFilterOptions);
57
90
  /**
58
- * Normalizes Arabic text so spelling variants compare equal:
59
- * removes diacritics (Tashkeel), tatweel and invisible characters, unifies
60
- * Alef/Teh Marbuta/Alef Maksura forms and the Persian/Kurdish keyboard letters
61
- * ی ک ە, turns Arabic-Indic digits into 0-9, and collapses stretched letters
62
- * (عاااارة -> عارة).
91
+ * Normalizes a word or token for comparison: removes diacritics, tatweel and
92
+ * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
93
+ * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
94
+ * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
63
95
  */
64
- private normalizeArabic;
96
+ private normalize;
65
97
  /**
66
- * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
67
- * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
68
- * teezak / tizak. Only words made purely of Latin letters and digits are
69
- * folded; expects text already passed through normalizeArabic.
98
+ * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
99
+ * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
70
100
  */
71
- private foldArabizi;
72
- /** Normalizes a word or token for comparison. */
73
- private normalize;
101
+ private readings;
102
+ /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
103
+ private dialectForms;
104
+ /** The key a list word is stored under. A گ in a list entry means ق. */
105
+ private entryKey;
106
+ /**
107
+ * A word list with surrounding whitespace trimmed and entries that are empty
108
+ * after normalization skipped. A stray "" in a user's file would otherwise
109
+ * match a lone tatweel as a word, or clear every match as a safe stem.
110
+ */
111
+ private cleanList;
112
+ /**
113
+ * Adds safe stems from one list. A safe stem written with گ protects only
114
+ * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
115
+ */
116
+ private addSafeStems;
74
117
  /**
75
118
  * Adds words from one list. A word already present keeps its first source.
76
119
  */
@@ -82,6 +125,19 @@ declare class IraqiBadWordsFilter {
82
125
  * A match cleared by a safe stem is returned with `suppressedBy` set.
83
126
  */
84
127
  private matchWord;
128
+ /**
129
+ * Works out which rules a match needed, by comparing the typed word with the
130
+ * listed spelling (plus any stripped prefix and suffix). A normalization rule
131
+ * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
132
+ * "stretched" but not "spelling", since both have ة.
133
+ */
134
+ private explain;
135
+ /** The word plus each form of it with an attached prefix removed. */
136
+ private prefixCandidates;
137
+ /** The first safe stem one of the candidates starts with, as listed. */
138
+ private findSafe;
139
+ /** Matches one normalized reading of a word. */
140
+ private matchReading;
85
141
  /**
86
142
  * Finds every match in the text, including ones cleared by a safe stem.
87
143
  * A token with invisible characters is matched whole first (عا‌رة -> عارة);
@@ -106,10 +162,14 @@ declare class IraqiBadWordsFilter {
106
162
  * Dynamically add words to the active dictionary.
107
163
  */
108
164
  addWords(newWords: string[]): void;
165
+ /**
166
+ * Add safe stems to this filter, like the safeWords option.
167
+ */
168
+ addSafeWords(newWords: string[]): void;
109
169
  /**
110
170
  * Remove words from the active filter dictionary.
111
171
  */
112
172
  removeWords(wordsToRemove: string[]): void;
113
173
  }
114
174
 
115
- export { type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
175
+ export { type Dialect, type InspectOptions, IraqiBadWordsFilter, type IraqiBadWordsFilterOptions, type MatchRule, type ProfanityMatch, type WordSource, IraqiBadWordsFilter as default };
package/dist/index.js CHANGED
@@ -233,9 +233,16 @@ var insults_default = [
233
233
  var safe_stems_default = [
234
234
  "\u0628\u0639\u064A\u0631",
235
235
  "\u0628\u0639\u064A\u063A",
236
+ "\u06AF\u0644\u0628",
237
+ "\u06AF\u0644\u0627\u0628",
238
+ "\u0637\u06AF\u0639\u0629",
236
239
  "\u0628\u0643\u0633",
237
240
  "\u0648\u0643\u0633",
238
- "\u0641\u0643\u0633"
241
+ "\u0641\u0643\u0633",
242
+ "\u0628\u062B\u0648\u0631",
243
+ "\u062B\u0648\u0631\u0629",
244
+ "\u062B\u0648\u0631\u064A",
245
+ "\u0628\u0627\u0644\u0639\u062C\u0644"
239
246
  ];
240
247
 
241
248
  // src/index.ts
@@ -251,48 +258,109 @@ var CONJUNCTIONS = ["", "\u0648", "\u0641"];
251
258
  var PREPOSITIONS = ["", "\u0628", "\u0644", "\u0643"];
252
259
  var ARTICLES = ["", "\u0627\u0644"];
253
260
  var SUFFIXES = ["\u064A", "\u0643", "\u062C", "\u0647", "\u0647\u0627", "\u0646\u0627", "\u0643\u0645", "\u0647\u0645"];
261
+ var HAS_ARABIC = /[\u0600-\uFEFF]/;
262
+ function foldArabizi(word) {
263
+ if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
264
+ return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
265
+ }
266
+ var NORMALIZE_STEPS = [
267
+ { rule: "diacritics", apply: (t) => t.replace(/[\u064B-\u065F\u0670\u0640]/g, "") },
268
+ { rule: "invisible", apply: (t) => t.replace(INVISIBLE_CHARS, "") },
269
+ { rule: "spelling", apply: (t) => t.replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A") },
270
+ { rule: "keyboard", apply: (t) => t.replace(/[ەہۀ]/g, "\u0647").replace(/ی/g, "\u064A").replace(/[کڪ]/g, "\u0643") },
271
+ {
272
+ rule: "arabic-digits",
273
+ apply: (t) => t.replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776))
274
+ },
275
+ { rule: "stretched", apply: (t) => t.replace(/(.)\1+/gu, "$1") },
276
+ { rule: null, apply: (t) => t.toLowerCase() },
277
+ // Digits glued to an Arabic word (عارة3, عا٣رة) are padding; in Arabizi they're letters
278
+ {
279
+ rule: "glued-digits",
280
+ apply: (t) => HAS_ARABIC.test(t) && /[0-9]/.test(t) ? t.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : t
281
+ },
282
+ { rule: "arabizi", apply: (t) => HAS_ARABIC.test(t) ? t : foldArabizi(t) }
283
+ ];
284
+ function normalizeSteps(text, skip = -1) {
285
+ return NORMALIZE_STEPS.reduce((t, step, i) => i === skip ? t : step.apply(t), text);
286
+ }
254
287
  var IraqiBadWordsFilter = class {
255
288
  constructor(options = {}) {
256
- const { customWords = [], includeInsults = false, stripSuffixes = false } = Array.isArray(options) ? { customWords: options } : options;
289
+ // Clean words that would otherwise match after a prefix is stripped
290
+ // (e.g. بعير "camel" = ب + عير). Also covers their prefixed and suffixed
291
+ // forms (وبعيرك). Listed in safe-stems.json, plus the safeWords option.
292
+ this.safeStems = [];
293
+ const {
294
+ customWords = [],
295
+ safeWords = [],
296
+ includeInsults = false,
297
+ stripSuffixes = false,
298
+ dialects = []
299
+ } = Array.isArray(options) ? { customWords: options } : options;
257
300
  this.words = /* @__PURE__ */ new Map();
258
301
  this.stripSuffixes = stripSuffixes;
259
- this.safeStems = safe_stems_default.map((word) => ({ normalized: this.normalize(word), word }));
302
+ this.mosul = dialects.includes("mosul");
303
+ this.addSafeStems(safe_stems_default);
304
+ this.addSafeStems(safeWords);
260
305
  this.addEntries(words_default, "words");
261
306
  if (includeInsults) this.addEntries(insults_default, "insults");
262
307
  this.addEntries(customWords, "custom");
263
308
  }
264
309
  /**
265
- * Normalizes Arabic text so spelling variants compare equal:
266
- * removes diacritics (Tashkeel), tatweel and invisible characters, unifies
267
- * Alef/Teh Marbuta/Alef Maksura forms and the Persian/Kurdish keyboard letters
268
- * ی ک ە, turns Arabic-Indic digits into 0-9, and collapses stretched letters
269
- * (عاااارة -> عارة).
310
+ * Normalizes a word or token for comparison: removes diacritics, tatweel and
311
+ * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
312
+ * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
313
+ * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
270
314
  */
271
- normalizeArabic(text) {
272
- return text.replace(/[\u064B-\u065F\u0670\u0640]/g, "").replace(INVISIBLE_CHARS, "").replace(/[أإآٱ]/g, "\u0627").replace(/[ةەہۀ]/g, "\u0647").replace(/[ىی]/g, "\u064A").replace(/[کڪ]/g, "\u0643").replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776)).replace(/(.)\1+/gu, "$1").toLowerCase();
315
+ normalize(text) {
316
+ return normalizeSteps(text);
273
317
  }
274
318
  /**
275
- * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
276
- * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
277
- * teezak / tizak. Only words made purely of Latin letters and digits are
278
- * folded; expects text already passed through normalizeArabic.
319
+ * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
320
+ * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
279
321
  */
280
- foldArabizi(word) {
281
- if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
282
- return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
322
+ readings(word) {
323
+ return word.includes("\u06AF") ? [word.replace(/گ/g, "\u0642"), word.replace(/گ/g, "\u0643")] : [word];
283
324
  }
284
- /** Normalizes a word or token for comparison. */
285
- normalize(text) {
286
- const word = this.normalizeArabic(text);
287
- return /[\u0600-\uFEFF]/.test(word) ? word.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : this.foldArabizi(word);
325
+ /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
326
+ dialectForms(word) {
327
+ return this.mosul && word.includes("\u0631") ? [word, word.replace(/ر/g, "\u063A")] : [word];
328
+ }
329
+ /** The key a list word is stored under. A گ in a list entry means ق. */
330
+ entryKey(word) {
331
+ return this.normalize(word).replace(/گ/g, "\u0642");
332
+ }
333
+ /**
334
+ * A word list with surrounding whitespace trimmed and entries that are empty
335
+ * after normalization skipped. A stray "" in a user's file would otherwise
336
+ * match a lone tatweel as a word, or clear every match as a safe stem.
337
+ */
338
+ cleanList(list) {
339
+ return list.filter((word) => typeof word === "string").map((word) => word.trim()).filter((word) => this.entryKey(word) !== "");
340
+ }
341
+ /**
342
+ * Adds safe stems from one list. A safe stem written with گ protects only
343
+ * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
344
+ */
345
+ addSafeStems(stems) {
346
+ for (const stem of this.cleanList(stems)) {
347
+ for (const word of this.dialectForms(stem)) {
348
+ this.safeStems.push({ normalized: this.normalize(word), word });
349
+ }
350
+ }
288
351
  }
289
352
  /**
290
353
  * Adds words from one list. A word already present keeps its first source.
291
354
  */
292
- addEntries(newWords, source) {
293
- for (const word of newWords) {
294
- const normalized = this.normalize(word);
295
- if (!this.words.has(normalized)) this.words.set(normalized, { word, source });
355
+ addEntries(listWords, source) {
356
+ const newWords = this.cleanList(listWords);
357
+ const forms = [
358
+ ...newWords.map((word) => ({ word, form: word })),
359
+ ...newWords.flatMap((word) => this.dialectForms(word).slice(1).map((form) => ({ word, form })))
360
+ ];
361
+ for (const { word, form } of forms) {
362
+ const key = this.entryKey(form);
363
+ if (!this.words.has(key)) this.words.set(key, { word, form, source });
296
364
  }
297
365
  }
298
366
  /**
@@ -302,11 +370,44 @@ var IraqiBadWordsFilter = class {
302
370
  * A match cleared by a safe stem is returned with `suppressedBy` set.
303
371
  */
304
372
  matchWord(token) {
305
- const word = this.normalize(token);
306
- const direct = this.words.get(word);
307
- if (direct) {
308
- return { normalized: word, entry: direct.word, source: direct.source, prefix: "", suffix: "" };
373
+ const spelled = this.normalize(token);
374
+ const spelledSafe = spelled.includes("\u06AF") ? this.findSafe(this.prefixCandidates(spelled)) : void 0;
375
+ let suppressed = null;
376
+ for (const reading of this.readings(spelled)) {
377
+ const found = this.matchReading(reading);
378
+ if (!found) continue;
379
+ const { form, ...rest } = found;
380
+ const match = { ...rest, rules: this.explain(token, found, spelled, reading) };
381
+ if (match.suppressedBy === void 0 && spelledSafe) match.suppressedBy = spelledSafe;
382
+ if (match.suppressedBy === void 0) return match;
383
+ suppressed = suppressed ?? match;
384
+ }
385
+ return suppressed;
386
+ }
387
+ /**
388
+ * Works out which rules a match needed, by comparing the typed word with the
389
+ * listed spelling (plus any stripped prefix and suffix). A normalization rule
390
+ * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
391
+ * "stretched" but not "spelling", since both have ة.
392
+ */
393
+ explain(token, found, spelled, reading) {
394
+ const listed = found.prefix + found.form + found.suffix;
395
+ const gaf = reading === spelled ? "" : reading === spelled.replace(/گ/g, "\u0642") ? "\u0642" : "\u0643";
396
+ const same = (typed, list) => typed.replace(/گ/g, gaf || "\u06AF") === list.replace(/گ/g, "\u0642");
397
+ const rules = [];
398
+ NORMALIZE_STEPS.forEach(({ rule }, i) => {
399
+ if (rule && !same(normalizeSteps(token, i), normalizeSteps(listed, i))) rules.push(rule);
400
+ });
401
+ if (gaf && normalizeSteps(token) !== normalizeSteps(listed)) {
402
+ rules.push(gaf === "\u0642" ? "gaf-as-qaf" : "gaf-as-kaf");
309
403
  }
404
+ if (found.prefix) rules.push("prefix");
405
+ if (found.suffix) rules.push("suffix");
406
+ if (found.form !== found.entry) rules.push("mosul");
407
+ return rules;
408
+ }
409
+ /** The word plus each form of it with an attached prefix removed. */
410
+ prefixCandidates(word) {
310
411
  const candidates = [{ prefix: "", stem: word, suffix: "" }];
311
412
  for (const conjunction of CONJUNCTIONS) {
312
413
  for (const preposition of PREPOSITIONS) {
@@ -318,6 +419,30 @@ var IraqiBadWordsFilter = class {
318
419
  }
319
420
  }
320
421
  }
422
+ return candidates;
423
+ }
424
+ /** The first safe stem one of the candidates starts with, as listed. */
425
+ findSafe(candidates) {
426
+ for (const { stem } of candidates) {
427
+ const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
428
+ if (safe) return safe.word;
429
+ }
430
+ return void 0;
431
+ }
432
+ /** Matches one normalized reading of a word. */
433
+ matchReading(word) {
434
+ const direct = this.words.get(word);
435
+ if (direct) {
436
+ return {
437
+ normalized: word,
438
+ entry: direct.word,
439
+ form: direct.form,
440
+ source: direct.source,
441
+ prefix: "",
442
+ suffix: ""
443
+ };
444
+ }
445
+ const candidates = this.prefixCandidates(word);
321
446
  if (this.stripSuffixes) {
322
447
  for (const { prefix, stem } of [...candidates]) {
323
448
  for (const suffix of SUFFIXES) {
@@ -329,19 +454,17 @@ var IraqiBadWordsFilter = class {
329
454
  }
330
455
  const match = candidates.find(({ stem }) => this.words.has(stem));
331
456
  if (!match) return null;
332
- const { word: entry, source } = this.words.get(match.stem);
457
+ const { word: entry, form, source } = this.words.get(match.stem);
333
458
  const result = {
334
459
  normalized: word,
335
460
  entry,
461
+ form,
336
462
  source,
337
463
  prefix: match.prefix,
338
464
  suffix: match.suffix
339
465
  };
340
- for (const { stem } of candidates) {
341
- const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
342
- if (safe) return { ...result, suppressedBy: safe.word };
343
- }
344
- return result;
466
+ const safe = this.findSafe(candidates);
467
+ return safe ? { ...result, suppressedBy: safe } : result;
345
468
  }
346
469
  /**
347
470
  * Finds every match in the text, including ones cleared by a safe stem.
@@ -414,11 +537,19 @@ var IraqiBadWordsFilter = class {
414
537
  addWords(newWords) {
415
538
  this.addEntries(newWords, "custom");
416
539
  }
540
+ /**
541
+ * Add safe stems to this filter, like the safeWords option.
542
+ */
543
+ addSafeWords(newWords) {
544
+ this.addSafeStems(newWords);
545
+ }
417
546
  /**
418
547
  * Remove words from the active filter dictionary.
419
548
  */
420
549
  removeWords(wordsToRemove) {
421
- wordsToRemove.forEach((word) => this.words.delete(this.normalize(word)));
550
+ for (const word of wordsToRemove.flatMap((w) => this.dialectForms(w))) {
551
+ this.words.delete(this.entryKey(word));
552
+ }
422
553
  }
423
554
  };
424
555
  var index_default = IraqiBadWordsFilter;
package/dist/index.mjs CHANGED
@@ -206,9 +206,16 @@ var insults_default = [
206
206
  var safe_stems_default = [
207
207
  "\u0628\u0639\u064A\u0631",
208
208
  "\u0628\u0639\u064A\u063A",
209
+ "\u06AF\u0644\u0628",
210
+ "\u06AF\u0644\u0627\u0628",
211
+ "\u0637\u06AF\u0639\u0629",
209
212
  "\u0628\u0643\u0633",
210
213
  "\u0648\u0643\u0633",
211
- "\u0641\u0643\u0633"
214
+ "\u0641\u0643\u0633",
215
+ "\u0628\u062B\u0648\u0631",
216
+ "\u062B\u0648\u0631\u0629",
217
+ "\u062B\u0648\u0631\u064A",
218
+ "\u0628\u0627\u0644\u0639\u062C\u0644"
212
219
  ];
213
220
 
214
221
  // src/index.ts
@@ -224,48 +231,109 @@ var CONJUNCTIONS = ["", "\u0648", "\u0641"];
224
231
  var PREPOSITIONS = ["", "\u0628", "\u0644", "\u0643"];
225
232
  var ARTICLES = ["", "\u0627\u0644"];
226
233
  var SUFFIXES = ["\u064A", "\u0643", "\u062C", "\u0647", "\u0647\u0627", "\u0646\u0627", "\u0643\u0645", "\u0647\u0645"];
234
+ var HAS_ARABIC = /[\u0600-\uFEFF]/;
235
+ function foldArabizi(word) {
236
+ if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
237
+ return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
238
+ }
239
+ var NORMALIZE_STEPS = [
240
+ { rule: "diacritics", apply: (t) => t.replace(/[\u064B-\u065F\u0670\u0640]/g, "") },
241
+ { rule: "invisible", apply: (t) => t.replace(INVISIBLE_CHARS, "") },
242
+ { rule: "spelling", apply: (t) => t.replace(/[أإآٱ]/g, "\u0627").replace(/ة/g, "\u0647").replace(/ى/g, "\u064A") },
243
+ { rule: "keyboard", apply: (t) => t.replace(/[ەہۀ]/g, "\u0647").replace(/ی/g, "\u064A").replace(/[کڪ]/g, "\u0643") },
244
+ {
245
+ rule: "arabic-digits",
246
+ apply: (t) => t.replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776))
247
+ },
248
+ { rule: "stretched", apply: (t) => t.replace(/(.)\1+/gu, "$1") },
249
+ { rule: null, apply: (t) => t.toLowerCase() },
250
+ // Digits glued to an Arabic word (عارة3, عا٣رة) are padding; in Arabizi they're letters
251
+ {
252
+ rule: "glued-digits",
253
+ apply: (t) => HAS_ARABIC.test(t) && /[0-9]/.test(t) ? t.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : t
254
+ },
255
+ { rule: "arabizi", apply: (t) => HAS_ARABIC.test(t) ? t : foldArabizi(t) }
256
+ ];
257
+ function normalizeSteps(text, skip = -1) {
258
+ return NORMALIZE_STEPS.reduce((t, step, i) => i === skip ? t : step.apply(t), text);
259
+ }
227
260
  var IraqiBadWordsFilter = class {
228
261
  constructor(options = {}) {
229
- const { customWords = [], includeInsults = false, stripSuffixes = false } = Array.isArray(options) ? { customWords: options } : options;
262
+ // Clean words that would otherwise match after a prefix is stripped
263
+ // (e.g. بعير "camel" = ب + عير). Also covers their prefixed and suffixed
264
+ // forms (وبعيرك). Listed in safe-stems.json, plus the safeWords option.
265
+ this.safeStems = [];
266
+ const {
267
+ customWords = [],
268
+ safeWords = [],
269
+ includeInsults = false,
270
+ stripSuffixes = false,
271
+ dialects = []
272
+ } = Array.isArray(options) ? { customWords: options } : options;
230
273
  this.words = /* @__PURE__ */ new Map();
231
274
  this.stripSuffixes = stripSuffixes;
232
- this.safeStems = safe_stems_default.map((word) => ({ normalized: this.normalize(word), word }));
275
+ this.mosul = dialects.includes("mosul");
276
+ this.addSafeStems(safe_stems_default);
277
+ this.addSafeStems(safeWords);
233
278
  this.addEntries(words_default, "words");
234
279
  if (includeInsults) this.addEntries(insults_default, "insults");
235
280
  this.addEntries(customWords, "custom");
236
281
  }
237
282
  /**
238
- * Normalizes Arabic text so spelling variants compare equal:
239
- * removes diacritics (Tashkeel), tatweel and invisible characters, unifies
240
- * Alef/Teh Marbuta/Alef Maksura forms and the Persian/Kurdish keyboard letters
241
- * ی ک ە, turns Arabic-Indic digits into 0-9, and collapses stretched letters
242
- * (عاااارة -> عارة).
283
+ * Normalizes a word or token for comparison: removes diacritics, tatweel and
284
+ * invisible characters, unifies letter variants (أ/ا, ة/ه, ی/ي, ک/ك, …),
285
+ * reads Arabic-Indic digits, collapses stretched letters (عاااارة -> عارة),
286
+ * ignores digits in Arabic words and folds Arabizi. See NORMALIZE_STEPS.
243
287
  */
244
- normalizeArabic(text) {
245
- return text.replace(/[\u064B-\u065F\u0670\u0640]/g, "").replace(INVISIBLE_CHARS, "").replace(/[أإآٱ]/g, "\u0627").replace(/[ةەہۀ]/g, "\u0647").replace(/[ىی]/g, "\u064A").replace(/[کڪ]/g, "\u0643").replace(/[٠-٩]/g, (digit) => String(digit.charCodeAt(0) - 1632)).replace(/[۰-۹]/g, (digit) => String(digit.charCodeAt(0) - 1776)).replace(/(.)\1+/gu, "$1").toLowerCase();
288
+ normalize(text) {
289
+ return normalizeSteps(text);
246
290
  }
247
291
  /**
248
- * Folds Arabizi (Arabic written in Latin letters and digits) so spelling
249
- * variants compare equal: ga7ba / qahba / gahba, kosomak / kusomak,
250
- * teezak / tizak. Only words made purely of Latin letters and digits are
251
- * folded; expects text already passed through normalizeArabic.
292
+ * The ways a normalized word can be read. Iraqi Arabic writes ق as گ, but
293
+ * Kurdish typists sometimes use گ for ك, so a word with گ is tried both ways.
252
294
  */
253
- foldArabizi(word) {
254
- if (!/^[a-z0-9]+$/.test(word) || !/[a-z]/.test(word)) return word;
255
- return word.replace(/7/g, "h").replace(/5/g, "kh").replace(/[23]/g, "").replace(/q/g, "g").replace(/ou|o/g, "u").replace(/e/g, "i").replace(/(.)\1+/g, "$1");
295
+ readings(word) {
296
+ return word.includes("\u06AF") ? [word.replace(/گ/g, "\u0642"), word.replace(/گ/g, "\u0643")] : [word];
256
297
  }
257
- /** Normalizes a word or token for comparison. */
258
- normalize(text) {
259
- const word = this.normalizeArabic(text);
260
- return /[\u0600-\uFEFF]/.test(word) ? word.replace(/[0-9]/g, "").replace(/(.)\1+/gu, "$1") : this.foldArabizi(word);
298
+ /** A list word plus its regional forms (عير -> عير, عيغ with the Mosul dialect). */
299
+ dialectForms(word) {
300
+ return this.mosul && word.includes("\u0631") ? [word, word.replace(/ر/g, "\u063A")] : [word];
301
+ }
302
+ /** The key a list word is stored under. A گ in a list entry means ق. */
303
+ entryKey(word) {
304
+ return this.normalize(word).replace(/گ/g, "\u0642");
305
+ }
306
+ /**
307
+ * A word list with surrounding whitespace trimmed and entries that are empty
308
+ * after normalization skipped. A stray "" in a user's file would otherwise
309
+ * match a lone tatweel as a word, or clear every match as a safe stem.
310
+ */
311
+ cleanList(list) {
312
+ return list.filter((word) => typeof word === "string").map((word) => word.trim()).filter((word) => this.entryKey(word) !== "");
313
+ }
314
+ /**
315
+ * Adds safe stems from one list. A safe stem written with گ protects only
316
+ * words spelled with گ (گلب "heart"), so it isn't read as ق or ك.
317
+ */
318
+ addSafeStems(stems) {
319
+ for (const stem of this.cleanList(stems)) {
320
+ for (const word of this.dialectForms(stem)) {
321
+ this.safeStems.push({ normalized: this.normalize(word), word });
322
+ }
323
+ }
261
324
  }
262
325
  /**
263
326
  * Adds words from one list. A word already present keeps its first source.
264
327
  */
265
- addEntries(newWords, source) {
266
- for (const word of newWords) {
267
- const normalized = this.normalize(word);
268
- if (!this.words.has(normalized)) this.words.set(normalized, { word, source });
328
+ addEntries(listWords, source) {
329
+ const newWords = this.cleanList(listWords);
330
+ const forms = [
331
+ ...newWords.map((word) => ({ word, form: word })),
332
+ ...newWords.flatMap((word) => this.dialectForms(word).slice(1).map((form) => ({ word, form })))
333
+ ];
334
+ for (const { word, form } of forms) {
335
+ const key = this.entryKey(form);
336
+ if (!this.words.has(key)) this.words.set(key, { word, form, source });
269
337
  }
270
338
  }
271
339
  /**
@@ -275,11 +343,44 @@ var IraqiBadWordsFilter = class {
275
343
  * A match cleared by a safe stem is returned with `suppressedBy` set.
276
344
  */
277
345
  matchWord(token) {
278
- const word = this.normalize(token);
279
- const direct = this.words.get(word);
280
- if (direct) {
281
- return { normalized: word, entry: direct.word, source: direct.source, prefix: "", suffix: "" };
346
+ const spelled = this.normalize(token);
347
+ const spelledSafe = spelled.includes("\u06AF") ? this.findSafe(this.prefixCandidates(spelled)) : void 0;
348
+ let suppressed = null;
349
+ for (const reading of this.readings(spelled)) {
350
+ const found = this.matchReading(reading);
351
+ if (!found) continue;
352
+ const { form, ...rest } = found;
353
+ const match = { ...rest, rules: this.explain(token, found, spelled, reading) };
354
+ if (match.suppressedBy === void 0 && spelledSafe) match.suppressedBy = spelledSafe;
355
+ if (match.suppressedBy === void 0) return match;
356
+ suppressed = suppressed ?? match;
357
+ }
358
+ return suppressed;
359
+ }
360
+ /**
361
+ * Works out which rules a match needed, by comparing the typed word with the
362
+ * listed spelling (plus any stripped prefix and suffix). A normalization rule
363
+ * counts only if leaving it out would keep them apart: عاااارة vs عارة needs
364
+ * "stretched" but not "spelling", since both have ة.
365
+ */
366
+ explain(token, found, spelled, reading) {
367
+ const listed = found.prefix + found.form + found.suffix;
368
+ const gaf = reading === spelled ? "" : reading === spelled.replace(/گ/g, "\u0642") ? "\u0642" : "\u0643";
369
+ const same = (typed, list) => typed.replace(/گ/g, gaf || "\u06AF") === list.replace(/گ/g, "\u0642");
370
+ const rules = [];
371
+ NORMALIZE_STEPS.forEach(({ rule }, i) => {
372
+ if (rule && !same(normalizeSteps(token, i), normalizeSteps(listed, i))) rules.push(rule);
373
+ });
374
+ if (gaf && normalizeSteps(token) !== normalizeSteps(listed)) {
375
+ rules.push(gaf === "\u0642" ? "gaf-as-qaf" : "gaf-as-kaf");
282
376
  }
377
+ if (found.prefix) rules.push("prefix");
378
+ if (found.suffix) rules.push("suffix");
379
+ if (found.form !== found.entry) rules.push("mosul");
380
+ return rules;
381
+ }
382
+ /** The word plus each form of it with an attached prefix removed. */
383
+ prefixCandidates(word) {
283
384
  const candidates = [{ prefix: "", stem: word, suffix: "" }];
284
385
  for (const conjunction of CONJUNCTIONS) {
285
386
  for (const preposition of PREPOSITIONS) {
@@ -291,6 +392,30 @@ var IraqiBadWordsFilter = class {
291
392
  }
292
393
  }
293
394
  }
395
+ return candidates;
396
+ }
397
+ /** The first safe stem one of the candidates starts with, as listed. */
398
+ findSafe(candidates) {
399
+ for (const { stem } of candidates) {
400
+ const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
401
+ if (safe) return safe.word;
402
+ }
403
+ return void 0;
404
+ }
405
+ /** Matches one normalized reading of a word. */
406
+ matchReading(word) {
407
+ const direct = this.words.get(word);
408
+ if (direct) {
409
+ return {
410
+ normalized: word,
411
+ entry: direct.word,
412
+ form: direct.form,
413
+ source: direct.source,
414
+ prefix: "",
415
+ suffix: ""
416
+ };
417
+ }
418
+ const candidates = this.prefixCandidates(word);
294
419
  if (this.stripSuffixes) {
295
420
  for (const { prefix, stem } of [...candidates]) {
296
421
  for (const suffix of SUFFIXES) {
@@ -302,19 +427,17 @@ var IraqiBadWordsFilter = class {
302
427
  }
303
428
  const match = candidates.find(({ stem }) => this.words.has(stem));
304
429
  if (!match) return null;
305
- const { word: entry, source } = this.words.get(match.stem);
430
+ const { word: entry, form, source } = this.words.get(match.stem);
306
431
  const result = {
307
432
  normalized: word,
308
433
  entry,
434
+ form,
309
435
  source,
310
436
  prefix: match.prefix,
311
437
  suffix: match.suffix
312
438
  };
313
- for (const { stem } of candidates) {
314
- const safe = this.safeStems.find(({ normalized }) => stem.startsWith(normalized));
315
- if (safe) return { ...result, suppressedBy: safe.word };
316
- }
317
- return result;
439
+ const safe = this.findSafe(candidates);
440
+ return safe ? { ...result, suppressedBy: safe } : result;
318
441
  }
319
442
  /**
320
443
  * Finds every match in the text, including ones cleared by a safe stem.
@@ -387,11 +510,19 @@ var IraqiBadWordsFilter = class {
387
510
  addWords(newWords) {
388
511
  this.addEntries(newWords, "custom");
389
512
  }
513
+ /**
514
+ * Add safe stems to this filter, like the safeWords option.
515
+ */
516
+ addSafeWords(newWords) {
517
+ this.addSafeStems(newWords);
518
+ }
390
519
  /**
391
520
  * Remove words from the active filter dictionary.
392
521
  */
393
522
  removeWords(wordsToRemove) {
394
- wordsToRemove.forEach((word) => this.words.delete(this.normalize(word)));
523
+ for (const word of wordsToRemove.flatMap((w) => this.dialectForms(w))) {
524
+ this.words.delete(this.entryKey(word));
525
+ }
395
526
  }
396
527
  };
397
528
  var index_default = IraqiBadWordsFilter;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "iraqi-dialect-bad-words",
3
- "version": "1.2.1",
3
+ "version": "1.3.0",
4
4
  "description": "An Iraqi dialect profanity filter for React, React Native Expo, and Node.js",
5
5
  "main": "./dist/index.js",
6
6
  "module": "./dist/index.mjs",