persian-normalize 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Qatreh (https://qatrehai.ir)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,145 @@
+# persian-normalize
+
+Make Persian and Arabic-script text comparable. Zero dependencies, ESM, TypeScript types included.
+
+```bash
+npm install persian-normalize
+```
+
+## The problem
+
+In Persian, one word can be written several ways that look identical to a reader and are completely different to a computer:
+
+```js
+const a = "چت‌بات";   // with a zero-width non-joiner (U+200C)
+const b = "چت بات";   // with an ordinary space
+
+a === b                 // false
+a.length === b.length   // true  ← the trap: length tells you nothing
+```
+
+None of these variants is a spelling mistake. All appear in published Persian, and all are encoded distinctly in Unicode:
+
+| Variation | Example |
+|---|---|
+| Zero-width non-joiner | `می‌رود` · `می رود` · `میرود` |
+| Persian yeh vs Arabic yeh | `ی` (U+06CC) vs `ي` (U+064A) |
+| Persian keheh vs Arabic kaf | `ک` (U+06A9) vs `ك` (U+0643) |
+| Heh vs teh marbuta | `شرکه` vs `شرکة` |
+| Three digit sets | `۱۴۰۳` · `١٤٠٣` · `1403` |
+| Diacritics and tatweel | `مَدرسه` · `مدـــرسه` |
+
+A system that recognises one form of `چت‌بات` fails three users in four — and it fails **silently**, because nothing crashes. It just returns the wrong answer.
+
+## Two jobs, opposite settings
+
+The library separates the two things people constantly conflate.
+
+### `cleanText` — for text you store or display
+
+Repairs what a wrong keyboard layout produced, drops decoration, tidies spacing. **Keeps the text readable.**
+
+```js
+import { cleanText } from "persian-normalize";
+
+cleanText("مي‌كنم")     // "می‌کنم"    Arabic yeh and kaf repaired
+cleanText("شركة")       // "شرکه"      teh marbuta fixed
+cleanText("مَدرسه")     // "مدرسه"     harakat dropped
+cleanText("١٤٠٣")       // "۱۴۰۳"      Arabic-Indic digits to Persian
+cleanText("آموزش")      // "آموزش"     the madda is correct Persian — untouched
+```
+
+The ZWNJ survives, because deleting it turns `می‌رود` into `میرود`, which is a different and worse spelling.
+
+### `foldForSearch` — for text you compare
+
+Flattens every variant into one form. The output is for matching, not for reading.
+
+```js
+import { foldForSearch, equals } from "persian-normalize";
+
+foldForSearch("چت‌بات")      // "چت بات"
+foldForSearch("۱۴۰۳")        // "1403"
+foldForSearch("سلام، دنیا!") // "سلام دنیا"
+
+equals("چت‌بات", "چت بات")   // true
+equals("شرکة", "شرکه")       // true
+equals("۱۴۰۳", "1403")       // true
+equals("آموزش", "سازمان")    // false — different words stay different
+```
+
+**The ZWNJ becomes a space, not nothing.** Delete it and `چت‌بات` becomes `چتبات`, which still does not equal `چت بات`. Turn it into a space and both spellings arrive at the same two tokens.
+
+## Matching without the substring trap
+
+The other half of the problem. Persian attaches prefixes and suffixes freely, so a plain `includes()` finds words that are not there:
+
+```js
+"آموزش سازمانی دارید؟".includes("زمان")   // true — «زمان» hides inside «سازمانی»
+```
+
+That one line routes a question about *enterprise training* to a page about *project timelines*, and the answer reads perfectly well. Nobody notices.
+
+```js
+import { containsWord, tokenize } from "persian-normalize";
+
+containsWord("آموزش سازمانی دارید؟", "زمان")   // false ✓
+containsWord("دوره آموزشی ما", "آموزش")        // true  — long enough to prefix-match
+containsWord("قیمت چت‌بات چقدر است", "چت بات")  // true  — phrase, across spellings
+
+tokenize("چت‌بات سازمانی!")                     // ["چت", "بات", "سازمانی"]
+```
+
+Only the final token of a phrase may match by prefix, and only when it is at least `minPrefix` characters (default 5) — so `آموزش` still matches `آموزشی`, while `زمان` no longer matches `سازمانی`.
+
+## API
+
+| Function | Purpose |
+|---|---|
+| `cleanText(s, opts?)` | Repair for storage or display |
+| `foldForSearch(s, opts?)` | Flatten for comparison |
+| `equals(a, b)` | Compare ignoring spelling variation |
+| `tokenize(s)` | Folded tokens |
+| `containsWord(haystack, needle, opts?)` | Whole-word or phrase match |
+| `ZWNJ` | The U+200C character |
+
+```ts
+cleanText(input, {
+  digitsToLatin?: boolean,          // ۱۲۳ -> 123        default false
+  arabicDigitsToPersian?: boolean,  // ١٢٣ -> ۱۲۳        default true
+  diacritics?: boolean,             // strip harakat     default true
+  collapseSpaces?: boolean,         //                   default true
+})
+
+foldForSearch(input, { keepPunctuation?: boolean })   // default false
+containsWord(haystack, needle, { minPrefix?: number }) // default 5
+```
+
+## Notes
+
+**This is not `String.prototype.normalize()`.** Unicode NFC/NFKC compose and decompose characters, but Persian yeh and Arabic yeh are *separate letters with separate meanings*, not two encodings of one character. Unicode will not merge them, and it should not. Script folding is an application-level decision.
+
+**Normalize your reference strings too**, at start-up. A keyword list typed by a developer on one layout and a query typed by a user on another will otherwise never meet, however good the folding is on the input side.
+
+**It is not only chatbots.** Anywhere Persian text is compared: product search, customer-name lookup, address matching, deduplication. In a database, two spellings of one company name remain two separate records.
+
+## Tests
+
+Zero dependencies, including dev ones.
+
+```bash
+node --test test/
+```
+
+Every case came from a real failure in a production Persian assistant, which is why the negative assertions — the things that must *not* fold together — carry as much weight as the positive ones.
+
+## Background
+
+The reasoning behind each step, and how the failures were found:
+[Why Your Persian Chatbot Answers the Wrong Question](https://qatrehai.ir/blog/persian-chatbot-text-normalization-en)
+
+## Licence
+
+MIT © [Qatreh](https://qatrehai.ir) — an AI team in Karaj, Iran.
+
+Found a variant this misses? Please open an issue — that is exactly the contribution this needs.

package/package.json

@@ -0,0 +1,56 @@
+{
+  "name": "persian-normalize",
+  "version": "1.0.0",
+  "description": "Normalize Persian and Arabic-script text. Fold ZWNJ, Arabic yeh/kaf, three digit sets and diacritics into one comparable form — or clean text for storage without destroying it.",
+  "type": "module",
+  "main": "./src/index.js",
+  "module": "./src/index.js",
+  "types": "./src/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./src/index.d.ts",
+      "import": "./src/index.js",
+      "default": "./src/index.js"
+    }
+  },
+  "files": [
+    "src"
+  ],
+  "sideEffects": false,
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "test": "node --test test/"
+  },
+  "keywords": [
+    "persian",
+    "farsi",
+    "arabic",
+    "normalize",
+    "normalization",
+    "zwnj",
+    "half-space",
+    "nim-fasele",
+    "rtl",
+    "unicode",
+    "text",
+    "search",
+    "nlp",
+    "tokenize",
+    "diacritics",
+    "persian-digits",
+    "arabic-digits",
+    "i18n"
+  ],
+  "author": "Qatreh <qatreh.ai@gmail.com> (https://qatrehai.ir)",
+  "license": "MIT",
+  "homepage": "https://qatrehai.ir/blog/persian-chatbot-text-normalization-en",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/qatrehai/persian-normalize.git"
+  },
+  "bugs": {
+    "url": "https://github.com/qatrehai/persian-normalize/issues"
+  }
+}

package/src/index.d.ts

@@ -0,0 +1,62 @@
+/** Zero-width non-joiner (U+200C) — invisible, meaningful, and the usual culprit. */
+export declare const ZWNJ: "‌";
+
+export interface CleanTextOptions {
+  /** ۱۲۳ -> 123. Off by default: Persian text usually wants Persian digits. */
+  digitsToLatin?: boolean;
+  /** ١٢٣ -> ۱۲۳. On by default; the Arabic-Indic set is a keyboard accident. */
+  arabicDigitsToPersian?: boolean;
+  /** Strip harakat and tatweel. On by default. */
+  diacritics?: boolean;
+  /** Collapse runs of spaces without disturbing the ZWNJ. On by default. */
+  collapseSpaces?: boolean;
+}
+
+export interface FoldOptions {
+  /** Keep punctuation instead of collapsing it to whitespace. */
+  keepPunctuation?: boolean;
+}
+
+export interface ContainsOptions {
+  /** Shortest token allowed to match by prefix. Default 5. */
+  minPrefix?: number;
+}
+
+/**
+ * Repair text for storage or display. Keeps the ZWNJ, punctuation and case, so
+ * the result still reads as the language it came from.
+ */
+export declare function cleanText(input: string, options?: CleanTextOptions): string;
+
+/**
+ * Fold text into one comparable form for matching. Not meant to be read back:
+ * letter variants, all three digit sets, case and punctuation are flattened.
+ */
+export declare function foldForSearch(input: string, options?: FoldOptions): string;
+
+/** Do two strings mean the same thing, ignoring spelling variation? */
+export declare function equals(a: string, b: string): boolean;
+
+/** Split folded text into tokens. */
+export declare function tokenize(input: string): string[];
+
+/**
+ * Does `haystack` contain `needle` as whole words? The needle may be a phrase;
+ * its tokens must appear in order and adjacent. Only the final token may match
+ * by prefix, and only when it is at least `minPrefix` characters.
+ */
+export declare function containsWord(
+  haystack: string,
+  needle: string,
+  options?: ContainsOptions
+): boolean;
+
+declare const _default: {
+  cleanText: typeof cleanText;
+  foldForSearch: typeof foldForSearch;
+  equals: typeof equals;
+  tokenize: typeof tokenize;
+  containsWord: typeof containsWord;
+  ZWNJ: typeof ZWNJ;
+};
+export default _default;

package/src/index.js

@@ -0,0 +1,216 @@
+/**
+ * persian-normalize — make Persian and Arabic-script text comparable.
+ *
+ * Two jobs that are constantly confused, and that need opposite settings:
+ *
+ *   cleanText(s)   Repair text you are going to STORE or DISPLAY. Fixes the
+ *                  characters a wrong keyboard layout produces, drops
+ *                  decoration, tidies spacing — and keeps the text readable.
+ *                  The zero-width non-joiner survives, because «می‌رود» without
+ *                  it is «میرود», which is a different and worse spelling.
+ *
+ *   foldForSearch(s)  Reduce text you are going to COMPARE to a single form.
+ *                     Folds every variant together, ASCII digits, no
+ *                     punctuation, no case. The output is not meant to be read.
+ *
+ * Using cleanText for matching leaves «چت‌بات» and «چت بات» unequal. Using
+ * foldForSearch for storage destroys the writing. Pick by what you are doing.
+ */
+
+/** Zero-width non-joiner: invisible, meaningful, and the usual culprit. */
+export const ZWNJ = "‌";
+
+/**
+ * Characters an Arabic keyboard produces where Persian expects its own letter.
+ * These are always safe to repair: no Persian word wants the Arabic form.
+ */
+const KEYBOARD_FIXES = {
+  "ي": "ی", // ARABIC YEH            ي -> ی
+  "ى": "ی", // ALEF MAKSURA          ى -> ی
+  "ك": "ک", // ARABIC KAF            ك -> ک
+  "ة": "ه", // TEH MARBUTA           ة -> ه
+  "ۀ": "ه", // HEH WITH YEH ABOVE    ۀ -> ه
+  "ۍ": "ی", // YEH WITH TAIL
+  "ـ": "",       // TATWEEL — decoration only
+};
+
+/**
+ * Further folds that lose information, so they belong to search only.
+ * «آب» and «اب» are different words; collapsing them is right for matching
+ * and wrong for storage.
+ */
+const SEARCH_FOLDS = {
+  "آ": "ا", // ALEF WITH MADDA       آ -> ا
+  "أ": "ا", // ALEF WITH HAMZA ABOVE أ -> ا
+  "إ": "ا", // ALEF WITH HAMZA BELOW إ -> ا
+  "ٱ": "ا", // ALEF WASLA
+  "ؤ": "و", // WAW WITH HAMZA        ؤ -> و
+  "ئ": "ی", // YEH WITH HAMZA        ئ -> ی
+  "ء": "",       // HAMZA on its own
+};
+
+/** Harakat, sukun, dagger alef — optional vowel marks that survive pasting. */
+const DIACRITICS = /[ً-ْٰٓ-ٕٖ-ٟۖ-ۭ]/g;
+
+/** Bidi controls and other invisibles that break comparison silently. */
+const INVISIBLES = /[​‍‎‏‪-‮⁦-⁩]/g;
+
+const PERSIAN_DIGITS = /[۰-۹]/g; // ۰-۹
+const ARABIC_DIGITS = /[٠-٩]/g;  // ٠-٩
+
+const mapChars = (input, table) => {
+  let out = "";
+  for (const ch of input) out += ch in table ? table[ch] : ch;
+  return out;
+};
+
+/**
+ * Repair text for storage or display.
+ *
+ * Keeps the ZWNJ, keeps punctuation, keeps letter case, and leaves Persian
+ * digits as Persian digits unless asked otherwise — the result should still
+ * read as the language it came from.
+ *
+ * @param {string} input
+ * @param {object} [options]
+ * @param {boolean} [options.digitsToLatin=false]  ۱۲۳ -> 123
+ * @param {boolean} [options.arabicDigitsToPersian=true]  ١٢٣ -> ۱۲۳
+ * @param {boolean} [options.diacritics=true]  strip harakat and tatweel
+ * @param {boolean} [options.collapseSpaces=true]
+ * @returns {string}
+ */
+export function cleanText(input, options = {}) {
+  if (typeof input !== "string") return "";
+  const {
+    digitsToLatin = false,
+    arabicDigitsToPersian = true,
+    diacritics = true,
+    collapseSpaces = true,
+  } = options;
+
+  let s = input.replace(INVISIBLES, "");
+  if (diacritics) s = s.replace(DIACRITICS, "");
+  s = mapChars(s, KEYBOARD_FIXES);
+
+  if (digitsToLatin) {
+    s = s.replace(PERSIAN_DIGITS, (d) => String(d.charCodeAt(0) - 0x06f0))
+         .replace(ARABIC_DIGITS, (d) => String(d.charCodeAt(0) - 0x0660));
+  } else if (arabicDigitsToPersian) {
+    s = s.replace(ARABIC_DIGITS, (d) => String.fromCharCode(d.charCodeAt(0) - 0x0660 + 0x06f0));
+  }
+
+  if (collapseSpaces) {
+    // Collapse runs of ordinary spaces, but never touch the ZWNJ, and never
+    // let a space sit next to one — that pair renders as a double gap.
+    s = s.replace(/[ \t]+/g, " ")
+         .replace(new RegExp(`\\s*${ZWNJ}\\s*`, "g"), ZWNJ)
+         .replace(/ *\n */g, "\n")
+         .trim();
+  }
+  return s;
+}
+
+/**
+ * Fold text into one comparable form.
+ *
+ * The output is for matching, not for reading. Everything that can vary is
+ * flattened: letter variants, all three digit sets, case, punctuation.
+ *
+ * The ZWNJ becomes a SPACE rather than being deleted. Delete it and «چت‌بات»
+ * becomes «چتبات», which still does not equal «چت بات»; turn it into a space
+ * and both spellings arrive at the same two tokens.
+ *
+ * @param {string} input
+ * @param {object} [options]
+ * @param {boolean} [options.keepPunctuation=false]
+ * @returns {string}
+ */
+export function foldForSearch(input, options = {}) {
+  if (typeof input !== "string") return "";
+  const { keepPunctuation = false } = options;
+
+  let s = input
+    .replace(/&(?:amp|lt|gt|quot|nbsp|#x27|#39);/g, " ")
+    .replace(DIACRITICS, "");
+
+  // Every invisible, the ZWNJ included, becomes a space.
+  s = s.replace(new RegExp(`[${ZWNJ}​‍‎‏‪-‮⁦-⁩]`, "g"), " ");
+
+  s = mapChars(s, KEYBOARD_FIXES);
+  s = mapChars(s, SEARCH_FOLDS);
+
+  s = s.replace(PERSIAN_DIGITS, (d) => String(d.charCodeAt(0) - 0x06f0))
+       .replace(ARABIC_DIGITS, (d) => String(d.charCodeAt(0) - 0x0660))
+       .toLowerCase();
+
+  if (!keepPunctuation) s = s.replace(/[^\p{L}\p{N}]+/gu, " ");
+  return s.replace(/\s+/g, " ").trim();
+}
+
+/**
+ * Do two strings mean the same thing, ignoring spelling variation?
+ * @param {string} a
+ * @param {string} b
+ * @returns {boolean}
+ */
+export function equals(a, b) {
+  return foldForSearch(a) === foldForSearch(b);
+}
+
+/**
+ * Split folded text into tokens.
+ *
+ * Compare at token level rather than by substring. `"آموزش سازمانی".includes("زمان")`
+ * is true — «زمان» hides inside «سازمانی» — and that one mistake routes a
+ * question about enterprise training to a page about project timelines.
+ *
+ * @param {string} input
+ * @returns {string[]}
+ */
+export function tokenize(input) {
+  const folded = foldForSearch(input);
+  return folded ? folded.split(" ") : [];
+}
+
+/**
+ * Does `haystack` contain `needle` as whole words?
+ *
+ * The needle may be a phrase. That is not a nicety: «چت بات» is two tokens and
+ * «چت‌بات» is one string, and a term-matching function that only handles single
+ * words cannot answer the most common question anyone asks of Persian text.
+ * The tokens must appear in order and adjacent.
+ *
+ * Only the final token may match by prefix, and only when it is long enough
+ * that a shared opening cannot be coincidental — so «آموزش» still matches
+ * «آموزشی», while «زمان» no longer matches «سازمانی».
+ *
+ * @param {string} haystack
+ * @param {string} needle  single word or phrase
+ * @param {object} [options]
+ * @param {number} [options.minPrefix=5] shortest token allowed to prefix-match
+ * @returns {boolean}
+ */
+export function containsWord(haystack, needle, options = {}) {
+  const { minPrefix = 5 } = options;
+  const target = tokenize(needle);
+  if (!target.length) return false;
+  const words = tokenize(haystack);
+  if (target.length > words.length) return false;
+
+  for (let i = 0; i <= words.length - target.length; i++) {
+    let matched = true;
+    for (let j = 0; j < target.length; j++) {
+      const word = words[i + j];
+      const tok = target[j];
+      if (word === tok) continue;
+      const isLast = j === target.length - 1;
+      if (isLast && tok.length >= minPrefix && word.startsWith(tok)) continue;
+      matched = false;
+      break;
+    }
+    if (matched) return true;
+  }
+  return false;
+}
+
+export default { cleanText, foldForSearch, equals, tokenize, containsWord, ZWNJ };