@stll/text-normalize 0.0.1-placeholder.0 → 0.1.1

package/README.md

@@ -1 +1,81 @@
-Placeholder for @stll/text-normalize; real releases are published from https://github.com/stella/stella.
+# @stll/text-normalize
+
+Search match-key text normalization. Folds orthographic variants so a
+search query matches text regardless of how it was typed.
+
+The normalized output is a **match key**: it is lossy and intended only
+for indexing and query comparison. Never store or display it in place of
+the original text.
+
+## Arabic
+
+`arabicNormalize` folds the orthographic variants that make Arabic
+search miss otherwise-identical words:
+
+- alef variants and alef-wasla (`أ إ آ ٱ`) to bare alef `ا`
+- waw-hamza `ؤ` to `و`, yeh-hamza `ئ` to `ي`, standalone hamza `ء` dropped
+- teh marbuta `ة` to heh `ه`
+- alef maksura `ى` to yeh `ي`
+- tashkeel (harakat), superscript alef, and tatweel removed
+- Arabic-Indic and Extended Arabic-Indic digits to ASCII
+
+It also runs NFKC (folding presentation forms), locale-stable ASCII case
+folding, and whitespace collapse, so it is a safe pass-through for
+non-Arabic scripts.
+
+The fold tables are vendored from Lucene's `ArabicNormalizer` (Apache-2.0)
+and cross-checked against CAMeL Tools (MIT), extended for the classes
+Lucene omits.
+
+```ts
+import { arabicNormalize } from "@stll/text-normalize";
+
+arabicNormalize("أحمد") === arabicNormalize("احمد"); // true
+```
+
+The same fold must be reproduced by the PostgreSQL `arabic_normalize()`
+function used in contacts trigram expression indexes and search predicates;
+the golden vectors in `src/normalize.test.ts` pin the shared contract.
+
+## Diacritics
+
+`stripDiacritics` decomposes with NFD and removes every combining mark
+(`\p{Diacritic}`, so marks beyond the U+0300–U+036F block are covered
+too). Use it for accent-insensitive search keys.
+
+`foldToAscii` keeps that strip as its base and adds the folds a
+decomposition cannot express, so it matches PostgreSQL `unaccent()` for
+every Latin-script character for which the extension declares a rule:
+`ł` to `l`, `đ` to `d`, `ø` to `o`, `ß` to `ss`, `æ` to `ae`, and the
+rest of the table in `src/ascii-fold-table.ts`. Use it wherever the
+counterpart is an `unaccent`-folded index (or a Tantivy `ascii_folding`
+one); a plain mark strip leaves `ł` standing and under-matches the
+lexemes the index actually stored. Non-Latin scripts are out of contract
+and pass through with their marks stripped.
+
+`stripDiacriticsForSlug` is the same strip but decomposes with NFKD, so
+compatibility characters (ligatures, full-width forms, superscripts) fold
+to their base before a `[a-z0-9]` slug filter runs. Slugs are persisted,
+public URL segments; this variant pins the exact form existing slugs were
+generated with.
+
+## Substring matching with highlight offsets
+
+`foldSearchMatchText` is the client-side match key for interactive
+filtering: NFKD, the `unaccent()`-parity ASCII folds, mark strip, Arabic
+folds, lowercase — `capek` matches `Čapek` and `wroclaw` matches
+`Wrocław`, both ways. `findSearchMatchRanges` finds a folded query in
+folded content and returns ranges into the **original** string, so
+highlights wrap the text the user sees even where folding changes string
+length; it accepts content pre-folded with `foldSearchMatchTextWithOffsets`
+so repeated queries against the same text fold it once.
+
+## Spaced letters
+
+`collapseSpacedLetters` joins letter-spaced court headings
+(`r o z h o d o l :` to `rozhodol:`) back into searchable words. It
+requires at least four spaced letters, so Czech/Slovak single-letter word
+lists (`a b c`) are left intact. `spacedLetterRunRegex()` returns the
+underlying matcher for callers that need per-match offsets (find-in-page
+highlighting). Index-time and query-time collapse share this one
+threshold so a heading collapses identically on both sides.

package/dist/arabic.d.ts

@@ -0,0 +1,67 @@
+/**
+ * Arabic orthographic fold tables for SEARCH match-key normalization.
+ *
+ * Vendored from Lucene's ArabicNormalizer (Apache-2.0) and cross-checked
+ * against CAMeL Tools (MIT), then extended for the classes Lucene omits:
+ * alef-wasla, waw/yeh/standalone hamza, superscript alef, and Arabic-Indic
+ * digits. Fold directions (alef variants to bare alef, teh-marbuta to heh,
+ * alef-maksura to yeh) are the settled Lucene/CAMeL consensus.
+ *
+ * These folds are lossy match-key transforms: never apply them to stored
+ * or displayed text, only to search keys.
+ */
+export declare const ARABIC_LETTER_FOLDS: Readonly<Record<string, string>>;
+export declare const ARABIC_REMOVED: readonly string[];
+export declare const ARABIC_DIGIT_FOLDS: Readonly<Record<string, string>>;
+/**
+ * Apply the Arabic letter, digit, and removal folds codepoint by
+ * codepoint. Non-Arabic characters pass through unchanged.
+ *
+ * Deliberately does not normalize. Unicode normalization is whole-string
+ * work — NFKC composes across codepoint boundaries — so it belongs to
+ * the caller, before the fold: `arabicNormalize` applies NFKC to the
+ * whole string and then calls this. Folding is a per-codepoint table
+ * lookup and stays that way.
+ */
+export declare const applyArabicFolds: (text: string) => string;
+export type FoldedText = {
+    sourceEndIndex: number[];
+    text: string;
+    truncated: boolean;
+    sourceIndex: number[];
+};
+type ApplyArabicFoldsWithOffsetsOptions = {
+    maxFoldedUnits?: number;
+};
+/**
+ * Fold Arabic variants and return an offset map, so a caller matching
+ * against the folded text (find-in-page, search previews) can slice the
+ * original at the right positions.
+ *
+ * Not simply `applyArabicFolds` plus offsets. This variant additionally
+ * normalizes NFKC one character at a time, because it is meant for raw
+ * document text: Arabic PDFs carry presentation forms (`ﺍﺣﻤﺪ`) and
+ * ligatures (`ﷲ`, `ﷺ`) that must expand to canonical letters to be
+ * searchable at all. Per character, not whole string, is what keeps
+ * every unit of an expansion pointing back at the one source character
+ * it came from — whole-string NFKC composes across boundaries and would
+ * describe positions the map no longer has. The `maxFoldedUnits` budget
+ * exists for the same reason: one ligature can expand to eighteen
+ * characters.
+ *
+ * The consequence is a deliberate asymmetry with `applyArabicFolds`,
+ * which never normalizes. On a compatibility character the two disagree:
+ *
+ *     applyArabicFolds("fi")            === "fi"
+ *     applyArabicFoldsWithOffsets("fi") === "fi"
+ *
+ * They agree on NFKC-normalized input, and that is the state both sides
+ * of a comparison are in wherever the two are used together: the API's
+ * search highlighter folds its source through `normalizeSourceWithMappings`
+ * and each candidate through `arabicNormalize`, and both apply
+ * whole-string NFKC before either fold runs. Feeding raw text to one side
+ * and normalized text to the other is what would misplace a highlight, so
+ * normalize both or neither.
+ */
+export declare const applyArabicFoldsWithOffsets: (input: string, options?: ApplyArabicFoldsWithOffsetsOptions) => FoldedText;
+export {};

package/dist/arabic.js

@@ -0,0 +1,156 @@
+/**
+ * Arabic orthographic fold tables for SEARCH match-key normalization.
+ *
+ * Vendored from Lucene's ArabicNormalizer (Apache-2.0) and cross-checked
+ * against CAMeL Tools (MIT), then extended for the classes Lucene omits:
+ * alef-wasla, waw/yeh/standalone hamza, superscript alef, and Arabic-Indic
+ * digits. Fold directions (alef variants to bare alef, teh-marbuta to heh,
+ * alef-maksura to yeh) are the settled Lucene/CAMeL consensus.
+ *
+ * These folds are lossy match-key transforms: never apply them to stored
+ * or displayed text, only to search keys.
+ */
+// One-to-one letter folds (source codepoint to target codepoint).
+export const ARABIC_LETTER_FOLDS = {
+    آ: "ا", // آ alef madda       -> ا alef
+    أ: "ا", // أ alef hamza above -> ا alef
+    إ: "ا", // إ alef hamza below -> ا alef
+    ٱ: "ا", // ٱ alef wasla       -> ا alef
+    ؤ: "و", // ؤ waw hamza        -> و waw
+    ئ: "ي", // ئ yeh hamza        -> ي yeh
+    ة: "ه", // ة teh marbuta      -> ه heh
+    ى: "ي", // ى alef maksura     -> ي yeh
+};
+// Codepoints folded to nothing (removed). Tatweel, the eight harakat
+// (U+064B–U+0652), decomposed hamza/madda marks, superscript alef, and
+// standalone hamza.
+export const ARABIC_REMOVED = [
+    "ء", // ء standalone hamza
+    "ـ", // ـ tatweel / kashida
+    "ً", // fathatan
+    "ٌ", // dammatan
+    "ٍ", // kasratan
+    "َ", // fatha
+    "ُ", // damma
+    "ِ", // kasra
+    "ّ", // shadda
+    "ْ", // sukun
+    "ٓ", // madda above
+    "ٔ", // hamza above
+    "ٕ", // hamza below
+    "ٰ", // superscript alef
+];
+// Arabic-Indic (U+0660–0669) and Extended Arabic-Indic (U+06F0–06F9)
+// digits to ASCII. Same mapping as @stll/stdnum's internal digit table;
+// inlined here to avoid depending on another package's private util.
+export const ARABIC_DIGIT_FOLDS = {
+    "٠": "0",
+    "١": "1",
+    "٢": "2",
+    "٣": "3",
+    "٤": "4",
+    "٥": "5",
+    "٦": "6",
+    "٧": "7",
+    "٨": "8",
+    "٩": "9",
+    "۰": "0",
+    "۱": "1",
+    "۲": "2",
+    "۳": "3",
+    "۴": "4",
+    "۵": "5",
+    "۶": "6",
+    "۷": "7",
+    "۸": "8",
+    "۹": "9",
+};
+const FOLD_MAP = new Map([
+    ...Object.entries(ARABIC_LETTER_FOLDS),
+    ...Object.entries(ARABIC_DIGIT_FOLDS),
+    ...ARABIC_REMOVED.map((char) => [char, ""]),
+]);
+/**
+ * Apply the Arabic letter, digit, and removal folds codepoint by
+ * codepoint. Non-Arabic characters pass through unchanged.
+ *
+ * Deliberately does not normalize. Unicode normalization is whole-string
+ * work — NFKC composes across codepoint boundaries — so it belongs to
+ * the caller, before the fold: `arabicNormalize` applies NFKC to the
+ * whole string and then calls this. Folding is a per-codepoint table
+ * lookup and stays that way.
+ */
+export const applyArabicFolds = (text) => {
+    // Fast path for the single-code-unit calls made character-by-character
+    // when building offset maps; avoids an array allocation per character.
+    if (text.length === 1) {
+        return FOLD_MAP.get(text) ?? text;
+    }
+    const out = [];
+    for (const char of text) {
+        out.push(FOLD_MAP.get(char) ?? char);
+    }
+    return out.join("");
+};
+/**
+ * Fold Arabic variants and return an offset map, so a caller matching
+ * against the folded text (find-in-page, search previews) can slice the
+ * original at the right positions.
+ *
+ * Not simply `applyArabicFolds` plus offsets. This variant additionally
+ * normalizes NFKC one character at a time, because it is meant for raw
+ * document text: Arabic PDFs carry presentation forms (`ﺍﺣﻤﺪ`) and
+ * ligatures (`ﷲ`, `ﷺ`) that must expand to canonical letters to be
+ * searchable at all. Per character, not whole string, is what keeps
+ * every unit of an expansion pointing back at the one source character
+ * it came from — whole-string NFKC composes across boundaries and would
+ * describe positions the map no longer has. The `maxFoldedUnits` budget
+ * exists for the same reason: one ligature can expand to eighteen
+ * characters.
+ *
+ * The consequence is a deliberate asymmetry with `applyArabicFolds`,
+ * which never normalizes. On a compatibility character the two disagree:
+ *
+ *     applyArabicFolds("fi")            === "fi"
+ *     applyArabicFoldsWithOffsets("fi") === "fi"
+ *
+ * They agree on NFKC-normalized input, and that is the state both sides
+ * of a comparison are in wherever the two are used together: the API's
+ * search highlighter folds its source through `normalizeSourceWithMappings`
+ * and each candidate through `arabicNormalize`, and both apply
+ * whole-string NFKC before either fold runs. Feeding raw text to one side
+ * and normalized text to the other is what would misplace a highlight, so
+ * normalize both or neither.
+ */
+export const applyArabicFoldsWithOffsets = (input, options = {}) => {
+    const parts = [];
+    const sourceEndIndex = [];
+    const sourceIndex = [];
+    const maxFoldedUnits = options.maxFoldedUnits ?? Number.POSITIVE_INFINITY;
+    let foldedUnits = 0;
+    let originalUnit = 0;
+    let truncated = false;
+    for (const char of input) {
+        const replacement = applyArabicFolds(char.normalize("NFKC"));
+        if (foldedUnits + replacement.length > maxFoldedUnits) {
+            truncated = true;
+            break;
+        }
+        const originalEnd = originalUnit + char.length;
+        parts.push(replacement);
+        // One offset entry per UTF-16 code unit of the replacement; folds are
+        // BMP, but an unfolded astral passthrough spans two code units.
+        let unit = 0;
+        while (unit < replacement.length) {
+            sourceIndex.push(originalUnit);
+            sourceEndIndex.push(originalEnd);
+            unit += 1;
+        }
+        foldedUnits += replacement.length;
+        originalUnit = originalEnd;
+    }
+    sourceIndex.push(originalUnit);
+    sourceEndIndex.push(originalUnit);
+    return { sourceEndIndex, text: parts.join(""), truncated, sourceIndex };
+};
+//# sourceMappingURL=arabic.js.map

package/dist/arabic.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"arabic.js","sourceRoot":"","sources":["../src/arabic.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,kEAAkE;AAClE,MAAM,CAAC,MAAM,mBAAmB,GAAqC;IACnE,CAAC,EAAE,GAAG,EAAE,+BAA+B;IACvC,CAAC,EAAE,GAAG,EAAE,+BAA+B;IACvC,CAAC,EAAE,GAAG,EAAE,+BAA+B;IACvC,CAAC,EAAE,GAAG,EAAE,+BAA+B;IACvC,CAAC,EAAE,GAAG,EAAE,8BAA8B;IACtC,CAAC,EAAE,GAAG,EAAE,8BAA8B;IACtC,CAAC,EAAE,GAAG,EAAE,8BAA8B;IACtC,CAAC,EAAE,GAAG,EAAE,8BAA8B;CACvC,CAAC;AAEF,qEAAqE;AACrE,uEAAuE;AACvE,oBAAoB;AACpB,MAAM,CAAC,MAAM,cAAc,GAAsB;IAC/C,GAAG,EAAE,qBAAqB;IAC1B,GAAG,EAAE,sBAAsB;IAC3B,GAAG,EAAE,WAAW;IAChB,GAAG,EAAE,WAAW;IAChB,GAAG,EAAE,WAAW;IAChB,GAAG,EAAE,QAAQ;IACb,GAAG,EAAE,QAAQ;IACb,GAAG,EAAE,QAAQ;IACb,GAAG,EAAE,SAAS;IACd,GAAG,EAAE,QAAQ;IACb,GAAG,EAAE,cAAc;IACnB,GAAG,EAAE,cAAc;IACnB,GAAG,EAAE,cAAc;IACnB,GAAG,EAAE,mBAAmB;CACzB,CAAC;AAEF,qEAAqE;AACrE,wEAAwE;AACxE,qEAAqE;AACrE,MAAM,CAAC,MAAM,kBAAkB,GAAqC;IAClE,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;IACR,GAAG,EAAE,GAAG;CACT,CAAC;AAEF,MAAM,QAAQ,GAAgC,IAAI,GAAG,CAAiB;IACpE,GAAG,MAAM,CAAC,OAAO,CAAC,mBAAmB,CAAC;IACtC,GAAG,MAAM,CAAC,OAAO,CAAC,kBAAkB,CAAC;IACrC,GAAG,cAAc,CAAC,GAAG,CAAC,CAAC,IAAI,EAA6B,EAAE,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;CACvE,CAAC,CAAC;AAEH;;;;;;;;;GASG;AACH,MAAM,CAAC,MAAM,gBAAgB,GAAG,CAAC,IAAY,EAAU,EAAE;IACvD,uEAAuE;IACvE,uEAAuE;IACvE,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtB,OAAO,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC;IACpC,CAAC;IACD,MAAM,GAAG,GAAa,EAAE,CAAC;IACzB,KAAK,MAAM,IAAI,IAAI,IAAI,EAAE,CAAC;QACxB,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,CAAC;IACvC,CAAC;IACD,OAAO,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;AACtB,CAAC,CAAC;AAsBF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6BG;AACH,MAAM,CAAC,MAAM,2BAA2B,GAAG,CACzC,KAAa,EACb,OAAO,GAAuC,EAAE,EACpC,EAAE;IACd,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,MAAM,cAAc,GAAa,EAAE,CAAC;IACpC,MAAM,WAAW,GAAa,EAAE,CAAC;IACjC,MAAM,cAAc,GAAG,OAAO,CAAC,cAAc,IAAI,MAAM,CAAC,iBAAiB,CAAC;IAC1E,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,IAAI,YAAY,GAAG,CAAC,CAAC;IACrB,IAAI,SAAS,GAAG,KAAK,CAAC;IACtB,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,WAAW,GAAG,gBAAgB,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC,CAAC;QAC7D,IAAI,WAAW,GAAG,WAAW,CAAC,MAAM,GAAG,cAAc,EAAE,CAAC;YACtD,SAAS,GAAG,IAAI,CAAC;YACjB,MAAM;QACR,CAAC;QAED,MAAM,WAAW,GAAG,YAAY,GAAG,IAAI,CAAC,MAAM,CAAC;QAC/C,KAAK,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QACxB,sEAAsE;QACtE,gEAAgE;QAChE,IAAI,IAAI,GAAG,CAAC,CAAC;QACb,OAAO,IAAI,GAAG,WAAW,CAAC,MAAM,EAAE,CAAC;YACjC,WAAW,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC/B,cAAc,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;YACjC,IAAI,IAAI,CAAC,CAAC;QACZ,CAAC;QACD,WAAW,IAAI,WAAW,CAAC,MAAM,CAAC;QAClC,YAAY,GAAG,WAAW,CAAC;IAC7B,CAAC;IACD,WAAW,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;IAC/B,cAAc,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;IAClC,OAAO,EAAE,cAAc,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;AAC1E,CAAC,CAAC"}

package/dist/ascii-fold-table.d.ts

@@ -0,0 +1,29 @@
+/**
+ * Latin-script fold table: the characters PostgreSQL `unaccent()` folds that
+ * NFD plus combining-mark removal cannot reach, either because the letter has
+ * no canonical decomposition (`ł`, `đ`, `ø`, `ß`) or because Unicode
+ * classifies it as a diacritic and a mark strip deletes it outright (`ʰ`, `ᴬ`).
+ *
+ * Generated from the extension's own rules, so the two sides cannot drift by
+ * hand:
+ *
+ * ```sql
+ * SELECT cp, unaccent(chr(cp))
+ * FROM generate_series(1, 196607) AS cp
+ * WHERE NOT (cp BETWEEN 55296 AND 57343)
+ *   AND unaccent(chr(cp)) IS DISTINCT FROM chr(cp);
+ * ```
+ *
+ * A row becomes an entry below when its character is `\p{Script=Latin}` and its
+ * `unaccent` output differs from NFD plus mark removal. Non-Latin scripts stay
+ * out of the contract and pass through unchanged; see `foldToAscii`.
+ *
+ * `apps/api/src/lib/search/ascii-fold-unaccent-parity.db.test.ts` re-runs the
+ * query against a live PostgreSQL and asserts both directions, so an extension
+ * upgrade that adds, drops, or retargets a rule fails CI instead of silently
+ * splitting the query-side fold from the indexed lexemes.
+ *
+ * Keys are single code points; values are ASCII, which is what makes
+ * `foldToAscii` idempotent.
+ */
+export declare const ASCII_FOLD_TABLE: Readonly<Record<string, string>>;