souqnorm 0.1.0__tar.gz

souqnorm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Muhammad Tanzil
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

souqnorm-0.1.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: souqnorm
+Version: 0.1.0
+Summary: Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries
+Author-email: Muhammad Tanzil <mtanzil666@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/mtanzil6/souqnorm
+Project-URL: Issues, https://github.com/mtanzil6/souqnorm/issues
+Keywords: arabic,nlp,e-commerce,search,query normalization,information retrieval
+Classifier: Programming Language :: Python :: 3
+Classifier: Natural Language :: Arabic
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# souqnorm
+
+**Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries.**
+
+Shoppers in Arabic-speaking markets type the same product many ways: `شامبو ٤٠٠مل`, `شامبو 400 ملل`,
+`شاـــمبو 400ml`, `shampoo 13.5 fl oz`. Off-the-shelf search analyzers treat each of these as different
+text. `souqnorm` is a small, dependency-free Python library that turns these forms into one consistent
+representation before indexing or querying, whatever search engine you use.
+
+## Features (v0.1)
+
+- **Arabic orthographic normalization.** It unifies alef forms (أ إ آ ٱ → ا), maps ى → ي, ة → ه and ؤ/ئ → و/ي,
+  and strips diacritics (harakat) and tatweel. It also converts Arabic-Indic and Persian digits and the Arabic
+  decimal mark (٫), applies NFKC, lowercases Latin text and splits number/script boundaries
+  (`400مل` → `400 مل`). Every step can be switched off in `NormalizerConfig`.
+- **Size and quantity parsing.** It recognizes Arabic and English unit spellings for volume (مل/ملل/مللي/ml, لتر/l,
+  oz/fl oz/أونصة), mass (جم/ج/غرام/g, كجم/kg, ملغ/mg, باوند/lb), counts (قطعة/حبة/كبسولة/قرص/منديل/كيس/pcs) and
+  length (ملم/سم/م/mm/cm/m), plus pack multipliers (`2×50 مل`, `10x6مل`). Battery capacity (`7000 ملي امبير`) is
+  not mistaken for volume. It converts each size to a
+  base unit (ml, g, pcs, mm) and gives it a unit-independent `match_key` (400 ml ≡ 13.5 fl oz ≡ 0.4 l).
+- **Alternative keywords.** It builds a clean keyword list for query expansion. Numbers, unit words (e.g. `مل`),
+  special characters and empty strings are removed. Hyphenated words are both split and joined
+  (`anti-aging` → `anti`, `aging`, `antiaging`), and article-stripped forms are added (`للشعر` → `شعر`, `بالصبار` → `صبار`).
+- **CLI and batch mode** with JSON / JSON Lines output.
+
+## Install
+
+```bash
+pip install git+https://github.com/mtanzil6/souqnorm.git@v0.1.0
+```
+
+For development:
+
+```bash
+git clone https://github.com/mtanzil6/souqnorm.git
+cd souqnorm
+pip install -e ".[dev]"
+pytest
+```
+
+## Usage
+
+```python
+from souqnorm import process
+
+r = process("شامبـــو للشعر الجاف ٤٠٠مل")
+r.normalized                  # 'شامبو للشعر الجاف 400 مل'
+r.quantities[0].label         # '400 ml'
+r.quantities[0].match_key()   # 'qty:400ml'
+r.text_without_quantities     # 'شامبو للشعر الجاف'
+r.keywords                    # ['شامبو', 'للشعر', 'شعر', 'الجاف', 'جاف']
+```
+
+```bash
+souqnorm "عطر ٣٫٤ fl oz"
+souqnorm --file queries.txt > normalized.jsonl
+```
+
+Custom configuration:
+
+```python
+from souqnorm import NormalizerConfig, QueryNormalizer
+qn = QueryNormalizer(NormalizerConfig(taa_marbuta_to_haa=False), strip_article=False)
+```
+
+## Reproducing the evaluation
+
+```bash
+python benchmarks/build_benchmark.py      # synthetic catalog + perturbed queries (seeded)
+python benchmarks/run_benchmark.py        # writes benchmarks/results/results.json
+```
+
+To evaluate on a real catalog (not distributed here), run:
+
+```bash
+pip install openpyxl
+python benchmarks/real_titles.py path/to/titles.xlsx   # writes benchmarks/results/real_titles.json
+```
+
+The `build_benchmark.py` retrieval benchmark is **synthetic**. It uses fictional brands and template titles, and its queries are made by
+applying common Arabic spelling variations. It shows how each normalization step affects results, but it is not a
+substitute for evaluation on real query logs.
+
+## Known limitations
+
+- Article stripping is a heuristic and can wrongly shorten transliterated brand names that start with فال/بال
+  (it only adds variants and keeps the original keyword).
+- `oz` is treated as fluid ounces (volume), the usual meaning in beauty retail.
+- There is no spelling correction, stemming beyond articles, or Arabizi (Latin-script Arabic) transliteration yet.
+
+## License
+
+MIT © 2026 Muhammad Tanzil

souqnorm-0.1.0/README.md

@@ -0,0 +1,93 @@
+# souqnorm
+
+**Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries.**
+
+Shoppers in Arabic-speaking markets type the same product many ways: `شامبو ٤٠٠مل`, `شامبو 400 ملل`,
+`شاـــمبو 400ml`, `shampoo 13.5 fl oz`. Off-the-shelf search analyzers treat each of these as different
+text. `souqnorm` is a small, dependency-free Python library that turns these forms into one consistent
+representation before indexing or querying, whatever search engine you use.
+
+## Features (v0.1)
+
+- **Arabic orthographic normalization.** It unifies alef forms (أ إ آ ٱ → ا), maps ى → ي, ة → ه and ؤ/ئ → و/ي,
+  and strips diacritics (harakat) and tatweel. It also converts Arabic-Indic and Persian digits and the Arabic
+  decimal mark (٫), applies NFKC, lowercases Latin text and splits number/script boundaries
+  (`400مل` → `400 مل`). Every step can be switched off in `NormalizerConfig`.
+- **Size and quantity parsing.** It recognizes Arabic and English unit spellings for volume (مل/ملل/مللي/ml, لتر/l,
+  oz/fl oz/أونصة), mass (جم/ج/غرام/g, كجم/kg, ملغ/mg, باوند/lb), counts (قطعة/حبة/كبسولة/قرص/منديل/كيس/pcs) and
+  length (ملم/سم/م/mm/cm/m), plus pack multipliers (`2×50 مل`, `10x6مل`). Battery capacity (`7000 ملي امبير`) is
+  not mistaken for volume. It converts each size to a
+  base unit (ml, g, pcs, mm) and gives it a unit-independent `match_key` (400 ml ≡ 13.5 fl oz ≡ 0.4 l).
+- **Alternative keywords.** It builds a clean keyword list for query expansion. Numbers, unit words (e.g. `مل`),
+  special characters and empty strings are removed. Hyphenated words are both split and joined
+  (`anti-aging` → `anti`, `aging`, `antiaging`), and article-stripped forms are added (`للشعر` → `شعر`, `بالصبار` → `صبار`).
+- **CLI and batch mode** with JSON / JSON Lines output.
+
+## Install
+
+```bash
+pip install git+https://github.com/mtanzil6/souqnorm.git@v0.1.0
+```
+
+For development:
+
+```bash
+git clone https://github.com/mtanzil6/souqnorm.git
+cd souqnorm
+pip install -e ".[dev]"
+pytest
+```
+
+## Usage
+
+```python
+from souqnorm import process
+
+r = process("شامبـــو للشعر الجاف ٤٠٠مل")
+r.normalized                  # 'شامبو للشعر الجاف 400 مل'
+r.quantities[0].label         # '400 ml'
+r.quantities[0].match_key()   # 'qty:400ml'
+r.text_without_quantities     # 'شامبو للشعر الجاف'
+r.keywords                    # ['شامبو', 'للشعر', 'شعر', 'الجاف', 'جاف']
+```
+
+```bash
+souqnorm "عطر ٣٫٤ fl oz"
+souqnorm --file queries.txt > normalized.jsonl
+```
+
+Custom configuration:
+
+```python
+from souqnorm import NormalizerConfig, QueryNormalizer
+qn = QueryNormalizer(NormalizerConfig(taa_marbuta_to_haa=False), strip_article=False)
+```
+
+## Reproducing the evaluation
+
+```bash
+python benchmarks/build_benchmark.py      # synthetic catalog + perturbed queries (seeded)
+python benchmarks/run_benchmark.py        # writes benchmarks/results/results.json
+```
+
+To evaluate on a real catalog (not distributed here), run:
+
+```bash
+pip install openpyxl
+python benchmarks/real_titles.py path/to/titles.xlsx   # writes benchmarks/results/real_titles.json
+```
+
+The `build_benchmark.py` retrieval benchmark is **synthetic**. It uses fictional brands and template titles, and its queries are made by
+applying common Arabic spelling variations. It shows how each normalization step affects results, but it is not a
+substitute for evaluation on real query logs.
+
+## Known limitations
+
+- Article stripping is a heuristic and can wrongly shorten transliterated brand names that start with فال/بال
+  (it only adds variants and keeps the original keyword).
+- `oz` is treated as fluid ounces (volume), the usual meaning in beauty retail.
+- There is no spelling correction, stemming beyond articles, or Arabizi (Latin-script Arabic) transliteration yet.
+
+## License
+
+MIT © 2026 Muhammad Tanzil

souqnorm-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "souqnorm"
+version = "0.1.0"
+description = "Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Muhammad Tanzil", email = "mtanzil666@gmail.com" }]
+keywords = ["arabic", "nlp", "e-commerce", "search", "query normalization", "information retrieval"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Natural Language :: Arabic",
+  "Topic :: Text Processing :: Linguistic",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/mtanzil6/souqnorm"
+Issues = "https://github.com/mtanzil6/souqnorm/issues"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[project.scripts]
+souqnorm = "souqnorm.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+addopts = "--doctest-modules"
+testpaths = ["tests", "src"]

souqnorm-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

souqnorm-0.1.0/src/souqnorm/__init__.py

@@ -0,0 +1,8 @@
+"""souqnorm: normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries."""
+from .arabic import NormalizerConfig, normalize_text
+from .pipeline import NormalizedQuery, QueryNormalizer, process
+from .units import UNIT_LEXICON, Quantity, UnitParser
+
+__version__ = "0.1.0"
+__all__ = ["NormalizerConfig", "normalize_text", "NormalizedQuery", "QueryNormalizer", "process",
+           "UNIT_LEXICON", "Quantity", "UnitParser", "__version__"]

souqnorm-0.1.0/src/souqnorm/arabic.py

@@ -0,0 +1,74 @@
+"""Character-level normalization for Arabic/English product search queries."""
+from __future__ import annotations
+
+import re
+import unicodedata
+from dataclasses import dataclass
+
+# Harakat, Quranic annotation marks, superscript alef.
+_DIACRITICS = re.compile(r"[ؐ-ًؚ-ٰٟۖ-ۭ]")
+_TATWEEL = "ـ"
+
+_ALEF = str.maketrans({"أ": "ا", "إ": "ا", "آ": "ا", "ٱ": "ا"})
+_ALEF_MAQSURA = str.maketrans({"ى": "ي"})
+_TAA_MARBUTA = str.maketrans({"ة": "ه"})
+_HAMZA_CARRIERS = str.maketrans({"ؤ": "و", "ئ": "ي"})
+_DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "0123456789" * 2)
+_PUNCT = str.maketrans({"٫": ".", "٬": ",", "،": ",", "؛": ";", "؟": "?", "٪": "%", "×": "x"})
+
+_DECIMAL_COMMA = re.compile(r"(?<=\d),(?=\d{1,2}\b)")
+_WS = re.compile(r"\s+")
+
+AR_RANGE = "؀-ۿݐ-ݿࢠ-ࣿ"
+
+
+@dataclass(frozen=True)
+class NormalizerConfig:
+    """Switches for each normalization step (all on by default)."""
+
+    unicode_nfkc: bool = True
+    digits: bool = True
+    punctuation: bool = True
+    remove_diacritics: bool = True
+    remove_tatweel: bool = True
+    unify_alef: bool = True
+    alef_maqsura_to_yaa: bool = True
+    taa_marbuta_to_haa: bool = True
+    hamza_carriers: bool = True
+    lowercase: bool = True
+    split_scripts: bool = True  # "50مل" -> "50 مل", "spf50" -> "spf 50"
+
+
+_SPLIT_DIGIT = re.compile(r"(?<=\d)(?=[^\d\s.,%])|(?<=[^\d\s.,])(?=\d)")
+_SPLIT_SCRIPT = re.compile(rf"(?<=[{AR_RANGE}])(?=[a-z])|(?<=[a-z])(?=[{AR_RANGE}])")
+
+
+def normalize_text(text: str, config: NormalizerConfig | None = None) -> str:
+    """Return a normalized form of *text* suitable for indexing and matching."""
+    c = config or NormalizerConfig()
+    s = text
+    if c.unicode_nfkc:
+        s = unicodedata.normalize("NFKC", s)
+    if c.digits:
+        s = s.translate(_DIGITS)
+    if c.punctuation:
+        s = s.translate(_PUNCT)
+        s = _DECIMAL_COMMA.sub(".", s)
+    if c.remove_diacritics:
+        s = _DIACRITICS.sub("", s)
+    if c.remove_tatweel:
+        s = s.replace(_TATWEEL, "")
+    if c.unify_alef:
+        s = s.translate(_ALEF)
+    if c.alef_maqsura_to_yaa:
+        s = s.translate(_ALEF_MAQSURA)
+    if c.taa_marbuta_to_haa:
+        s = s.translate(_TAA_MARBUTA)
+    if c.hamza_carriers:
+        s = s.translate(_HAMZA_CARRIERS)
+    if c.lowercase:
+        s = s.lower()
+    if c.split_scripts:
+        s = _SPLIT_DIGIT.sub(" ", s)
+        s = _SPLIT_SCRIPT.sub(" ", s)
+    return _WS.sub(" ", s).strip()

souqnorm-0.1.0/src/souqnorm/cli.py

@@ -0,0 +1,38 @@
+"""Command-line interface: `souqnorm "query"` or `souqnorm --file queries.txt`."""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from .arabic import NormalizerConfig
+from .pipeline import QueryNormalizer
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(prog="souqnorm", description="Normalize Arabic/English product search queries.")
+    p.add_argument("query", nargs="*", help="query text (omit when using --file)")
+    p.add_argument("-f", "--file", help="read one query per line ('-' for stdin); writes JSON Lines")
+    p.add_argument("--keep-taa-marbuta", action="store_true", help="do not map ة to ه")
+    p.add_argument("--keep-hamza", action="store_true", help="do not map ؤ/ئ to و/ي")
+    p.add_argument("--no-article-strip", action="store_true", help="do not add ال-stripped keywords")
+    a = p.parse_args(argv)
+
+    cfg = NormalizerConfig(taa_marbuta_to_haa=not a.keep_taa_marbuta, hamza_carriers=not a.keep_hamza)
+    qn = QueryNormalizer(cfg, strip_article=not a.no_article_strip)
+
+    if a.file:
+        fh = sys.stdin if a.file == "-" else open(a.file, encoding="utf-8")
+        with fh:
+            for line in fh:
+                if line.strip():
+                    print(json.dumps(qn.process(line.strip()).to_dict(), ensure_ascii=False))
+        return 0
+    if not a.query:
+        p.error("provide a query or --file")
+    print(json.dumps(qn.process(" ".join(a.query)).to_dict(), ensure_ascii=False, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

souqnorm-0.1.0/src/souqnorm/pipeline.py

@@ -0,0 +1,120 @@
+"""End-to-end query processing: normalize -> extract quantities -> generate keywords and variants."""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+
+from .arabic import AR_RANGE, NormalizerConfig, normalize_text
+from .units import Quantity, UnitParser
+
+_TOKEN = re.compile(rf"[{AR_RANGE}a-z0-9]+(?:[-'.][{AR_RANGE}a-z0-9]+)*")
+_NUMERIC = re.compile(r"^\d+(?:\.\d+)?$")
+# definite-article forms, longest first: wa-al, bi-al, ka-al, fa-al, li-l, al
+_ARTICLE_PREFIXES = ("وال", "بال", "كال", "فال", "لل", "ال")
+# Function words dropped from keywords. "او" is deliberately kept: in catalogs it is usually "eau" (او دو بارفيوم).
+DEFAULT_STOPWORDS = frozenset({"من", "في", "مع", "علي", "الي", "عن", "لـ", "و", "ب", "ل", "الذي", "التي",
+                               "for", "with", "and", "of", "the", "by", "from", "in", "to", "a", "an"})
+
+
+@dataclass
+class NormalizedQuery:
+    original: str
+    normalized: str
+    tokens: list[str]
+    quantities: list[Quantity]
+    text_without_quantities: str
+    keywords: list[str]
+    variants: list[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict:
+        return {
+            "original": self.original,
+            "normalized": self.normalized,
+            "tokens": self.tokens,
+            "quantities": [q.to_dict() for q in self.quantities],
+            "text_without_quantities": self.text_without_quantities,
+            "keywords": self.keywords,
+            "variants": self.variants,
+        }
+
+
+def _dedupe(items):
+    seen, out = set(), []
+    for x in items:
+        if x and x not in seen:
+            seen.add(x)
+            out.append(x)
+    return out
+
+
+class QueryNormalizer:
+    """Main entry point.
+
+    >>> qn = QueryNormalizer()
+    >>> r = qn.process("شامبو للشعر الجاف ٤٠٠مل")
+    >>> r.normalized
+    'شامبو للشعر الجاف 400 مل'
+    >>> r.quantities[0].label
+    '400 ml'
+    """
+
+    def __init__(self, config: NormalizerConfig | None = None, strip_article: bool = True,
+                 min_keyword_len: int = 2, stopwords: frozenset[str] | None = DEFAULT_STOPWORDS):
+        self.config = config or NormalizerConfig()
+        self.units = UnitParser(self.config)
+        self.strip_article = strip_article
+        self.min_keyword_len = min_keyword_len
+        self._unit_words = self.units.unit_words
+        self.stopwords = frozenset(normalize_text(w, self.config) for w in (stopwords or ()))
+
+    def normalize(self, text: str) -> str:
+        return normalize_text(text, self.config)
+
+    def tokenize(self, normalized: str) -> list[str]:
+        return _TOKEN.findall(normalized)
+
+    def keywords(self, tokens: list[str]) -> list[str]:
+        """Alternative keywords: drops numbers, unit words (e.g. 'مل'), stopwords, special characters
+        and empty strings; splits and joins hyphenated words; optionally adds article-stripped forms."""
+        out: list[str] = []
+        for tok in tokens:
+            parts = [p for p in re.split(r"[-'.]", tok) if p]
+            candidates = parts + (["".join(parts)] if len(parts) > 1 else [])
+            for c in candidates:
+                if _NUMERIC.match(c) or c.isdigit() or c in self._unit_words:
+                    continue
+                if len(c) < self.min_keyword_len or c in self.stopwords:
+                    continue
+                out.append(c)
+                if self.strip_article:
+                    for pre in _ARTICLE_PREFIXES:
+                        if c.startswith(pre) and len(c) - len(pre) >= 3:
+                            out.append(c[len(pre):])
+                            break
+        return _dedupe(out)
+
+    def process(self, text: str) -> NormalizedQuery:
+        norm = self.normalize(text)
+        quantities = self.units.parse(norm)
+        stripped = norm
+        for q in reversed(quantities):
+            stripped = stripped[: q.start] + " " + stripped[q.end:]
+        stripped = re.sub(r"\s+", " ", stripped).strip()
+        tokens = self.tokenize(norm)
+        kws = self.keywords(self.tokenize(stripped))
+        canonical = stripped
+        if quantities:
+            canonical = (stripped + " " + " ".join(q.label for q in quantities)).strip()
+        variants = _dedupe([norm, canonical, stripped, " ".join(kws)])
+        return NormalizedQuery(text, norm, tokens, quantities, stripped, kws, variants)
+
+
+_default: QueryNormalizer | None = None
+
+
+def process(text: str) -> NormalizedQuery:
+    """Process *text* with a shared default :class:`QueryNormalizer`."""
+    global _default
+    if _default is None:
+        _default = QueryNormalizer()
+    return _default.process(text)

souqnorm-0.1.0/src/souqnorm/units.py

@@ -0,0 +1,104 @@
+"""Extraction and canonicalization of size/quantity expressions (e.g. '50 مل', '1.7oz', '2x100ml')."""
+from __future__ import annotations
+
+import math
+import re
+from dataclasses import asdict, dataclass
+
+from .arabic import NormalizerConfig, normalize_text
+
+# canonical unit -> (base unit, factor to base, surface forms in natural spelling)
+UNIT_LEXICON: dict[str, tuple[str, float, tuple[str, ...]]] = {
+    "ml": ("ml", 1.0, ("ml", "mls", "milliliter", "milliliters", "millilitre", "millilitres",
+                       "مل", "ملل", "مللي", "ملي", "مليلتر", "ملليلتر", "ميليلتر")),
+    "l": ("ml", 1000.0, ("l", "ltr", "ltrs", "liter", "liters", "litre", "litres", "لتر", "لترات", "ليتر")),
+    "fl_oz": ("ml", 29.5735, ("fl oz", "fl. oz", "fl.oz", "floz")),
+    "oz": ("ml", 29.5735, ("oz", "ounce", "ounces", "أونصة", "اونصة", "أوقية", "اوقية")),
+    "g": ("g", 1.0, ("g", "gm", "gms", "gr", "grm", "gram", "grams", "gramme",
+                     "غ", "غم", "غرام", "جم", "جرام", "قرام", "ج")),
+    "kg": ("g", 1000.0, ("kg", "kgs", "kilo", "kilogram", "kilograms",
+                         "كغ", "كجم", "كلغ", "كيلو", "كيلوغرام", "كيلوجرام")),
+    "mg": ("g", 0.001, ("mg", "ملغ", "ملجم", "مجم", "ملغم", "مليغرام", "مليجرام")),
+    "lb": ("g", 453.592, ("lb", "lbs", "pound", "pounds", "باوند", "رطل")),
+    "pcs": ("pcs", 1.0, ("pc", "pcs", "piece", "pieces", "count", "ct", "capsules", "caps", "tablets", "tabs",
+                         "sachets", "patches", "ampoules", "wipes", "pads",
+                         "قطعة", "قطع", "حبة", "حبات", "كبسولة", "كبسولات", "قرص", "أقراص", "منديل", "مناديل",
+                         "كيس", "أكياس", "لصقة", "لصقات", "أمبولة", "أمبولات", "وحدة", "وحدات", "قطنة",
+                         "شريحة", "شرائح", "ظرف", "أظرف")),
+    # length (dimensions of tools, cables, towels); multipliers are read as dimensions, not packs
+    "mm": ("mm", 1.0, ("mm", "ملم", "مم", "مليمتر", "ملليمتر")),
+    "cm": ("mm", 10.0, ("cm", "سم", "سنتيمتر", "سنتي")),
+    "m": ("mm", 1000.0, ("m", "م", "متر", "meter", "meters", "metre", "metres")),
+}
+
+_LENGTH = {"mm", "cm", "m"}
+
+
+@dataclass(frozen=True)
+class Quantity:
+    value: float          # numeric value as written (per item)
+    unit: str             # canonical unit key, e.g. "ml", "fl_oz"
+    multiplier: int       # pack multiplier, 1 if absent ("2x50ml" -> 2)
+    base_value: float     # value * factor * multiplier in the base unit
+    base_unit: str        # "ml", "g", "pcs" or "mm"
+    text: str             # matched surface text (in normalized query)
+    start: int
+    end: int
+
+    @property
+    def label(self) -> str:
+        unit = "fl oz" if self.unit == "fl_oz" else self.unit
+        core = f"{self.value:g} {unit}"
+        return f"{self.multiplier} x {core}" if self.multiplier > 1 else core
+
+    def match_key(self, sig: int = 2) -> str:
+        """Unit-independent key for matching sizes across unit systems, rounding the base value to
+        *sig* significant figures (e.g. 400 ml and 13.5 fl oz -> 'qty:400ml')."""
+        v = self.base_value
+        if v > 0:
+            v = round(v, sig - 1 - int(math.floor(math.log10(abs(v)))))
+        return f"qty:{v:g}{self.base_unit}"
+
+    def to_dict(self) -> dict:
+        d = asdict(self)
+        d["match_key"] = self.match_key()
+        d["label"] = self.label
+        return d
+
+
+class UnitParser:
+    """Finds quantity expressions in text that was produced by :func:`normalize_text`."""
+
+    def __init__(self, config: NormalizerConfig | None = None):
+        self.config = config or NormalizerConfig()
+        self._surface_to_unit: dict[str, str] = {}
+        for unit, (_, _, forms) in UNIT_LEXICON.items():
+            for form in forms:
+                self._surface_to_unit[normalize_text(form, self.config)] = unit
+        alts = sorted(self._surface_to_unit, key=len, reverse=True)
+        unit_re = "|".join(re.escape(a) for a in alts)
+        self._pattern = re.compile(
+            rf"(?:(?P<mult>\d+)\s*[x*]\s*)?(?P<num>\d+(?:\.\d+)?)\s*(?P<unit>{unit_re})(?![\w])"
+            r"(?!\s*(?:امبير|امب|amp|ampere|a\b|h\b))"  # "7000 ملي امبير" / "mAh" is battery capacity
+        )
+
+    @property
+    def unit_words(self) -> frozenset[str]:
+        """Normalized measurement-unit tokens (volume, mass, length), e.g. {'مل', 'ml', 'جم', ...}.
+        Count nouns such as 'مناديل' or 'امبوله' are excluded because they also name products."""
+        return frozenset(w for s, u in self._surface_to_unit.items() if UNIT_LEXICON[u][0] != "pcs"
+                         for w in s.split())
+
+    def parse(self, normalized: str) -> list[Quantity]:
+        out: list[Quantity] = []
+        for m in self._pattern.finditer(normalized):
+            unit = self._surface_to_unit[m.group("unit")]
+            base_unit, factor, _ = UNIT_LEXICON[unit]
+            value = float(m.group("num"))
+            mult = int(m.group("mult")) if m.group("mult") else 1
+            start = m.start()
+            if unit in _LENGTH and mult > 1:  # "38*29.5 سم" is a size, not a pack of 38
+                mult, start = 1, m.start("num")
+            out.append(Quantity(value, unit, mult, round(value * factor * mult, 3), base_unit,
+                                normalized[start:m.end()], start, m.end()))
+        return out

souqnorm-0.1.0/src/souqnorm.egg-info/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.4
+Name: souqnorm
+Version: 0.1.0
+Summary: Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries
+Author-email: Muhammad Tanzil <mtanzil666@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/mtanzil6/souqnorm
+Project-URL: Issues, https://github.com/mtanzil6/souqnorm/issues
+Keywords: arabic,nlp,e-commerce,search,query normalization,information retrieval
+Classifier: Programming Language :: Python :: 3
+Classifier: Natural Language :: Arabic
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# souqnorm
+
+**Normalization, size parsing and keyword expansion for Arabic/English e-commerce search queries.**
+
+Shoppers in Arabic-speaking markets type the same product many ways: `شامبو ٤٠٠مل`, `شامبو 400 ملل`,
+`شاـــمبو 400ml`, `shampoo 13.5 fl oz`. Off-the-shelf search analyzers treat each of these as different
+text. `souqnorm` is a small, dependency-free Python library that turns these forms into one consistent
+representation before indexing or querying, whatever search engine you use.
+
+## Features (v0.1)
+
+- **Arabic orthographic normalization.** It unifies alef forms (أ إ آ ٱ → ا), maps ى → ي, ة → ه and ؤ/ئ → و/ي,
+  and strips diacritics (harakat) and tatweel. It also converts Arabic-Indic and Persian digits and the Arabic
+  decimal mark (٫), applies NFKC, lowercases Latin text and splits number/script boundaries
+  (`400مل` → `400 مل`). Every step can be switched off in `NormalizerConfig`.
+- **Size and quantity parsing.** It recognizes Arabic and English unit spellings for volume (مل/ملل/مللي/ml, لتر/l,
+  oz/fl oz/أونصة), mass (جم/ج/غرام/g, كجم/kg, ملغ/mg, باوند/lb), counts (قطعة/حبة/كبسولة/قرص/منديل/كيس/pcs) and
+  length (ملم/سم/م/mm/cm/m), plus pack multipliers (`2×50 مل`, `10x6مل`). Battery capacity (`7000 ملي امبير`) is
+  not mistaken for volume. It converts each size to a
+  base unit (ml, g, pcs, mm) and gives it a unit-independent `match_key` (400 ml ≡ 13.5 fl oz ≡ 0.4 l).
+- **Alternative keywords.** It builds a clean keyword list for query expansion. Numbers, unit words (e.g. `مل`),
+  special characters and empty strings are removed. Hyphenated words are both split and joined
+  (`anti-aging` → `anti`, `aging`, `antiaging`), and article-stripped forms are added (`للشعر` → `شعر`, `بالصبار` → `صبار`).
+- **CLI and batch mode** with JSON / JSON Lines output.
+
+## Install
+
+```bash
+pip install git+https://github.com/mtanzil6/souqnorm.git@v0.1.0
+```
+
+For development:
+
+```bash
+git clone https://github.com/mtanzil6/souqnorm.git
+cd souqnorm
+pip install -e ".[dev]"
+pytest
+```
+
+## Usage
+
+```python
+from souqnorm import process
+
+r = process("شامبـــو للشعر الجاف ٤٠٠مل")
+r.normalized                  # 'شامبو للشعر الجاف 400 مل'
+r.quantities[0].label         # '400 ml'
+r.quantities[0].match_key()   # 'qty:400ml'
+r.text_without_quantities     # 'شامبو للشعر الجاف'
+r.keywords                    # ['شامبو', 'للشعر', 'شعر', 'الجاف', 'جاف']
+```
+
+```bash
+souqnorm "عطر ٣٫٤ fl oz"
+souqnorm --file queries.txt > normalized.jsonl
+```
+
+Custom configuration:
+
+```python
+from souqnorm import NormalizerConfig, QueryNormalizer
+qn = QueryNormalizer(NormalizerConfig(taa_marbuta_to_haa=False), strip_article=False)
+```
+
+## Reproducing the evaluation
+
+```bash
+python benchmarks/build_benchmark.py      # synthetic catalog + perturbed queries (seeded)
+python benchmarks/run_benchmark.py        # writes benchmarks/results/results.json
+```
+
+To evaluate on a real catalog (not distributed here), run:
+
+```bash
+pip install openpyxl
+python benchmarks/real_titles.py path/to/titles.xlsx   # writes benchmarks/results/real_titles.json
+```
+
+The `build_benchmark.py` retrieval benchmark is **synthetic**. It uses fictional brands and template titles, and its queries are made by
+applying common Arabic spelling variations. It shows how each normalization step affects results, but it is not a
+substitute for evaluation on real query logs.
+
+## Known limitations
+
+- Article stripping is a heuristic and can wrongly shorten transliterated brand names that start with فال/بال
+  (it only adds variants and keeps the original keyword).
+- `oz` is treated as fluid ounces (volume), the usual meaning in beauty retail.
+- There is no spelling correction, stemming beyond articles, or Arabizi (Latin-script Arabic) transliteration yet.
+
+## License
+
+MIT © 2026 Muhammad Tanzil

souqnorm-0.1.0/src/souqnorm.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+src/souqnorm/__init__.py
+src/souqnorm/arabic.py
+src/souqnorm/cli.py
+src/souqnorm/pipeline.py
+src/souqnorm/units.py
+src/souqnorm.egg-info/PKG-INFO
+src/souqnorm.egg-info/SOURCES.txt
+src/souqnorm.egg-info/dependency_links.txt
+src/souqnorm.egg-info/entry_points.txt
+src/souqnorm.egg-info/requires.txt
+src/souqnorm.egg-info/top_level.txt
+tests/test_souqnorm.py

souqnorm-0.1.0/src/souqnorm.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

souqnorm-0.1.0/src/souqnorm.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+souqnorm = souqnorm.cli:main

souqnorm-0.1.0/src/souqnorm.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[dev]
+pytest>=7

souqnorm-0.1.0/src/souqnorm.egg-info/top_level.txt

@@ -0,0 +1 @@
+souqnorm

souqnorm-0.1.0/tests/test_souqnorm.py

@@ -0,0 +1,136 @@
+import json
+
+import pytest
+
+from souqnorm import NormalizerConfig, QueryNormalizer, normalize_text, process
+from souqnorm.cli import main
+
+
+@pytest.mark.parametrize("raw, expected", [
+    ("أحمر شفاه", "احمر شفاه"),
+    ("إيلاف", "ايلاف"),
+    ("آمن", "امن"),
+    ("مُرَطِّب", "مرطب"),
+    ("الدهنيـــة", "الدهنيه"),
+    ("مستشفى", "مستشفي"),
+    ("لؤلؤ", "لولو"),
+    ("مضيئة", "مضييه"),
+    ("٤٠٠", "400"),
+    ("۵۰", "50"),
+    ("٣٫٤", "3.4"),
+    ("3,4 oz", "3.4 oz"),
+    ("1,000", "1,000"),
+    ("Shampoo  ANTI-Dandruff", "shampoo anti-dandruff"),
+    ("400مل", "400 مل"),
+    ("spf50", "spf 50"),
+    ("كريمcream", "كريم cream"),
+    ("ﻻ", "لا"),  # presentation form via NFKC
+])
+def test_normalize_text(raw, expected):
+    assert normalize_text(raw) == expected
+
+
+def test_config_switches():
+    cfg = NormalizerConfig(taa_marbuta_to_haa=False, unify_alef=False)
+    assert normalize_text("أداة", cfg) == "أداة"
+
+
+@pytest.mark.parametrize("query, label, base", [
+    ("شامبو ٤٠٠مل", "400 ml", 400),
+    ("شامبو 400 ملل", "400 ml", 400),
+    ("شامبو 400 مللي", "400 ml", 400),
+    ("lotion 250ML", "250 ml", 250),
+    ("زيت 1 لتر", "1 l", 1000),
+    ("perfume 3.4 fl oz", "3.4 fl oz", 100.55),
+    ("serum 1.7oz", "1.7 oz", 50.275),
+    ("عطر ١٠٠ مل", "100 ml", 100),
+    ("كريم 50 جم", "50 g", 50),
+    ("powder 1.5kg", "1.5 kg", 1500),
+    ("فيتامين 500 ملغ", "500 mg", 0.5),
+    ("مناديل 3 قطع", "3 pcs", 3),
+    ("2x100ml", "2 x 100 ml", 200),
+    ("٢×٥٠ مل", "2 x 50 ml", 100),
+])
+def test_quantities(query, label, base):
+    qs = process(query).quantities
+    assert len(qs) == 1
+    assert qs[0].label == label
+    assert qs[0].base_value == pytest.approx(base, rel=1e-3)
+
+
+@pytest.mark.parametrize("query", ["spf 50 lotion", "50 غسول", "عطر رقم 5", "vitamin c 20%"])
+def test_no_false_quantities(query):
+    assert process(query).quantities == []
+
+
+def test_keywords_filter_numbers_units_and_hyphens():
+    r = process("كريم anti-aging للبشرة 50 مل")
+    assert "مل" not in r.keywords
+    assert "50" not in r.keywords
+    assert all("-" not in k and " " not in k and k for k in r.keywords)
+    assert {"anti", "aging", "antiaging"} <= set(r.keywords)
+
+
+def test_article_stripping():
+    r = process("شامبو للشعر الجاف بالصبار")
+    assert {"شعر", "جاف", "صبار"} <= set(r.keywords)
+    assert "شعر" not in QueryNormalizer(strip_article=False).process("للشعر").keywords
+
+
+def test_variants_and_text_without_quantities():
+    r = process("شامبو للشعر 400مل")
+    assert r.text_without_quantities == "شامبو للشعر"
+    assert "شامبو للشعر 400 ml" in r.variants
+    assert r.variants[0] == r.normalized
+
+
+def test_to_dict_is_json_serializable():
+    json.dumps(process("عطر 100 مل").to_dict(), ensure_ascii=False)
+
+
+def test_cli_single(capsys):
+    assert main(["عطر", "١٠٠مل"]) == 0
+    out = json.loads(capsys.readouterr().out)
+    assert out["quantities"][0]["label"] == "100 ml"
+
+
+def test_cli_file(tmp_path, capsys):
+    f = tmp_path / "q.txt"
+    f.write_text("عطر 50 مل\n\nشامبو\n", encoding="utf-8")
+    assert main(["--file", str(f)]) == 0
+    lines = capsys.readouterr().out.strip().splitlines()
+    assert len(lines) == 2
+
+
+def test_match_key_cross_unit():
+    a = process("شامبو 400 مل").quantities[0].match_key()
+    b = process("shampoo 13.5 fl oz").quantities[0].match_key()
+    c = process("شامبو 0.4 لتر").quantities[0].match_key()
+    assert a == b == c == "qty:400ml"
+
+
+@pytest.mark.parametrize("query, label", [
+    ("جل شعر - 340ج", "340 g"),
+    ("كبسولات فيتامين - 60 كبسولة", "60 pcs"),
+    ("مناديل - 48 منديل", "48 pcs"),
+    ("بروتين - 1.6 باوند", "1.6 lb"),
+    ("كيبل شحن بطول 1.2م", "1.2 m"),
+    ("مجعد الشعر - 32 ملم", "32 mm"),
+    ("حقيبة - 38*29.5سم", "29.5 cm"),
+    ("أمبولة - 10x6مل", "10 x 6 ml"),
+])
+def test_real_catalog_patterns(query, label):
+    assert [q.label for q in process(query).quantities] == [label]
+
+
+@pytest.mark.parametrize("query", ["بطارية 7000 ملي امبير", "power bank 10000 mah"])
+def test_battery_capacity_not_volume(query):
+    assert process(query).quantities == []
+
+
+def test_stopwords_and_count_nouns_in_keywords():
+    r = process("أمبولة لتساقط الشعر من كيراستاس - 10x6مل")
+    assert "من" not in r.keywords
+    assert "امبوله" in r.keywords
+    assert "مل" not in r.keywords
+    assert "او" in process("عطر او دو بارفيوم").keywords