lexmatch 0.1.0__tar.gz

lexmatch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 janbalitaan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lexmatch-0.1.0/PKG-INFO

@@ -0,0 +1,132 @@
+Metadata-Version: 2.4
+Name: lexmatch
+Version: 0.1.0
+Summary: Dependency-free BM25 question matching with phrase scoring, for when embeddings are more machinery than you need
+Keywords: bm25,search,retrieval,faq,matching,nlp,tfidf
+Author: janbalitaan
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/janbalitaan/lexmatch
+Project-URL: Documentation, https://github.com/janbalitaan/lexmatch#readme
+Project-URL: Source, https://github.com/janbalitaan/lexmatch
+Project-URL: Changelog, https://github.com/janbalitaan/lexmatch/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/janbalitaan/lexmatch/issues
+Description-Content-Type: text/markdown
+
+# lexmatch
+
+BM25 question matching with phrase scoring, and no dependencies.
+
+[![PyPI](https://img.shields.io/pypi/v/lexmatch)](https://pypi.org/project/lexmatch/)
+[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://github.com/janbalitaan/lexmatch/blob/main/pyproject.toml)
+[![CI](https://github.com/janbalitaan/lexmatch/actions/workflows/ci.yml/badge.svg)](https://github.com/janbalitaan/lexmatch/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/janbalitaan/lexmatch/blob/main/LICENSE)
+
+```python
+from lexmatch import Entry, Index
+
+index = Index([
+    Entry("What do you do?", "I build backend services.",
+          aliases=["what is your job", "what is your role"]),
+    Entry("Are you open to remote work?", "Yes, and to relocation.",
+          aliases=["remote", "relocate", "would you move"]),
+])
+
+entry, score = index.answer("what is your role")
+# -> (Entry("What do you do?", ...), 1.0)
+
+index.answer("what is the capital of France")
+# -> (None, 0.0)      below the threshold, so it refuses
+```
+
+## When this beats embeddings
+
+For a few dozen to a few thousand short question/answer pairs where the
+answers are written by hand, a vector database is more machinery than the
+problem deserves — a model to build the embeddings, a store to query them, and
+a deployment that depends on both.
+
+Lexical scoring needs none of that, runs in microseconds, and has a property
+embeddings lack: **you can predict what it will match**, because the math is
+legible. For an FAQ or a help desk, being able to explain why a question
+matched is often worth more than a few points of recall.
+
+Use embeddings when you need paraphrase matching across a large, unseen
+corpus. Use this when you wrote the corpus yourself.
+
+## Two things it adds to plain BM25
+
+**Phrase scoring.** BM25 cannot rank a query made entirely of stopwords —
+`"what do you do?"` — because those words appear in every entry and carry no
+IDF. So phrasing is scored separately, with one guard: a phrase match only
+counts when the substantive words overlap. Without that, `"what is 2+2"`
+scores 0.57 against `"what is your job"` on `"what is"` alone.
+
+**A refusal threshold.** Below `DEFAULT_THRESHOLD` (0.45), `answer()` returns
+`None`. For question answering this matters more than ranking — an off-topic
+question scores 0.2–0.4 from shared filler words, and a confident wrong answer
+is worse than no answer. Scores are scaled to 0–1, so a threshold means the
+same thing whatever the corpus size.
+
+## Aliases do the heavy lifting
+
+```python
+Entry("Are you open to remote work?", "Yes.",
+      aliases=["remote", "relocate", "would you move", "hybrid"])
+```
+
+Listing how people actually ask is the cheapest accuracy you can buy. Aliases
+are weighted as heavily as the question itself.
+
+## Tuning
+
+| Setting | Default | Effect |
+| --- | --- | --- |
+| `threshold` | `0.45` | Higher refuses more. |
+| `TERM_FREQUENCY_SATURATION` | `1.4` | Above this, repeating a term stops helping. |
+| `LENGTH_PENALTY` | `0.72` | How much long entries are discounted. |
+
+## Documentation
+
+| Page | |
+| --- | --- |
+| [Getting started](https://github.com/janbalitaan/lexmatch/blob/main/docs/getting-started.md) | An index, a query, an answer |
+| [How scoring works](https://github.com/janbalitaan/lexmatch/blob/main/docs/scoring.md) | BM25, phrase overlap, and why both |
+| [Refusing to answer](https://github.com/janbalitaan/lexmatch/blob/main/docs/refusing.md) | The threshold, and why a wrong answer is worse |
+| [Replying like a chatbot](https://github.com/janbalitaan/lexmatch/blob/main/docs/replying.md) | Ambiguity, negation, relevance-ranked suggestions |
+| [Composing a reply](https://github.com/janbalitaan/lexmatch/blob/main/docs/composing.md) | One sentence out of several answers |
+| [Aliases](https://github.com/janbalitaan/lexmatch/blob/main/docs/aliases.md) | Where most of the accuracy comes from |
+| [Synonyms](https://github.com/janbalitaan/lexmatch/blob/main/docs/synonyms.md) | A word that means the same thing index-wide |
+| [Compound questions](https://github.com/janbalitaan/lexmatch/blob/main/docs/compound-questions.md) | A message that asks more than one thing |
+| [Typos](https://github.com/janbalitaan/lexmatch/blob/main/docs/typos.md) | Reading a word somebody mistyped |
+| [Small talk](https://github.com/janbalitaan/lexmatch/blob/main/docs/small-talk.md) | Hello, thanks, who are you |
+| [Tuning](https://github.com/janbalitaan/lexmatch/blob/main/docs/tuning.md) | Every number, and when to move it |
+| [API reference](https://github.com/janbalitaan/lexmatch/blob/main/docs/api.md) | Everything importable |
+| [Limitations](https://github.com/janbalitaan/lexmatch/blob/main/docs/limitations.md) | What to use instead, and when |
+
+Runnable [examples](https://github.com/janbalitaan/lexmatch/blob/main/examples) too.
+
+## Install
+
+```bash
+pip install lexmatch
+```
+
+Python 3.10+. No dependencies, so it runs in a serverless function, in CI, or
+on a Raspberry Pi with no install step.
+
+## License
+
+MIT

lexmatch-0.1.0/README.md

@@ -0,0 +1,105 @@
+# lexmatch
+
+BM25 question matching with phrase scoring, and no dependencies.
+
+[![PyPI](https://img.shields.io/pypi/v/lexmatch)](https://pypi.org/project/lexmatch/)
+[![Python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)](https://github.com/janbalitaan/lexmatch/blob/main/pyproject.toml)
+[![CI](https://github.com/janbalitaan/lexmatch/actions/workflows/ci.yml/badge.svg)](https://github.com/janbalitaan/lexmatch/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/janbalitaan/lexmatch/blob/main/LICENSE)
+
+```python
+from lexmatch import Entry, Index
+
+index = Index([
+    Entry("What do you do?", "I build backend services.",
+          aliases=["what is your job", "what is your role"]),
+    Entry("Are you open to remote work?", "Yes, and to relocation.",
+          aliases=["remote", "relocate", "would you move"]),
+])
+
+entry, score = index.answer("what is your role")
+# -> (Entry("What do you do?", ...), 1.0)
+
+index.answer("what is the capital of France")
+# -> (None, 0.0)      below the threshold, so it refuses
+```
+
+## When this beats embeddings
+
+For a few dozen to a few thousand short question/answer pairs where the
+answers are written by hand, a vector database is more machinery than the
+problem deserves — a model to build the embeddings, a store to query them, and
+a deployment that depends on both.
+
+Lexical scoring needs none of that, runs in microseconds, and has a property
+embeddings lack: **you can predict what it will match**, because the math is
+legible. For an FAQ or a help desk, being able to explain why a question
+matched is often worth more than a few points of recall.
+
+Use embeddings when you need paraphrase matching across a large, unseen
+corpus. Use this when you wrote the corpus yourself.
+
+## Two things it adds to plain BM25
+
+**Phrase scoring.** BM25 cannot rank a query made entirely of stopwords —
+`"what do you do?"` — because those words appear in every entry and carry no
+IDF. So phrasing is scored separately, with one guard: a phrase match only
+counts when the substantive words overlap. Without that, `"what is 2+2"`
+scores 0.57 against `"what is your job"` on `"what is"` alone.
+
+**A refusal threshold.** Below `DEFAULT_THRESHOLD` (0.45), `answer()` returns
+`None`. For question answering this matters more than ranking — an off-topic
+question scores 0.2–0.4 from shared filler words, and a confident wrong answer
+is worse than no answer. Scores are scaled to 0–1, so a threshold means the
+same thing whatever the corpus size.
+
+## Aliases do the heavy lifting
+
+```python
+Entry("Are you open to remote work?", "Yes.",
+      aliases=["remote", "relocate", "would you move", "hybrid"])
+```
+
+Listing how people actually ask is the cheapest accuracy you can buy. Aliases
+are weighted as heavily as the question itself.
+
+## Tuning
+
+| Setting | Default | Effect |
+| --- | --- | --- |
+| `threshold` | `0.45` | Higher refuses more. |
+| `TERM_FREQUENCY_SATURATION` | `1.4` | Above this, repeating a term stops helping. |
+| `LENGTH_PENALTY` | `0.72` | How much long entries are discounted. |
+
+## Documentation
+
+| Page | |
+| --- | --- |
+| [Getting started](https://github.com/janbalitaan/lexmatch/blob/main/docs/getting-started.md) | An index, a query, an answer |
+| [How scoring works](https://github.com/janbalitaan/lexmatch/blob/main/docs/scoring.md) | BM25, phrase overlap, and why both |
+| [Refusing to answer](https://github.com/janbalitaan/lexmatch/blob/main/docs/refusing.md) | The threshold, and why a wrong answer is worse |
+| [Replying like a chatbot](https://github.com/janbalitaan/lexmatch/blob/main/docs/replying.md) | Ambiguity, negation, relevance-ranked suggestions |
+| [Composing a reply](https://github.com/janbalitaan/lexmatch/blob/main/docs/composing.md) | One sentence out of several answers |
+| [Aliases](https://github.com/janbalitaan/lexmatch/blob/main/docs/aliases.md) | Where most of the accuracy comes from |
+| [Synonyms](https://github.com/janbalitaan/lexmatch/blob/main/docs/synonyms.md) | A word that means the same thing index-wide |
+| [Compound questions](https://github.com/janbalitaan/lexmatch/blob/main/docs/compound-questions.md) | A message that asks more than one thing |
+| [Typos](https://github.com/janbalitaan/lexmatch/blob/main/docs/typos.md) | Reading a word somebody mistyped |
+| [Small talk](https://github.com/janbalitaan/lexmatch/blob/main/docs/small-talk.md) | Hello, thanks, who are you |
+| [Tuning](https://github.com/janbalitaan/lexmatch/blob/main/docs/tuning.md) | Every number, and when to move it |
+| [API reference](https://github.com/janbalitaan/lexmatch/blob/main/docs/api.md) | Everything importable |
+| [Limitations](https://github.com/janbalitaan/lexmatch/blob/main/docs/limitations.md) | What to use instead, and when |
+
+Runnable [examples](https://github.com/janbalitaan/lexmatch/blob/main/examples) too.
+
+## Install
+
+```bash
+pip install lexmatch
+```
+
+Python 3.10+. No dependencies, so it runs in a serverless function, in CI, or
+on a Raspberry Pi with no install step.
+
+## License
+
+MIT

lexmatch-0.1.0/pyproject.toml

@@ -0,0 +1,128 @@
+[project]
+name = "lexmatch"
+version = "0.1.0"
+description = "Dependency-free BM25 question matching with phrase scoring, for when embeddings are more machinery than you need"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = [
+    "bm25",
+    "search",
+    "retrieval",
+    "faq",
+    "matching",
+    "nlp",
+    "tfidf",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Text Processing :: Indexing",
+    "Typing :: Typed",
+]
+dependencies = []
+
+[[project.authors]]
+name = "janbalitaan"
+
+[project.urls]
+Homepage = "https://github.com/janbalitaan/lexmatch"
+Documentation = "https://github.com/janbalitaan/lexmatch#readme"
+Source = "https://github.com/janbalitaan/lexmatch"
+Changelog = "https://github.com/janbalitaan/lexmatch/blob/main/CHANGELOG.md"
+Issues = "https://github.com/janbalitaan/lexmatch/issues"
+
+[build-system]
+requires = ["uv_build>=0.5"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = "src"
+
+[tool.ruff]
+line-length = 96
+target-version = "py310"
+src = [
+    "src",
+    "tests",
+]
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "W",
+    "F",
+    "I",
+    "N",
+    "UP",
+    "B",
+    "A",
+    "C4",
+    "SIM",
+    "TID",
+    "PTH",
+    "RET",
+    "ARG",
+    "PL",
+    "RUF",
+    "S",
+    "D",
+]
+ignore = [
+    "D107",
+    "D203",
+    "D213",
+]
+
+[tool.ruff.lint.per-file-ignores]
+"src/lexmatch/index.py" = ["PLR0911"]
+"src/lexmatch/composer.py" = ["PLR0911"]
+"examples/*" = [
+    "S",
+    "D",
+    "T201",
+    "INP001",
+    "E402",
+]
+"tests/*" = [
+    "S",
+    "PLR2004",
+    "ARG",
+    "D",
+]
+
+[tool.ruff.lint.flake8-tidy-imports]
+ban-relative-imports = "all"
+
+[tool.ruff.lint.pydocstyle]
+convention = "pep257"
+
+[tool.ruff.lint.isort]
+known-first-party = ["lexmatch"]
+force-sort-within-sections = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+addopts = "-q --strict-markers"
+
+[tool.coverage.run]
+source = ["src/lexmatch"]
+branch = true
+
+[dependency-groups]
+dev = [
+    "pytest>=8.3",
+    "pytest-cov>=6.0",
+    "complexipy>=8.1",
+    "ruff>=0.8",
+    "ty>=0.0.1a1",
+]

lexmatch-0.1.0/pyproject.toml.orig

@@ -0,0 +1,87 @@
+[project]
+name = "lexmatch"
+version = "0.1.0"
+description = "Dependency-free BM25 question matching with phrase scoring, for when embeddings are more machinery than you need"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "janbalitaan" }]
+keywords = ["bm25", "search", "retrieval", "faq", "matching", "nlp", "tfidf"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Operating System :: OS Independent",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Text Processing :: Indexing",
+  "Typing :: Typed",
+]
+# No dependencies, and no model. That is the whole point: this runs in a
+# serverless function, in CI, or on a Raspberry Pi with no install step.
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/janbalitaan/lexmatch"
+Documentation = "https://github.com/janbalitaan/lexmatch#readme"
+Source = "https://github.com/janbalitaan/lexmatch"
+Changelog = "https://github.com/janbalitaan/lexmatch/blob/main/CHANGELOG.md"
+Issues = "https://github.com/janbalitaan/lexmatch/issues"
+
+[build-system]
+requires = ["uv_build>=0.5"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-root = "src"
+
+[dependency-groups]
+dev = ["pytest>=8.3", "pytest-cov>=6.0", "complexipy>=8.1", "ruff>=0.8", "ty>=0.0.1a1"]
+
+[tool.ruff]
+line-length = 96
+target-version = "py310"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "N", "UP", "B", "A", "C4", "SIM", "TID", "PTH",
+          "RET", "ARG", "PL", "RUF", "S", "D"]
+ignore = ["D107", "D203", "D213"]
+
+[tool.ruff.lint.per-file-ignores]
+# `reply()` has one branch per way a query can fail, and each returns. An
+# `if/elif` chain assigning to one variable would be the same decisions with
+# a mutable in the middle, and the rule cannot tell a clean dispatch from a
+# tangle.
+"src/lexmatch/index.py" = ["PLR0911"]
+# `compose_reply` has one branch per outcome and each returns a different
+# sentence. An if/elif chain assigning to one variable would be the same
+# decisions with a mutable in the middle.
+"src/lexmatch/composer.py" = ["PLR0911"]
+# Examples are illustrative scripts, with the same exemptions as the tests
+# and for the same reason: they demonstrate rather than implement.
+"examples/*" = ["S", "D", "T201", "INP001", "E402"]
+"tests/*" = ["S", "PLR2004", "ARG", "D"]
+
+[tool.ruff.lint.flake8-tidy-imports]
+ban-relative-imports = "all"
+
+[tool.ruff.lint.pydocstyle]
+convention = "pep257"
+
+[tool.ruff.lint.isort]
+known-first-party = ["lexmatch"]
+force-sort-within-sections = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]
+addopts = "-q --strict-markers"
+
+[tool.coverage.run]
+source = ["src/lexmatch"]
+branch = true

lexmatch-0.1.0/src/lexmatch/__init__.py

@@ -0,0 +1,100 @@
+"""BM25 question matching with phrase scoring, and no dependencies.
+
+Built for the case where a vector database is more machinery than the problem
+deserves: a few dozen to a few thousand short question/answer pairs, where the
+answers are written by hand and the matching has to be predictable.
+
+    from lexmatch import Entry, Index
+
+    index = Index([
+        Entry("What do you do?", "I build backend services.",
+              aliases=["what is your job", "what is your role"]),
+        Entry("Are you open to remote work?", "Yes, and to relocation."),
+    ])
+
+    entry, score = index.answer("what is your role")
+    # -> the first entry, 1.0
+
+Two things it does that plain BM25 does not:
+
+**Phrase scoring.** BM25 cannot rank a query made entirely of stopwords —
+"what do you do?" — because those words appear in every entry and carry no
+IDF. Phrasing is scored separately so such questions still match.
+
+**A refusal threshold.** Below it, `answer()` returns None. For a question
+answering system this matters more than ranking: a confident wrong answer is
+worse than no answer, and an off-topic question scores 0.2-0.4 on shared
+filler words alone.
+
+Scores are scaled to 0..1, so a threshold means the same thing whatever the
+corpus size.
+"""
+
+from lexmatch.composer import Composer
+from lexmatch.constants import (
+    DEFAULT_THRESHOLD,
+    LENGTH_PENALTY,
+    MIN_CONTENT_OVERLAP,
+    TERM_FREQUENCY_SATURATION,
+)
+from lexmatch.contractions import Contractions
+from lexmatch.exceptions import (
+    EntryError,
+    LexmatchError,
+    ParseError,
+    SmallTalkError,
+    SynonymError,
+    TuningError,
+)
+from lexmatch.index import Index
+from lexmatch.models import (
+    Corrections,
+    Entry,
+    Outcome,
+    Reply,
+    Synonyms,
+    Tuning,
+)
+from lexmatch.morphology import Morphology
+from lexmatch.negation import Negation
+from lexmatch.parsers import CsvParser, JsonParser, MarkdownParser
+from lexmatch.questions import QuestionSplitter
+from lexmatch.smalltalk import SmallTalk
+from lexmatch.stemming import Stemmer
+from lexmatch.stopwords import STOPWORDS
+from lexmatch.text import normalize, query_terms
+
+__all__ = [
+    "DEFAULT_THRESHOLD",
+    "LENGTH_PENALTY",
+    "MIN_CONTENT_OVERLAP",
+    "STOPWORDS",
+    "TERM_FREQUENCY_SATURATION",
+    "Composer",
+    "Contractions",
+    "Corrections",
+    "CsvParser",
+    "Entry",
+    "EntryError",
+    "Index",
+    "JsonParser",
+    "LexmatchError",
+    "MarkdownParser",
+    "Morphology",
+    "Negation",
+    "Outcome",
+    "ParseError",
+    "QuestionSplitter",
+    "Reply",
+    "SmallTalk",
+    "SmallTalkError",
+    "Stemmer",
+    "SynonymError",
+    "Synonyms",
+    "Tuning",
+    "TuningError",
+    "__version__",
+    "normalize",
+    "query_terms",
+]
+__version__ = "0.1.0"