chunkrank 0.2.0__tar.gz

chunkrank-0.2.0/LICENCE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Amit Nautiyal
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chunkrank-0.2.0/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: chunkrank
+Version: 0.2.0
+Summary: Model-Aware Chunking + Answer Ranking
+License: MIT
+License-File: LICENCE
+Author: Your Name
+Author-email: you@example.com
+Requires-Python: >=3.14,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: deptry (>=0.24.0,<0.25.0)
+Requires-Dist: mypy (>=1.19.1,<2.0.0)
+Requires-Dist: numpy (>=1.26)
+Requires-Dist: rank-bm25 (>=0.2.2)
+Requires-Dist: regex (>=2023.10.3)
+Requires-Dist: ruff (>=0.14.10,<0.15.0)
+Requires-Dist: scikit-learn (>=1.5)
+Description-Content-Type: text/markdown
+
+# ChunkRank: Model-Aware Chunking + Answer Ranking
+
+## Problem
+When using LLMs, text often exceeds the model’s context window.  
+To handle this, text must be **chunked** into pieces that fit the model’s maximum token length.  
+
+Two challenges arise:
+1. **Model-aware chunking**  
+   Each model (OpenAI, Anthropic, Llama, Gemini, t5, Bert, BigBert, LangBert etc.) has a different context length and tokenizer.  
+   Current libraries require users to manually configure chunk sizes; no unified library automatically adapts to the chosen model.
+
+2. **Answer consolidation & ranking**  
+   Once text is chunked, a query may return multiple answers from different chunks.  
+   A **ranking step** is needed to decide the best, most relevant answer.  
+   Existing solutions (e.g., RAG frameworks) combine retrieval + generation, but there’s no standalone library that couples **chunking** and **answer re-ranking**.
+
+---
+
+## Existing Libraries & Gaps
+
+### Chunking
+- **LangChain Text Splitters** → Token-based, works with `tiktoken`, but requires manual chunk size config.  
+- **LlamaIndex `TokenTextSplitter`** → Similar functionality, manual sizing.  
+- **Haystack `PreProcessor`** → Can split by tokens, overlap supported, but not model-aware by default.  
+- **semantic-text-splitter / semchunk** → Standalone, supports tiktoken/HF tokenizers, still needs user-specified chunk length.
+
+**Gap:** None of these libraries automatically map a model → tokenizer → context window → chunk size.
+
+---
+
+### Ranking
+- **pygaggle** (Waterloo CAST) → neural re-ranker.  
+- **Tevatron** → dense retrieval + re-ranking toolkit.  
+- **Pyserini** (with pygaggle) → BM25 + neural re-rankers.  
+- **Haystack, LlamaIndex** → include ranking in RAG pipelines.  
+
+**Gap:** Ranking exists, but **not combined with chunking** in a single, simple package.
+
+---
+
+## What We Want to Build
+A standalone Python library that:
+
+1. **Model-Aware Chunking**
+   - User specifies a model name (e.g., `gpt-4o-mini`, `claude-3.5-sonnet`, `Llama-3.1-8B`).
+   - Library looks up the model’s max context window and tokenizer.
+   - Automatically chunks text into model-compatible pieces with optional overlap and reserve space.
+
+2. **Answer Consolidation & Ranking**
+   - Given multiple answers from chunks, apply a re-ranking step to select the best one.
+   - Should integrate with existing ranking models (cross-encoder, bi-encoder, BM25 + re-ranker).
+   - Should work standalone, without needing a full RAG pipeline.
+
+3. **Unified Workflow**
+   - `chunks = chunkrank.split(text, model="gpt-4o-mini")`
+   - `answers = chunkrank.answer(question, chunks)`
+   - `best = chunkrank.rank(answers)`
+
+---
+
+## Vision
+- Lightweight, model-agnostic utility library.  
+- Bridges the gap between **text preparation** (chunking) and **answer quality** (ranking).  
+- Complements existing RAG frameworks but can also work independently.  
+- Easy to drop into pipelines: preprocessing for QA, summarization, or information extraction.
+
+---
+
+## Next Steps
+1. Build the **model registry** (model → context window + tokenizer).  
+2. Implement **chunking strategies** (tokens, sentences, paragraphs).  
+3. Integrate a **re-ranking engine** (start with Hugging Face cross-encoder).  
+4. Package and release to PyPI with a simple API.  
+
+---
+
+## Community
+
+- [Contributors](CONTRIBUTORS.md)
+- [Maintainers](MAINTAINERS.md)
+- [Contributing Guidelines](CONTRIBUTING.md)
+
+

chunkrank-0.2.0/README.md

@@ -0,0 +1,82 @@
+# ChunkRank: Model-Aware Chunking + Answer Ranking
+
+## Problem
+When using LLMs, text often exceeds the model’s context window.  
+To handle this, text must be **chunked** into pieces that fit the model’s maximum token length.  
+
+Two challenges arise:
+1. **Model-aware chunking**  
+   Each model (OpenAI, Anthropic, Llama, Gemini, t5, Bert, BigBert, LangBert etc.) has a different context length and tokenizer.  
+   Current libraries require users to manually configure chunk sizes; no unified library automatically adapts to the chosen model.
+
+2. **Answer consolidation & ranking**  
+   Once text is chunked, a query may return multiple answers from different chunks.  
+   A **ranking step** is needed to decide the best, most relevant answer.  
+   Existing solutions (e.g., RAG frameworks) combine retrieval + generation, but there’s no standalone library that couples **chunking** and **answer re-ranking**.
+
+---
+
+## Existing Libraries & Gaps
+
+### Chunking
+- **LangChain Text Splitters** → Token-based, works with `tiktoken`, but requires manual chunk size config.  
+- **LlamaIndex `TokenTextSplitter`** → Similar functionality, manual sizing.  
+- **Haystack `PreProcessor`** → Can split by tokens, overlap supported, but not model-aware by default.  
+- **semantic-text-splitter / semchunk** → Standalone, supports tiktoken/HF tokenizers, still needs user-specified chunk length.
+
+**Gap:** None of these libraries automatically map a model → tokenizer → context window → chunk size.
+
+---
+
+### Ranking
+- **pygaggle** (Waterloo CAST) → neural re-ranker.  
+- **Tevatron** → dense retrieval + re-ranking toolkit.  
+- **Pyserini** (with pygaggle) → BM25 + neural re-rankers.  
+- **Haystack, LlamaIndex** → include ranking in RAG pipelines.  
+
+**Gap:** Ranking exists, but **not combined with chunking** in a single, simple package.
+
+---
+
+## What We Want to Build
+A standalone Python library that:
+
+1. **Model-Aware Chunking**
+   - User specifies a model name (e.g., `gpt-4o-mini`, `claude-3.5-sonnet`, `Llama-3.1-8B`).
+   - Library looks up the model’s max context window and tokenizer.
+   - Automatically chunks text into model-compatible pieces with optional overlap and reserve space.
+
+2. **Answer Consolidation & Ranking**
+   - Given multiple answers from chunks, apply a re-ranking step to select the best one.
+   - Should integrate with existing ranking models (cross-encoder, bi-encoder, BM25 + re-ranker).
+   - Should work standalone, without needing a full RAG pipeline.
+
+3. **Unified Workflow**
+   - `chunks = chunkrank.split(text, model="gpt-4o-mini")`
+   - `answers = chunkrank.answer(question, chunks)`
+   - `best = chunkrank.rank(answers)`
+
+---
+
+## Vision
+- Lightweight, model-agnostic utility library.  
+- Bridges the gap between **text preparation** (chunking) and **answer quality** (ranking).  
+- Complements existing RAG frameworks but can also work independently.  
+- Easy to drop into pipelines: preprocessing for QA, summarization, or information extraction.
+
+---
+
+## Next Steps
+1. Build the **model registry** (model → context window + tokenizer).  
+2. Implement **chunking strategies** (tokens, sentences, paragraphs).  
+3. Integrate a **re-ranking engine** (start with Hugging Face cross-encoder).  
+4. Package and release to PyPI with a simple API.  
+
+---
+
+## Community
+
+- [Contributors](CONTRIBUTORS.md)
+- [Maintainers](MAINTAINERS.md)
+- [Contributing Guidelines](CONTRIBUTING.md)
+

chunkrank-0.2.0/chunkrank/__init__.py

@@ -0,0 +1,8 @@
+from .chunker import Chunker, chunk_text
+from .pipeline import ChunkRankPipeline
+
+__all__ = [
+    "Chunker",
+    "chunk_text",
+    "ChunkRankPipeline",
+]

chunkrank-0.2.0/chunkrank/answerers.py

@@ -0,0 +1,37 @@
+from __future__ import annotations
+from dataclasses import dataclass
+from typing import List, Optional, Tuple
+import re
+
+
+@dataclass
+class LocalExtractiveAnswerer:
+    min_overlap: int = 2
+
+    def answer(self, question: str, context: str) -> str:
+        sentences = _split_sentences(context)
+        if not sentences:
+            return ""
+
+        q_words = _norm_words(question)
+        best: Tuple[str, int] = ("", 0)
+
+        for s in sentences:
+            s_words = _norm_words(s)
+            overlap = len(q_words.intersection(s_words))
+            if overlap > best[1]:
+                best = (s, overlap)
+
+        if best[1] < self.min_overlap:
+            return ""
+        return best[0].strip()
+
+
+def _split_sentences(text: str) -> List[str]:
+    parts = re.split(r"(?<=[.!?])\s+", text.strip())
+    return [p.strip() for p in parts if p.strip()]
+
+
+def _norm_words(text: str) -> set[str]:
+    words = re.findall(r"[A-Za-z0-9']+", text.lower())
+    return set(words)

chunkrank-0.2.0/chunkrank/chunker.py

@@ -0,0 +1,93 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import List, Literal, Optional
+
+from .models import get_model_info
+from .tokenizers import build_tokenizer
+
+Strategy = Literal["tokens"]  # keep it simple for now
+
+
+@dataclass
+class ChunkerConfig:
+    model: str
+    strategy: Strategy = "tokens"
+    overlap_tokens: int = 0
+    reserve_tokens: Optional[int] = None
+
+
+class Chunker:
+    def __init__(self, config: ChunkerConfig):
+        info = get_model_info(config.model)
+
+        reserve = config.reserve_tokens if config.reserve_tokens is not None else info.default_reserve
+        self.window = max(1, info.max_context - max(0, reserve))
+
+        self.overlap = max(0, config.overlap_tokens)
+        if self.overlap >= self.window:
+            raise ValueError("overlap_tokens must be < usable window size")
+
+        self.strategy = config.strategy
+        self.tok = build_tokenizer(info.tokenizer, info.tokenizer_id)
+
+    def split(self, text: str) -> List[str]:
+        if not isinstance(text, str) or not text:
+            return []
+
+        if self.strategy != "tokens":
+            raise NotImplementedError("Only 'tokens' strategy is implemented in this version.")
+
+        return list(self._chunk_by_token_budget(text))
+
+    def _chunk_by_token_budget(self, text: str):
+        """
+        Robust approach: grow a slice until token budget reached, then emit slice.
+        Avoids needing tokenizer.decode() (so no None chunks).
+        """
+        start = 0
+        n = len(text)
+
+        # Fast path: already fits
+        if self.tok.count(text) <= self.window:
+            yield text
+            return
+
+        # Character-based upper bound for initial probe (roughly 4 chars/token)
+        approx_chars = max(64, self.window * 4)
+
+        while start < n:
+            end = min(n, start + approx_chars)
+            chunk = text[start:end]
+
+            # If too big, shrink
+            while end > start and self.tok.count(chunk) > self.window:
+                end = start + max(1, (end - start) * 9 // 10)
+                chunk = text[start:end]
+
+            # If somehow cannot shrink (pathological), force a minimal progress
+            if end <= start:
+                end = min(n, start + 200)
+                chunk = text[start:end]
+
+            yield chunk
+
+            if end >= n:
+                break
+
+            # overlap handling (approx char backoff)
+            if self.overlap > 0:
+                backoff_chars = self.overlap * 4
+                start = max(0, end - backoff_chars)
+            else:
+                start = end
+
+
+def chunk_text(
+    text: str,
+    model: str,
+    overlap_tokens: int = 0,
+    reserve_tokens: Optional[int] = None,
+) -> List[str]:
+    cfg = ChunkerConfig(model=model, overlap_tokens=overlap_tokens, reserve_tokens=reserve_tokens)
+    return Chunker(cfg).split(text)

chunkrank-0.2.0/chunkrank/models.py

@@ -0,0 +1,31 @@
+import importlib.resources
+import json
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class ModelInfo:
+    name: str
+    max_context: int
+    tokenizer: Optional[str]
+    tokenizer_id: Optional[str]
+    default_reserve: int = 256
+
+
+def load_registry() -> Dict[str, ModelInfo]:
+    "Loads the model registry from the json file"
+    with importlib.resources.open_text("chunkrank.registry", "model_registry.json") as file:
+        data = json.load(file)
+        return {k: ModelInfo(**v) for k, v in data.items()}
+
+
+def get_model_info(model: str) -> ModelInfo:
+    registry = load_registry()
+    if model in registry:
+        return registry[model]
+    return ModelInfo(model,
+                     128_000,
+                     "tiktoken",
+                     "o200k_base",
+                     512)

chunkrank-0.2.0/chunkrank/pipeline.py

@@ -0,0 +1,14 @@
+from .chunker import Chunker, ChunkerConfig
+from .ranker import Ranker
+
+
+class ChunkRankPipeline:
+    def __init__(self, model: str):
+        self.chunker = Chunker(ChunkerConfig(model=model))
+        self.ranker = Ranker()
+
+    def process(self, question: str, text: str) -> str:
+        chunks = self.chunker.split(text)
+        answers = [f"Answer from chunk {i}" for i, _ in enumerate(chunks, 1)]  # placeholder
+        best = self.ranker.rank(question, answers)[0][0]
+        return best

chunkrank-0.2.0/chunkrank/ranker.py

@@ -0,0 +1,56 @@
+from typing import List, Tuple
+import numpy as np
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+from rank_bm25 import BM25Okapi
+
+
+class Ranker:
+
+    def __init__(self, method: str = "bm25"):
+        self.method = method
+
+    def rank(self, question: str, answers: List[str]) -> List[Tuple[str, float]]:
+        clean = [a for a in answers if isinstance(a, str) and a.strip()]
+        if not clean:
+            return []
+
+        if self.method == "tfidf":
+            return self._rank_tfidf(question, clean)
+        elif self.method == "bm25":
+            return self._rank_bm25(question, clean)
+        else:
+            raise ValueError(f"Unknown ranking method: {self.method}")
+
+    def _rank_tfidf(self, question: str, answers: List[str]) -> List[Tuple[str, float]]:
+        vectorizer = TfidfVectorizer(stop_words="english")
+        corpus = [question] + answers
+        vectors = vectorizer.fit_transform(corpus)
+
+        q_vec = vectors[0]
+        a_vecs = vectors[1:]
+
+        scores = cosine_similarity(q_vec, a_vecs)[0]
+        return sorted(zip(answers, scores), key=lambda x: x[1], reverse=True)
+
+    def _rank_bm25(self, question: str, answers: List[str]) -> List[Tuple[str, float]]:
+        tokenized_answers = [a.split() for a in answers if a and a.strip()]
+        if not tokenized_answers:
+            return []
+        bm25 = BM25Okapi(tokenized_answers)
+
+        q_tokens = question.split()
+        scores = bm25.get_scores(q_tokens)
+        return sorted(zip(answers, scores), key=lambda x: x[1], reverse=True)
+
+    def rank_texts(self, query: str, texts: List[str]) -> List[Tuple[str, float]]:
+        """
+        Rank raw texts (chunks) against a query.
+        """
+        return self.rank(query, texts)
+
+
+def rank_answers(question: str, answers: List[str], method: str = "bm25") -> str:
+    ranker = Ranker(method=method)
+    ranked = ranker.rank(question, answers)
+    return ranked[0][0]

chunkrank-0.2.0/chunkrank/registry/model_registry.json

@@ -0,0 +1,149 @@
+{
+  "gpt-4o-mini": {
+    "name": "gpt-4o-mini",
+    "max_context": 128000,
+    "tokenizer": "tiktoken",
+    "tokenizer_id": "o200k_base",
+    "default_reserve": 512
+  },
+  "claude-3-5-sonnet": {
+    "name": "claude-3-5-sonnet",
+    "max_context": 200000,
+    "tokenizer": "tiktoken",
+    "tokenizer_id": "o200k_base",
+    "default_reserve": 512
+  },
+  "Llama-3.1-8B": {
+    "name": "Llama-3.1-8B",
+    "max_context": 128000,
+    "tokenizer": "hf",
+    "tokenizer_id": "meta-llama/Llama-3.1-8B",
+    "default_reserve": 512
+  },
+  "Llama-3.1-70B": {
+    "name": "Llama-3.1-70B",
+    "max_context": 128000,
+    "tokenizer": "hf",
+    "tokenizer_id": "meta-llama/Llama-3.1-70B",
+    "default_reserve": 1024
+  },
+  "mistral-7b": {
+    "name": "Mistral-7B",
+    "max_context": 32768,
+    "tokenizer": "hf",
+    "tokenizer_id": "mistralai/Mistral-7B-v0.2",
+    "default_reserve": 512
+  },
+  "mixtral-8x7b": {
+    "name": "Mixtral-8x7B",
+    "max_context": 65536,
+    "tokenizer": "hf",
+    "tokenizer_id": "mistralai/Mixtral-8x7B-Instruct",
+    "default_reserve": 1024
+  },
+  "gpt-neo-2.7B": {
+    "name": "GPT-Neo-2.7B",
+    "max_context": 2048,
+    "tokenizer": "hf",
+    "tokenizer_id": "EleutherAI/gpt-neo-2.7B",
+    "default_reserve": 128
+  },
+  "gpt-j-6B": {
+    "name": "GPT-J-6B",
+    "max_context": 4096,
+    "tokenizer": "hf",
+    "tokenizer_id": "EleutherAI/gpt-j-6B",
+    "default_reserve": 256
+  },
+  "bert-base-uncased": {
+    "name": "BERT Base Uncased",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "bert-base-uncased",
+    "default_reserve": 64
+  },
+  "bert-large-uncased": {
+    "name": "BERT Large Uncased",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "bert-large-uncased",
+    "default_reserve": 64
+  },
+  "distilbert-base-uncased": {
+    "name": "DistilBERT Base Uncased",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "distilbert-base-uncased",
+    "default_reserve": 64
+  },
+  "bigbird-roberta-base": {
+    "name": "BigBird RoBERTa Base",
+    "max_context": 4096,
+    "tokenizer": "hf",
+    "tokenizer_id": "google/bigbird-roberta-base",
+    "default_reserve": 256
+  },
+  "bigbird-roberta-large": {
+    "name": "BigBird RoBERTa Large",
+    "max_context": 4096,
+    "tokenizer": "hf",
+    "tokenizer_id": "google/bigbird-roberta-large",
+    "default_reserve": 256
+  },
+  "longformer-base-4096": {
+    "name": "Longformer Base 4096",
+    "max_context": 4096,
+    "tokenizer": "hf",
+    "tokenizer_id": "allenai/longformer-base-4096",
+    "default_reserve": 256
+  },
+  "longformer-large-4096": {
+    "name": "Longformer Large 4096",
+    "max_context": 4096,
+    "tokenizer": "hf",
+    "tokenizer_id": "allenai/longformer-large-4096",
+    "default_reserve": 256
+  },
+  "deberta-v3-base": {
+    "name": "DeBERTa v3 Base",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "microsoft/deberta-v3-base",
+    "default_reserve": 64
+  },
+  "deberta-v3-large": {
+    "name": "DeBERTa v3 Large",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "microsoft/deberta-v3-large",
+    "default_reserve": 64
+  },
+  "t5-base": {
+    "name": "T5 Base",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "t5-base",
+    "default_reserve": 64
+  },
+  "t5-large": {
+    "name": "T5 Large",
+    "max_context": 512,
+    "tokenizer": "hf",
+    "tokenizer_id": "t5-large",
+    "default_reserve": 64
+  },
+  "flan-t5-xl": {
+    "name": "FLAN-T5 XL",
+    "max_context": 2048,
+    "tokenizer": "hf",
+    "tokenizer_id": "google/flan-t5-xl",
+    "default_reserve": 128
+  },
+  "gemini-pro-placeholder": {
+    "name": "Gemini Pro",
+    "max_context": 128000,
+    "tokenizer": "sentencepiece",
+    "tokenizer_id": "google/gemini",
+    "default_reserve": 512
+  }
+}

chunkrank-0.2.0/chunkrank/tokenizers.py

@@ -0,0 +1,40 @@
+
+def _try_importing_tiktoken():
+    try:
+        import tiktoken
+        return tiktoken
+    except ImportError:
+        return None
+
+def _try_importing_transformers():
+    try:
+        import transformers
+        return transformers
+    except ImportError:
+        return None
+
+
+class TokenizerAdapter:
+    def __init__(self, encode_fn: Callable[[str], List[int]]):
+        self._encode = encode_fn
+
+    def encode(self, text: str) -> List[int]:
+        return self._encode(text)
+
+    def count(self, text: str) -> int:
+        return len(self._encode(text))
+
+
+def build_tokenizer(backend: Optional[str], tokenizer_id: Optional[str]) -> TokenizerAdapter:
+    if backend == "tiktoken":
+        tiktoken = _try_importing_tiktoken()
+        if tiktoken:
+            enc = tiktoken.get_encoding(tokenizer_id or "o200k_base")
+            return TokenizerAdapter(lambda s: enc.encode(s, disallowed_special=()))
+    elif backend == "hf":
+        transformers = _try_importing_transformers()
+        if transformers:
+            tok = transformers.AutoTokenizer.from_pretrained(tokenizer_id, use_fast=True)
+            return TokenizerAdapter(lambda s: tok.encode(s, add_special_tokens=False))
+
+    return TokenizerAdapter(lambda s: list(range(len(s) // 4)))

chunkrank-0.2.0/pyproject.toml

@@ -0,0 +1,25 @@
+[project]
+name = "chunkrank"
+version = "0.2.0"
+description = "Model-Aware Chunking + Answer Ranking"
+authors = [
+    { name = "Your Name", email = "you@example.com" }
+]
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.14,<4.0"
+dependencies = [
+    "mypy (>=1.19.1,<2.0.0)",
+    "deptry (>=0.24.0,<0.25.0)",
+    "ruff (>=0.14.10,<0.15.0)",
+    "regex>=2023.10.3",
+    "numpy>=1.26",
+    "scikit-learn>=1.5",
+    "rank-bm25>=0.2.2"
+
+]
+
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"