raglite-toolkit 1.2.2__tar.gz → 1.3.0__tar.gz

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/ARCHITECTURE.md

@@ -128,17 +128,20 @@ graph TD
 2. Calculates SHA-256 hash of raw document content.
 3. Checks existing `IndexMetadata` in `VectorStore`. The cached index is reused when the index format version, content hash, chunk size, overlap and embedding provider/model all match (URL sources are hashed by their fetched text).
 4. If hash differs or force rebuild requested:
-   - `RecursiveChunker` splits text into word-based chunks (default 500 words, 50-word overlap).
+   - `RecursiveChunker` splits text into word-based chunks (default 500 words, 50-word overlap). In scripts written without spaces (Chinese, Japanese, Thai, ...) each character counts as one word.
    - `EmbeddingFactory` generates normalized vectors for each chunk.
    - `VectorStore.add()` saves chunks and `VectorStore.save_index_metadata()` persists index metadata.
+   - `KeywordIndex.build()` builds the BM25 keyword index from the chunk texts and saves it as `<storeDir>/<namespace>/keyword.json`.
 
 ### 4.2 Retrieval & Answer Synthesis
-1. `doc.search(query, top_k)`:
-   - Embeds query text.
-   - Executes vector search via `VectorStore.search()`, calculating cosine similarity over L2-normalized vectors.
-   - Returns top $K$ `SearchResult` objects.
+1. `doc.search(query, top_k=..., mode=...)`:
+   - `vector` (default): embeds the query and runs `VectorStore.search()` (cosine similarity over L2-normalized vectors).
+   - `keyword`: BM25 over the keyword index (`retrieval/keyword_index.py`), or the store's own `keyword_search()` if it has one.
+   - `hybrid`: both lists (each `candidates` long, the vector list filtered by `scoreThreshold`), merged with Reciprocal Rank Fusion (`retrieval/fusion.py`).
+   - Terms come from the `raglite-v1` tokenizer (`text/tokenizer.py`): NFKC + lowercase words, code identifiers kept whole, character bigrams for CJK and Thai. The TypeScript SDK must produce identical terms and scores; `tests/fixtures/shared/` checks this.
+   - `DocumentCollection` merges every document's vector list and keyword list first and fuses once, because BM25 statistics are per document.
 2. `doc.ask(question, options)`:
-   - Runs `search(question)`.
+   - Runs `search(question)` with the same retrieval options.
    - Constructs context-augmented system/user prompt via `build_prompt()`.
    - Calls `generate_answer()` or `stream_answer()` with selected LLM adapter (OpenAI, Anthropic, Google, Groq, Ollama, etc.).
 
@@ -152,7 +155,7 @@ Built using **FastAPI** framework for async capabilities, automatic OpenAPI docs
 | :--- | :--- | :--- | :--- |
 | `GET` | `/health` | No | Liveness status & total chunk count |
 | `GET` | `/info` | Yes (Bearer token) | Configuration details & index state |
-| `POST` | `/search` | Yes (Bearer token) | Semantic vector search |
+| `POST` | `/search` | Yes (Bearer token) | Vector, keyword or hybrid search (`mode`) |
 | `POST` | `/ask` | Yes (Bearer token) | Context Q&A generation (supports streaming via StreamingResponse) |
 
 ---
@@ -190,4 +193,8 @@ class VectorStore(ABC):
 
     @abstractmethod
     def read_index_metadata(self) -> Optional[IndexMetadata]: ...
+
+    # Optional (return None when unsupported):
+    def list_chunks(self) -> Optional[List[IndexedChunk]]: ...  # rebuild a missing keyword index
+    def keyword_search(self, query: str, top_k: int) -> Optional[List[VectorSearchHit]]: ...  # native keyword search
 ```

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/CHANGELOG.md

@@ -2,7 +2,32 @@
 
 All notable changes to this project will be documented in this file.
 
-## [1.2.2] - Unreleased
+## [1.3.0] - 2026-10-03
+
+### Added
+- **Hybrid search:** `search()`, `ask()` and `ask_stream()` accept `mode="vector" | "keyword" | "hybrid"` (or `{"mode": ...}` in the options dict; default `"vector"`), on `Document` and `DocumentCollection`. Keyword mode uses BM25; hybrid merges vector and keyword results with Reciprocal Rank Fusion, so exact terms such as `ERR_4021`, SKUs or function names are found even when embeddings miss them. Tune it with `hybrid={"rrfK", "candidates", "weights"}`, or set defaults with the new `retrieval` option. Results in these modes carry `scores` (`vector`, `keyword`, `fused`); vector-mode results are unchanged.
+- **Keyword index:** `build()` now also writes a BM25 index (`<storeDir>/<namespace>/keyword.json`) from the chunk texts. It needs no extra embedding calls, works with every vector store, and uses the same file layout as the TypeScript SDK.
+- **Multilingual keyword tokenizer (`raglite-v1`):** NFKC and lowercase normalisation, code identifiers kept whole (`gpt-4.1`, `snake_case`), and character bigrams for Chinese, Japanese, Korean, Thai, Lao, Khmer and Myanmar. Exported as `tokenize()`. It produces exactly the same terms as the TypeScript SDK.
+- **HTTP and CLI:** optional `mode` on `/search` and `/ask`; `--mode` on `raglite search`, `ask` and `serve`. `/info` reports `retrievalMode`.
+- **`VectorStore` extension points (optional):** `list_chunks()` lets keyword and hybrid search rebuild a missing keyword index from the store, and `keyword_search(query, top_k)` replaces the built-in BM25 index. The memory and Qdrant stores implement `list_chunks()`. Existing custom stores need no changes.
+
+### Changed
+- **Chunking of text without spaces:** Chinese, Japanese, Thai, Lao, Khmer and Myanmar text is now chunked by character instead of becoming one giant "word". Previously a document in these scripts became a single chunk of any length, which could exceed embedding model limits. `chunkSize` and `overlap` count characters for these scripts and words for everything else.
+
+### Fixed
+- **Custom chunk sizes are kept:** `build()` without `chunk_size` or `overlap` (in the call or the constructor) now reuses the existing index's values instead of the defaults. Previously `raglite search` or `raglite ask` after `raglite index --chunk-size N` silently re-embedded the whole index at the default 500 words. Defaults still apply to a new index, and explicit values still trigger a rebuild when they differ.
+- **Embedding provider is kept:** with no `embeddings` configured, `build()` now keeps the existing index's provider and model (reusing configured credentials when the provider matches) instead of switching to the local default. The CLI no longer assumes `--embed-provider local` when no `--embed-*` flag is given, so `raglite search` and `raglite ask` reuse an index built with `--embed-provider openai` instead of re-embedding it locally. New indexes still default to local embeddings.
+- **`build(options)` embeddings dict:** `build({"embeddings": {...}})` now accepts a plain dict, as the constructor does.
+- **`raglite serve`:** no longer crashes with `TypeError` when given `--llm-provider` or `--token`.
+- **`DocumentCollection.serve()`:** no longer fails with `ImportError`, so serving a directory works. It now also falls back to the collection's configured `llm`.
+- **`ask()` options:** an explicit `scoreThreshold` of `0` is no longer replaced by the configured default.
+
+### Upgrade notes
+- The index format version is now 2. Indexes whose source contains no Chinese, Japanese, Thai, Lao, Khmer or Myanmar text are upgraded in place on the next `build()`, without re-embedding (the source file is read once to check). Indexes of sources that do contain such text are rebuilt once.
+- Indexes built before 1.3.0 have no keyword index. The memory and Qdrant stores build it from the stored chunks on the first keyword or hybrid search. Pinecone and custom stores without `list_chunks()` log a warning and use vector search until you run `build(rebuild=True)`.
+- With Qdrant or Pinecone, the keyword index lives on local disk under `storeDir`. Keep `storeDir` on persistent storage when you use keyword or hybrid search.
+
+## [1.2.2] - 2026-10-02
 
 ### Changed
 - **Upgrades keep cached indexes:** The build cache is now keyed on an index format version (`formatVersion` in `IndexMetadata`) instead of the package version, so upgrading RAGLite no longer re-embeds every index. Indexes built by 1.2.1 are reused as-is; indexes from older releases are rebuilt once.

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: raglite-toolkit
-Version: 1.2.2
+Version: 1.3.0
 Summary: Build semantic search, multi-provider question answering, and REST APIs over your documents in a few lines of Python.
 Project-URL: Homepage, https://github.com/creatorpiyush/raglite-py
 Project-URL: Repository, https://github.com/creatorpiyush/raglite-py
@@ -59,6 +59,8 @@ Description-Content-Type: text/markdown
 - 🤖 **Multi-provider LLMs** — OpenAI, Anthropic (Claude), Google (Gemini), Mistral, Cohere, Groq, xAI, Ollama
 - 🔢 **Multi-provider embeddings** — OpenAI, Google, Mistral, Cohere, Voyage, Ollama, or a **local** offline sentence-transformer (no API key needed)
 - 📐 **Cosine similarity** scoring with L2-normalized vectors
+- 🔎 **Hybrid search** — BM25 keyword search fused with vector search, for exact terms like error codes and SKUs
+- 🌏 **Any language** — Chinese, Japanese, Korean and Thai text is chunked and keyword-indexed correctly
 - ♻️ **Content-hash cache** — reindexes only when the file actually changes
 - 🗂 **Per-document namespacing** — indexes are isolated, two documents never collide
 - 🌐 **REST API** via FastAPI with optional **bearer-token auth**
@@ -129,6 +131,41 @@ print(answer.text)
 
 ---
 
+## Hybrid Search (Keyword + Vector)
+
+Vector search matches meaning, but it can miss exact terms such as error codes, SKUs, function names or rare product names. Keyword search (BM25) finds those exactly. `hybrid` runs both and merges the results with Reciprocal Rank Fusion.
+
+```python
+doc.search("ERR_4021", mode="hybrid")  # "vector" (default) | "keyword" | "hybrid"
+doc.ask("What does ERR_4021 mean?", {"mode": "hybrid"})
+
+# Or set a default once; it applies to search(), ask() and ask_stream().
+Document("./runbook.md", {
+    "retrieval": {
+        "mode": "hybrid",
+        "hybrid": {"rrfK": 60, "candidates": 50, "weights": {"vector": 1, "keyword": 1}},
+    },
+})
+```
+
+- In `keyword` and `hybrid` modes, `score` is the fused rank score scaled to 0..1 (1 means ranked first by every retriever). Each result also has `scores` with `vector`, `keyword` and `fused`. Vector mode results are unchanged.
+- `scoreThreshold` is still a cosine similarity. In hybrid mode it filters the vector results before fusion; keyword matches are not filtered by it.
+- The keyword index is built by `build()` from the chunk texts, with no extra embedding calls, and saved as `<storeDir>/<namespace>/keyword.json`. It works with every vector store. With Qdrant or Pinecone, keep `storeDir` on persistent disk.
+- Indexes built before 1.3: the memory and Qdrant stores build the keyword index from the stored chunks on the first keyword or hybrid search. Other stores log a warning and use vector search until you run `build(rebuild=True)`.
+- The same options work over HTTP (`"mode"` on `/search` and `/ask`) and in the CLI (`--mode hybrid`).
+
+The tokenizer is the same in the Python and TypeScript SDKs:
+
+| Text | How it is indexed |
+|------|-------------------|
+| Latin, Cyrillic, Greek, Arabic, Devanagari and other spaced scripts | Words, lowercased and NFKC-normalised (`fi` → `fi`, `ＡＢＣ` → `abc`) |
+| Code identifiers | `gpt-4.1`, `snake_case` and `ERR_42` stay whole, and their parts are indexed too |
+| Chinese, Japanese, Korean, Thai, Lao, Khmer, Myanmar | Overlapping character pairs (`退款处理` → `退款`, `款处`, `处理`), so no dictionary is needed |
+
+There is no stemming or stopword list, because both are language-specific: in keyword mode `refund` does not match `refunds`. Hybrid mode's vector side covers those cases.
+
+---
+
 ## Fully Offline — No API Key Needed
 
 ```python
@@ -266,6 +303,7 @@ raglite index ./policy.pdf --embed-provider local
 
 # Semantic search
 raglite search ./docs "refund policy" --top-k 5
+raglite search ./docs "ERR_4021" --mode hybrid
 
 # Ask a question (streaming)
 raglite ask ./docs "What is the refund policy?" \
@@ -313,12 +351,13 @@ raglite serve https://example.com \
 ```python
 Document("./policy.pdf", {
     # Chunking
-    "chunkSize":      500,          # words per chunk (default: 500)
+    "chunkSize":      500,          # words per chunk; characters for Chinese, Japanese, Thai, ... (default: 500)
     "overlap":        50,           # overlapping words between chunks (default: 50)
 
     # Retrieval
     "topK":           5,            # default results returned (default: 5)
     "scoreThreshold": 0.0,          # minimum cosine similarity (0..1, default: 0)
+    "retrieval":      {"mode": "vector"},  # "vector" | "keyword" | "hybrid" (default: vector)
 
     # Storage
     "storeDir":       ".raglite",   # where indexes are persisted (default: .raglite)
@@ -341,9 +380,9 @@ Every `build()` call fingerprints the source (file bytes, or the fetched text fo
 | Factor | Triggers rebuild if changed |
 |--------|-----------------------------|
 | File content | SHA-256 hash differs |
-| Chunk size | `chunkSize` changed |
-| Overlap | `overlap` changed |
-| Embedding provider/model | Provider or model string changed |
+| Chunk size | `chunkSize` changed (when not set, the existing index's value is kept) |
+| Overlap | `overlap` changed (when not set, the existing index's value is kept) |
+| Embedding provider/model | Provider or model string changed (when no `embeddings` are configured, the existing index's are kept) |
 | Index format | Stored index layout changed by a release (rare; ordinary upgrades reuse the index) |
 
 Pass `rebuild=True` to `build()` to force a fresh index regardless.
@@ -362,6 +401,9 @@ from raglite.vectordb.base import VectorStore
 class MyVectorStore(VectorStore):
     # Implement: namespace (property), load, reset, add, search, count,
     #            save_index_metadata, read_index_metadata
+    # Optional:  list_chunks() lets hybrid search rebuild a missing keyword index
+    #            from your store; keyword_search(query, top_k) replaces the
+    #            built-in BM25 index with your own.
     ...
 ```
 

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/README.md

@@ -13,6 +13,8 @@
 - 🤖 **Multi-provider LLMs** — OpenAI, Anthropic (Claude), Google (Gemini), Mistral, Cohere, Groq, xAI, Ollama
 - 🔢 **Multi-provider embeddings** — OpenAI, Google, Mistral, Cohere, Voyage, Ollama, or a **local** offline sentence-transformer (no API key needed)
 - 📐 **Cosine similarity** scoring with L2-normalized vectors
+- 🔎 **Hybrid search** — BM25 keyword search fused with vector search, for exact terms like error codes and SKUs
+- 🌏 **Any language** — Chinese, Japanese, Korean and Thai text is chunked and keyword-indexed correctly
 - ♻️ **Content-hash cache** — reindexes only when the file actually changes
 - 🗂 **Per-document namespacing** — indexes are isolated, two documents never collide
 - 🌐 **REST API** via FastAPI with optional **bearer-token auth**
@@ -83,6 +85,41 @@ print(answer.text)
 
 ---
 
+## Hybrid Search (Keyword + Vector)
+
+Vector search matches meaning, but it can miss exact terms such as error codes, SKUs, function names or rare product names. Keyword search (BM25) finds those exactly. `hybrid` runs both and merges the results with Reciprocal Rank Fusion.
+
+```python
+doc.search("ERR_4021", mode="hybrid")  # "vector" (default) | "keyword" | "hybrid"
+doc.ask("What does ERR_4021 mean?", {"mode": "hybrid"})
+
+# Or set a default once; it applies to search(), ask() and ask_stream().
+Document("./runbook.md", {
+    "retrieval": {
+        "mode": "hybrid",
+        "hybrid": {"rrfK": 60, "candidates": 50, "weights": {"vector": 1, "keyword": 1}},
+    },
+})
+```
+
+- In `keyword` and `hybrid` modes, `score` is the fused rank score scaled to 0..1 (1 means ranked first by every retriever). Each result also has `scores` with `vector`, `keyword` and `fused`. Vector mode results are unchanged.
+- `scoreThreshold` is still a cosine similarity. In hybrid mode it filters the vector results before fusion; keyword matches are not filtered by it.
+- The keyword index is built by `build()` from the chunk texts, with no extra embedding calls, and saved as `<storeDir>/<namespace>/keyword.json`. It works with every vector store. With Qdrant or Pinecone, keep `storeDir` on persistent disk.
+- Indexes built before 1.3: the memory and Qdrant stores build the keyword index from the stored chunks on the first keyword or hybrid search. Other stores log a warning and use vector search until you run `build(rebuild=True)`.
+- The same options work over HTTP (`"mode"` on `/search` and `/ask`) and in the CLI (`--mode hybrid`).
+
+The tokenizer is the same in the Python and TypeScript SDKs:
+
+| Text | How it is indexed |
+|------|-------------------|
+| Latin, Cyrillic, Greek, Arabic, Devanagari and other spaced scripts | Words, lowercased and NFKC-normalised (`fi` → `fi`, `ＡＢＣ` → `abc`) |
+| Code identifiers | `gpt-4.1`, `snake_case` and `ERR_42` stay whole, and their parts are indexed too |
+| Chinese, Japanese, Korean, Thai, Lao, Khmer, Myanmar | Overlapping character pairs (`退款处理` → `退款`, `款处`, `处理`), so no dictionary is needed |
+
+There is no stemming or stopword list, because both are language-specific: in keyword mode `refund` does not match `refunds`. Hybrid mode's vector side covers those cases.
+
+---
+
 ## Fully Offline — No API Key Needed
 
 ```python
@@ -220,6 +257,7 @@ raglite index ./policy.pdf --embed-provider local
 
 # Semantic search
 raglite search ./docs "refund policy" --top-k 5
+raglite search ./docs "ERR_4021" --mode hybrid
 
 # Ask a question (streaming)
 raglite ask ./docs "What is the refund policy?" \
@@ -267,12 +305,13 @@ raglite serve https://example.com \
 ```python
 Document("./policy.pdf", {
     # Chunking
-    "chunkSize":      500,          # words per chunk (default: 500)
+    "chunkSize":      500,          # words per chunk; characters for Chinese, Japanese, Thai, ... (default: 500)
     "overlap":        50,           # overlapping words between chunks (default: 50)
 
     # Retrieval
     "topK":           5,            # default results returned (default: 5)
     "scoreThreshold": 0.0,          # minimum cosine similarity (0..1, default: 0)
+    "retrieval":      {"mode": "vector"},  # "vector" | "keyword" | "hybrid" (default: vector)
 
     # Storage
     "storeDir":       ".raglite",   # where indexes are persisted (default: .raglite)
@@ -295,9 +334,9 @@ Every `build()` call fingerprints the source (file bytes, or the fetched text fo
 | Factor | Triggers rebuild if changed |
 |--------|-----------------------------|
 | File content | SHA-256 hash differs |
-| Chunk size | `chunkSize` changed |
-| Overlap | `overlap` changed |
-| Embedding provider/model | Provider or model string changed |
+| Chunk size | `chunkSize` changed (when not set, the existing index's value is kept) |
+| Overlap | `overlap` changed (when not set, the existing index's value is kept) |
+| Embedding provider/model | Provider or model string changed (when no `embeddings` are configured, the existing index's are kept) |
 | Index format | Stored index layout changed by a release (rare; ordinary upgrades reuse the index) |
 
 Pass `rebuild=True` to `build()` to force a fresh index regardless.
@@ -316,6 +355,9 @@ from raglite.vectordb.base import VectorStore
 class MyVectorStore(VectorStore):
     # Implement: namespace (property), load, reset, add, search, count,
     #            save_index_metadata, read_index_metadata
+    # Optional:  list_chunks() lets hybrid search rebuild a missing keyword index
+    #            from your store; keyword_search(query, top_k) replaces the
+    #            built-in BM25 index with your own.
     ...
 ```
 

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/pyproject.toml

@@ -8,7 +8,7 @@ packages = ["src/raglite"]
 
 [project]
 name = "raglite-toolkit"
-version = "1.2.2"
+version = "1.3.0"
 description = "Build semantic search, multi-provider question answering, and REST APIs over your documents in a few lines of Python."
 readme = "README.md"
 requires-python = ">=3.10"

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/src/raglite/__init__.py

@@ -41,20 +41,34 @@ from .loaders import (
     is_supported_file,
     is_url,
 )
-from .retrieval import Retriever
+from .retrieval import (
+    KeywordHit,
+    KeywordIndex,
+    RankedList,
+    RetrievalPlan,
+    Retriever,
+    reciprocal_rank_fusion,
+    resolve_retrieval_plan,
+)
+from .text import TOKENIZER_NAME, tokenize
 from .types import (
     AnswerResult,
     ChunkMetadata,
     EmbeddingProviderConfig,
     EmbeddingProviderName,
+    HybridOptions,
     IndexMetadata,
     LLMProviderConfig,
     LLMProviderName,
+    RetrievalMode,
+    RetrievalOptions,
     SearchResult,
+    SearchScores,
     StoredChunk,
 )
 from .utils.logger import create_logger
 from .vectordb import MemoryVectorStore
+from .vectordb.base import IndexedChunk, VectorSearchHit, VectorStore
 
 VERSION = PACKAGE_VERSION
 
@@ -78,6 +92,10 @@ __all__ = [
     "ChunkMetadata",
     "StoredChunk",
     "SearchResult",
+    "SearchScores",
+    "RetrievalMode",
+    "RetrievalOptions",
+    "HybridOptions",
     "AnswerResult",
     "IndexMetadata",
     "DocumentOptions",
@@ -97,7 +115,18 @@ __all__ = [
     "BaseChunker",
     "RecursiveChunker",
     "MemoryVectorStore",
+    "VectorStore",
+    "VectorSearchHit",
+    "IndexedChunk",
     "Retriever",
+    "KeywordIndex",
+    "KeywordHit",
+    "RankedList",
+    "RetrievalPlan",
+    "reciprocal_rank_fusion",
+    "resolve_retrieval_plan",
+    "tokenize",
+    "TOKENIZER_NAME",
     "DEFAULT_EMBEDDING_MODELS",
     "LocalEmbedder",
     "RemoteEmbedder",

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/src/raglite/api/schemas.py

@@ -1,4 +1,4 @@
-from typing import Optional
+from typing import Literal, Optional
 
 from pydantic import BaseModel, ConfigDict, Field
 
@@ -11,6 +11,7 @@ class SearchRequest(BaseModel):
     scoreThreshold: Optional[float] = Field(
         default=None, alias="scoreThreshold", ge=-1.0, le=1.0
     )
+    mode: Optional[Literal["vector", "keyword", "hybrid"]] = None
 
 
 class AskRequest(BaseModel):
@@ -21,6 +22,7 @@ class AskRequest(BaseModel):
     scoreThreshold: Optional[float] = Field(
         default=None, alias="scoreThreshold", ge=-1.0, le=1.0
     )
+    mode: Optional[Literal["vector", "keyword", "hybrid"]] = None
     includeCitations: Optional[bool] = Field(
         default=None, alias="includeCitations"
     )

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/src/raglite/api/server.py

@@ -103,6 +103,7 @@ def build_app(document: Any, options: Dict[str, Any]) -> FastAPI:
             "chunkSize": cfg.chunkSize,
             "overlap": cfg.overlap,
             "topK": cfg.topK,
+            "retrievalMode": cfg.retrieval.mode or "vector",
             "embeddings": {
                 "provider": cfg.embeddings.provider,
                 "model": cfg.embeddings.model
@@ -118,6 +119,7 @@ def build_app(document: Any, options: Dict[str, Any]) -> FastAPI:
             req.query,
             top_k=req.topK,
             score_threshold=req.scoreThreshold,
+            mode=req.mode,
         )
         return {
             "results": [r.model_dump(by_alias=True) for r in results]
@@ -133,6 +135,7 @@ def build_app(document: Any, options: Dict[str, Any]) -> FastAPI:
                 "llm": ask_provider,
                 "topK": req.topK,
                 "scoreThreshold": req.scoreThreshold,
+                "mode": req.mode,
                 "includeCitations": req.includeCitations,
             }
 

raglite_toolkit-1.3.0/src/raglite/chunking/recursive.py

@@ -0,0 +1,90 @@
+import re
+from typing import List, Tuple
+
+from ..errors import ChunkingError
+from ..text.scripts import has_unspaced_text, is_mark, is_unspaced_char
+from .base import BaseChunker
+
+# The exact set JavaScript's \s matches. Python's \s differs (it includes
+# \x1c-\x1f and \x85 but not ), which would make the two SDKs chunk the
+# same text differently.
+_JS_WHITESPACE = re.compile(
+    "[\t\n\v\f\r    - 

  　]+"
+)
+
+# One chunking unit: (text, spaced). ``spaced`` is False when the unit
+# continues the previous unit's word, so no space goes between them.
+_Unit = Tuple[str, bool]
+
+
+class RecursiveChunker(BaseChunker):
+    """Word-based recursive chunker with overlap.
+
+    Scripts written without spaces (Chinese, Japanese, Thai, ...) have no word
+    boundaries to split on, so each of their characters counts as one unit.
+    Without this a whole unspaced document would be a single "word" and
+    therefore a single chunk of any length.
+    """
+
+    def split(self, text: str) -> List[str]:
+        if not _JS_WHITESPACE.sub("", text):
+            return []
+        if self.overlap >= self.chunk_size:
+            raise ChunkingError(
+                f"overlap ({self.overlap}) must be smaller than chunkSize ({self.chunk_size})"
+            )
+
+        words = [w for w in _JS_WHITESPACE.split(text) if w]
+        if has_unspaced_text(text):
+            units = [u for w in words for u in _word_units(w)]
+        else:
+            units = [(w, True) for w in words]
+        if len(units) <= self.chunk_size:
+            return [_render(units)]
+
+        step = self.chunk_size - self.overlap
+        chunks: List[str] = []
+
+        start = 0
+        while start < len(units):
+            end = start + self.chunk_size
+            slice_units = units[start:end]
+            if not slice_units:
+                break
+            chunks.append(_render(slice_units))
+            if end >= len(units):
+                break
+            start += step
+
+        return chunks
+
+
+def _word_units(word: str) -> List[_Unit]:
+    """Split a word into units: each unspaced-script character (with its
+    combining marks) is a unit, and every run of other characters is a unit."""
+    units: List[List] = []
+    run = ""
+    for ch in word:
+        if is_mark(ch) and run == "" and units:
+            units[-1][0] += ch
+        elif is_mark(ch):
+            run += ch
+        elif is_unspaced_char(ord(ch)):
+            if run:
+                units.append([run, not units])
+            run = ""
+            units.append([ch, not units])
+        else:
+            run += ch
+    if run:
+        units.append([run, not units])
+    return [(text, spaced) for text, spaced in units]
+
+
+def _render(units: List[_Unit]) -> str:
+    out = ""
+    for i, (text, spaced) in enumerate(units):
+        if i > 0 and spaced:
+            out += " "
+        out += text
+    return out

{raglite_toolkit-1.2.2 → raglite_toolkit-1.3.0}/src/raglite/cli.py

@@ -5,7 +5,7 @@ import sys
 import time
 from typing import Optional, Union
 
-from .constants import PACKAGE_VERSION
+from .constants import DEFAULT_HOST, DEFAULT_PORT, PACKAGE_VERSION
 from .core.collection import DocumentCollection
 from .core.document import Document
 from .loaders import is_url
@@ -14,9 +14,9 @@ HELP = f"""raglite v{PACKAGE_VERSION}
 
 Usage:
   raglite index <path|url>   [--chunk-size N] [--overlap N] [--embed-provider P] [--embed-model M] [--embed-key K] [--rebuild]
-  raglite search <path|url> "query"   [--top-k N]
-  raglite ask <path|url> "question"   --llm-provider P [--llm-model M] [--llm-key K] [--stream]
-  raglite serve <path|url>            --llm-provider P [--llm-key K] [--host H] [--port N] [--token T]
+  raglite search <path|url> "query"   [--top-k N] [--mode vector|keyword|hybrid]
+  raglite ask <path|url> "question"   --llm-provider P [--llm-model M] [--llm-key K] [--stream] [--mode M]
+  raglite serve <path|url>            --llm-provider P [--llm-key K] [--host H] [--port N] [--token T] [--mode M]
   raglite --help
   raglite --version
 
@@ -26,7 +26,13 @@ Providers:
 """
 
 
-def parse_common_embedding(args_dict: dict) -> dict:
+_MODES = ("vector", "keyword", "hybrid")
+
+
+def parse_common_embedding(args_dict: dict) -> Optional[dict]:
+    """None when no --embed-* flag is given, so an existing index keeps its provider."""
+    if not any(args_dict.get(k) for k in ("embed_provider", "embed_model", "embed_key")):
+        return None
     provider = args_dict.get("embed_provider") or "local"
     config = {"provider": provider}
     if args_dict.get("embed_model"):
@@ -36,6 +42,12 @@ def parse_common_embedding(args_dict: dict) -> dict:
     return config
 
 
+def _with_embeddings(options: dict, embeddings: Optional[dict]) -> dict:
+    if embeddings is not None:
+        options["embeddings"] = embeddings
+    return options
+
+
 def parse_llm(args_dict: dict) -> Optional[dict]:
     provider = args_dict.get("llm_provider")
     if not provider:
@@ -70,7 +82,7 @@ def run_index(args):
     parsed = parser.parse_args(args)
 
     embeddings = parse_common_embedding(vars(parsed))
-    target = resolve_target(parsed.file, {"embeddings": embeddings})
+    target = resolve_target(parsed.file, _with_embeddings({}, embeddings))
 
     build_opts = {}
     if parsed.chunk_size is not None:
@@ -90,6 +102,7 @@ def run_search(args):
     parser.add_argument("file")
     parser.add_argument("query")
     parser.add_argument("--top-k", type=int)
+    parser.add_argument("--mode", choices=_MODES)
     parser.add_argument("--embed-provider")
     parser.add_argument("--embed-model")
     parser.add_argument("--embed-key")
@@ -97,11 +110,13 @@ def run_search(args):
     parsed = parser.parse_args(args)
 
     embeddings = parse_common_embedding(vars(parsed))
-    target = resolve_target(parsed.file, {"embeddings": embeddings})
+    target = resolve_target(parsed.file, _with_embeddings({}, embeddings))
 
     search_opts = {}
     if parsed.top_k is not None:
         search_opts["topK"] = parsed.top_k
+    if parsed.mode is not None:
+        search_opts["mode"] = parsed.mode
 
     results = target.search(parsed.query, search_opts)
     serialized = [r.model_dump(by_alias=True) for r in results]
@@ -120,17 +135,20 @@ def run_ask(args):
     parser.add_argument("--llm-model")
     parser.add_argument("--llm-key")
     parser.add_argument("--stream", action="store_true")
+    parser.add_argument("--mode", choices=_MODES)
 
     parsed = parser.parse_args(args)
 
     embeddings = parse_common_embedding(vars(parsed))
     llm = parse_llm(vars(parsed))
 
-    target = resolve_target(parsed.file, {"embeddings": embeddings, "llm": llm})
+    target = resolve_target(parsed.file, _with_embeddings({"llm": llm}, embeddings))
 
     opts = {}
     if parsed.top_k is not None:
         opts["topK"] = parsed.top_k
+    if parsed.mode is not None:
+        opts["mode"] = parsed.mode
 
     if parsed.stream:
         for chunk in target.ask_stream(parsed.question, opts):
@@ -154,42 +172,40 @@ def run_serve(args):
     parser.add_argument("--host")
     parser.add_argument("--port", type=int)
     parser.add_argument("--token")
+    parser.add_argument("--mode", choices=_MODES)
 
     parsed = parser.parse_args(args)
 
     embeddings = parse_common_embedding(vars(parsed))
     llm = parse_llm(vars(parsed))
 
-    target = resolve_target(
-        parsed.file,
-        {"embeddings": embeddings, **({"llm": llm} if llm else {})},
-    )
+    options: dict = _with_embeddings({}, embeddings)
+    if llm:
+        options["llm"] = llm
+    if parsed.mode:
+        options["retrieval"] = {"mode": parsed.mode}
+    target = resolve_target(parsed.file, options)
     target.build()
 
-    serve_opts = {}
-    if llm:
-        serve_opts["llm"] = llm
-    if parsed.host:
-        serve_opts["host"] = parsed.host
-    if parsed.port is not None:
-        serve_opts["port"] = parsed.port
-    if parsed.token:
-        serve_opts["bearerToken"] = parsed.token
-
-    if hasattr(target, "serve"):
-        target.serve(**serve_opts)
-    else:
-        from .api.server import create_server
-        handle = create_server(target, serve_opts)
-        sys.stdout.write(f"RagLite listening on {handle.url}\n")
-        sys.stdout.flush()
+    host = parsed.host or DEFAULT_HOST
+    port = parsed.port if parsed.port is not None else DEFAULT_PORT
+
+    if isinstance(target, DocumentCollection):
+        # Blocks until the server stops.
+        target.serve(host=host, port=port, bearer_token=parsed.token, llm=llm)
+        return
 
-        try:
-            while True:
-                time.sleep(1)
-        except KeyboardInterrupt:
-            handle.close()
-            sys.exit(0)
+    handle = target.serve(
+        {"llm": llm} if llm else None, host=host, port=port, bearer_token=parsed.token
+    )
+    sys.stdout.write(f"RagLite listening on {handle.url}\n")
+    sys.stdout.flush()
+    try:
+        while True:
+            time.sleep(1)
+    except KeyboardInterrupt:
+        handle.close()
+        sys.exit(0)
 
 
 def main():