arara-rag 0.4.0__tar.gz

arara_rag-0.4.0/LICENSE

@@ -0,0 +1,15 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

arara_rag-0.4.0/PKG-INFO

@@ -0,0 +1,228 @@
+Metadata-Version: 2.4
+Name: arara-rag
+Version: 0.4.0
+Summary: Portuguese-first, CPU-only retrieval: tinyzchunk chunking, static embeddings, BM25, rank fusion and CXM25 reranking. numpy only, out-of-core, with metadata filtering and CRUD.
+Author: Carlo Moro
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/cnmoro/Arara-RAG
+Project-URL: Repository, https://github.com/cnmoro/Arara-RAG
+Keywords: rag,retrieval,portuguese,pt-br,embeddings,bm25,cpu,numpy,out-of-core
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: tokenizers>=0.20
+Requires-Dist: model2vec>=0.9
+Requires-Dist: tinyzchunk>=0.3.3
+Requires-Dist: cxm25>=0.1.0
+Requires-Dist: threadpoolctl>=3.1
+Provides-Extra: bench
+Requires-Dist: datasets>=2.19; extra == "bench"
+Requires-Dist: pyarrow; extra == "bench"
+Requires-Dist: huggingface_hub>=0.23; extra == "bench"
+Requires-Dist: matplotlib>=3.8; extra == "bench"
+Provides-Extra: validate
+Requires-Dist: pytrec_eval-terrier>=0.5.6; extra == "validate"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# arara-rag
+
+**Portuguese-first retrieval that runs entirely on CPU.** Chunking, dense and
+lexical retrieval, rank fusion and reranking — numpy only. No PyTorch, no ONNX
+Runtime, no FAISS. The whole install is **200 MB**.
+
+```bash
+pip install arara-rag
+```
+
+```python
+from arara_rag import Arara
+
+arara = Arara(path="./indice")            # out-of-core and persistent
+arara.add_documents(
+    {"lei_1234": open("lei.txt").read()},
+    metadata={"ano": 2024, "tipo": "lei", "uf": "BR"},
+)
+hits = arara.search("qual a alíquota?", top_k=5, where={"ano": {"$gte": 2020}})
+print(hits[0].doc_id, arara.resolve(hits[0]))     # exact source span
+```
+
+## What's in it
+
+| Stage | Component | Size |
+|---|---|---|
+| Chunking | [`tinyzchunk`](https://github.com/cnmoro/tinyzchunk) — tokenizer-free, distilled from an LLM teacher | 2.1 MB |
+| Dense | [`static-nomic-384-pten-v2`](https://huggingface.co/cnmoro/static-nomic-384-pten-v2) — Model2Vec static embeddings | 62 MB |
+| Lexical | BM25 over a numpy inverted index | — |
+| Rerank | [`CXM25`](https://github.com/cnmoro/CXM25) — PT-BR lexical scoring | bundled |
+| Fusion | Reciprocal Rank Fusion | — |
+
+An index can be **out-of-core**: vectors live in a memory-mapped file and
+documents, metadata and offsets in SQLite, so the index is about **1 KB per
+document** and cold pages can be evicted by the OS instead of being pinned on
+the heap. Same API either way — `Arara()` keeps everything in memory.
+
+## Speed and memory
+
+One CPU core, no GPU. Measured end to end with `python -m bench.profile`.
+
+| documents | index build | query p50 | query p95 | index size | peak RSS to serve |
+|---|---|---|---|---|---|
+| 1,000 | 3.1 s | **0.45 ms** | 0.54 ms | 1.8 MB | 472 MB |
+| 10,000 | 8.8 s | **0.86 ms** | 6.8 ms | 18 MB | 481 MB |
+| 50,000 | 33.8 s | **7.0 ms** | 8.0 ms | 91 MB | 553 MB |
+
+- **~1,300–1,500 documents/second** to chunk, embed, tokenise and index —
+  chunking and embedding are per-document, so they run across processes.
+  A single process manages ~300/second.
+- **~1.8 KB per document** of index: 1.5 KB of vectors plus BM25 postings.
+- Query latency scales with corpus size because both retrievers score the whole
+  corpus per query — that is what makes the ranking exact rather than
+  approximate.
+- Peak RSS is dominated by a **~470 MB fixed cost** (Python, numpy, the
+  embedding model and the tokenizer tables); the corpus adds ~1.7 KB per
+  document on top. Vectors are memory-mapped, so cold pages can be evicted.
+
+![Speed and memory](docs/scaling.png)
+
+## Retrieval quality
+
+MTEB-BR, the Brazilian Portuguese benchmark with a
+[public leaderboard](https://huggingface.co/spaces/MTEB-BR/leaderboard).
+nDCG@10, fixed-window chunking. Metrics are computed by `bench/metrics.py`,
+which `bench/validate_metrics.py` checks against `pytrec_eval` to **0.0e+00**.
+
+| Task | docs | dense | lexical | hybrid |
+|---|---|---|---|---|
+| BRTaxQAR (capped) | 478 | 0.2934 | **0.4051** | 0.3486 |
+| FaQuADIR | 244 | 0.7139 | **0.8961** | 0.8304 |
+| FaqBacenRetrieval | 1,673 | 0.3745 | **0.4881** | 0.4526 |
+| JurisTCU | 16,045 | 0.3887 | **0.5378** | 0.4890 |
+| Quati | 50,000 | 0.3268 | **0.4067** | 0.4046 |
+
+CXM25 reranking on top of the hybrid adds **+0.018 to +0.077 nDCG@10** across
+these tasks for 1–6 ms per query (FaQuADIR: 0.8304 → **0.9078**,
+BRTaxQAR full documents: 0.4801 → **0.5091**).
+
+## Why chunking matters most
+
+Legal documents in BR-TaxQA-R average 32,000 characters and reach 1.17M. MTEB-BR
+truncates them at 32k because transformer encoders cannot fit more. arara
+chunks, so it indexes the whole statute.
+
+| Configuration | chunks | nDCG@10 | R@100 |
+|---|---|---|---|
+| capped at 32k, one vector per doc *(the leaderboard's setting)* | 478 | 0.1496 | 0.4351 |
+| capped at 32k, fixed windows | 2,552 | 0.2934 | 0.6300 |
+| capped at 32k, **tinyzchunk** | 23,319 | 0.3088 | 0.6225 |
+| **full documents**, fixed windows | 6,439 | 0.4041 | 0.7497 |
+| **full documents**, paragraph splits | 6,439 | 0.4041 | 0.7497 |
+| **full documents**, tinyzchunk | 60,927 | 0.4287 | 0.7486 |
+| **full documents**, tinyzchunk + BM25 | 60,927 | 0.4801 | 0.8496 |
+| **full documents**, + CXM25 rerank | 60,927 | **0.5091** | 0.8102 |
+
+![Chunking a legal corpus beats truncating it by 3.4×](docs/ablation.png)
+
+## Against the leaderboard
+
+**On FaQuADIR, arara's best configuration outranks all 96 models on the board**
+— above `voyage-context-4`, `gemini-embedding-2` and `Qwen3-Embedding-8B` — on
+one CPU core. On BR-TaxQA-R it beats 90 of 95.
+
+The honest caveat: the leaderboard evaluates *embedding* models, and there is no
+BM25 entry on it. arara's strongest modes are lexical, and lexical retrieval is
+simply very good on short, high-overlap PT-BR documents — part of that gap is a
+missing baseline on their side, not a transformer-killing dense model on ours.
+
+![arara against the MTEB-BR leaderboard](docs/leaderboard.png)
+
+## Reranking
+
+MTEB-BR reranking hands you a fixed candidate list and scores only the order
+(MAP@1000), so `identity` is the baseline the benchmark ships with.
+
+| Task | identity | dense | lexical | hybrid | **CXM25** |
+|---|---|---|---|---|---|
+| QuatiReranking | 0.2839 | 0.2798 | 0.2939 | 0.3066 | **0.3100** |
+| JurisTCUReranking | 0.4150 | 0.3609 | 0.4279 | 0.4129 | **0.4845** |
+
+## Out-of-core, metadata, CRUD
+
+An index is read far more than it is written, so deletes are tombstones and
+freed slots are recycled on the next write.
+
+```python
+arara = Arara(path="./indice", max_chunk_chars=2000)
+
+arara.add_documents(docs, metadata={"ano": 2024})        # insert / replace
+arara.update_metadata("lei_1234", {"revisado": True})    # no re-embedding
+arara.delete_document("lei_1234")                        # tombstone + slot reuse
+arara.get_document("lei_1234")                           # (text, metadata)
+arara.compact()                                          # reclaim file space
+
+arara.search(q, where={"tipo": {"$in": ["lei", "decreto"]}, "ano": {"$gte": 2020}})
+arara.search(q, where={"$or": [{"uf": "SP"}, {"uf": "RJ"}]})
+```
+
+Supported per field: `$eq` (bare value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`,
+`$in`, `$nin`, `$exists`, `$contains`, `$startswith`, `$endswith`; plus
+top-level `$and` / `$or`. Field names are validated and values are bound as SQL
+parameters, so a filter cannot inject SQL.
+
+## Guarantees
+
+Enforced by 88 tests, not asserted in prose:
+
+- every chunk is an **exact substring** of the canonical document, ordered and
+  non-overlapping, with only whitespace between chunks — nothing is dropped;
+- **no chunk exceeds `max_chunk_chars`**, including on a 24,000-character line;
+- CRLF and LF inputs chunk **identically** and offsets still resolve;
+- the in-memory and on-disk paths return **identical rankings**;
+- importing the package never imports `torch` or `onnxruntime`.
+
+## Reproduce
+
+```bash
+python -m venv .venv && .venv/bin/pip install -e ".[bench,validate,dev]"
+python -m pytest tests/                 # 88 tests
+python bench/validate_metrics.py        # metrics vs pytrec_eval
+./bench/run_all.sh                      # every suite -> bench/results/
+python -m bench.profile                 # speed and memory -> docs/scaling.png
+python -m bench.charts                  # regenerate the figures
+python -m bench.leaderboard             # compare against MTEB-BR
+```
+
+## Layout
+
+```
+arara_rag/
+  chunk.py      chunking and the losslessness contract
+  dense.py      static encoder
+  lexical.py    BM25 inverted index + CXM25 reranker
+  store.py      memory-mapped vectors, SQLite catalog, filters
+  pipeline.py   Arara: add / search / rerank / CRUD
+bench/          task loaders, metrics, suites, profiling, charts
+tests/          88 contract and correctness tests
+space/          Gradio demo
+```
+
+## Limitations
+
+- **PT-BR and English.** The tokenizer, stemmer and stopwords are Portuguese.
+- **The dense model is small** and static; lexical retrieval carries the stack
+  on short, high-overlap documents.
+- **Per-chunk bookkeeping stays resident** (16 bytes/chunk); vectors, text and
+  metadata do not.
+- **CXM25 reranking is ~71 µs/document**, so it runs over a candidate set.
+- **Index build forks worker processes** to parallelise chunking and embedding.
+  Set `workers=1` where forking is unsafe or unavailable; the result is
+  byte-identical, only slower.
+
+## License
+
+Apache-2.0.

arara_rag-0.4.0/README.md

@@ -0,0 +1,196 @@
+# arara-rag
+
+**Portuguese-first retrieval that runs entirely on CPU.** Chunking, dense and
+lexical retrieval, rank fusion and reranking — numpy only. No PyTorch, no ONNX
+Runtime, no FAISS. The whole install is **200 MB**.
+
+```bash
+pip install arara-rag
+```
+
+```python
+from arara_rag import Arara
+
+arara = Arara(path="./indice")            # out-of-core and persistent
+arara.add_documents(
+    {"lei_1234": open("lei.txt").read()},
+    metadata={"ano": 2024, "tipo": "lei", "uf": "BR"},
+)
+hits = arara.search("qual a alíquota?", top_k=5, where={"ano": {"$gte": 2020}})
+print(hits[0].doc_id, arara.resolve(hits[0]))     # exact source span
+```
+
+## What's in it
+
+| Stage | Component | Size |
+|---|---|---|
+| Chunking | [`tinyzchunk`](https://github.com/cnmoro/tinyzchunk) — tokenizer-free, distilled from an LLM teacher | 2.1 MB |
+| Dense | [`static-nomic-384-pten-v2`](https://huggingface.co/cnmoro/static-nomic-384-pten-v2) — Model2Vec static embeddings | 62 MB |
+| Lexical | BM25 over a numpy inverted index | — |
+| Rerank | [`CXM25`](https://github.com/cnmoro/CXM25) — PT-BR lexical scoring | bundled |
+| Fusion | Reciprocal Rank Fusion | — |
+
+An index can be **out-of-core**: vectors live in a memory-mapped file and
+documents, metadata and offsets in SQLite, so the index is about **1 KB per
+document** and cold pages can be evicted by the OS instead of being pinned on
+the heap. Same API either way — `Arara()` keeps everything in memory.
+
+## Speed and memory
+
+One CPU core, no GPU. Measured end to end with `python -m bench.profile`.
+
+| documents | index build | query p50 | query p95 | index size | peak RSS to serve |
+|---|---|---|---|---|---|
+| 1,000 | 3.1 s | **0.45 ms** | 0.54 ms | 1.8 MB | 472 MB |
+| 10,000 | 8.8 s | **0.86 ms** | 6.8 ms | 18 MB | 481 MB |
+| 50,000 | 33.8 s | **7.0 ms** | 8.0 ms | 91 MB | 553 MB |
+
+- **~1,300–1,500 documents/second** to chunk, embed, tokenise and index —
+  chunking and embedding are per-document, so they run across processes.
+  A single process manages ~300/second.
+- **~1.8 KB per document** of index: 1.5 KB of vectors plus BM25 postings.
+- Query latency scales with corpus size because both retrievers score the whole
+  corpus per query — that is what makes the ranking exact rather than
+  approximate.
+- Peak RSS is dominated by a **~470 MB fixed cost** (Python, numpy, the
+  embedding model and the tokenizer tables); the corpus adds ~1.7 KB per
+  document on top. Vectors are memory-mapped, so cold pages can be evicted.
+
+![Speed and memory](docs/scaling.png)
+
+## Retrieval quality
+
+MTEB-BR, the Brazilian Portuguese benchmark with a
+[public leaderboard](https://huggingface.co/spaces/MTEB-BR/leaderboard).
+nDCG@10, fixed-window chunking. Metrics are computed by `bench/metrics.py`,
+which `bench/validate_metrics.py` checks against `pytrec_eval` to **0.0e+00**.
+
+| Task | docs | dense | lexical | hybrid |
+|---|---|---|---|---|
+| BRTaxQAR (capped) | 478 | 0.2934 | **0.4051** | 0.3486 |
+| FaQuADIR | 244 | 0.7139 | **0.8961** | 0.8304 |
+| FaqBacenRetrieval | 1,673 | 0.3745 | **0.4881** | 0.4526 |
+| JurisTCU | 16,045 | 0.3887 | **0.5378** | 0.4890 |
+| Quati | 50,000 | 0.3268 | **0.4067** | 0.4046 |
+
+CXM25 reranking on top of the hybrid adds **+0.018 to +0.077 nDCG@10** across
+these tasks for 1–6 ms per query (FaQuADIR: 0.8304 → **0.9078**,
+BRTaxQAR full documents: 0.4801 → **0.5091**).
+
+## Why chunking matters most
+
+Legal documents in BR-TaxQA-R average 32,000 characters and reach 1.17M. MTEB-BR
+truncates them at 32k because transformer encoders cannot fit more. arara
+chunks, so it indexes the whole statute.
+
+| Configuration | chunks | nDCG@10 | R@100 |
+|---|---|---|---|
+| capped at 32k, one vector per doc *(the leaderboard's setting)* | 478 | 0.1496 | 0.4351 |
+| capped at 32k, fixed windows | 2,552 | 0.2934 | 0.6300 |
+| capped at 32k, **tinyzchunk** | 23,319 | 0.3088 | 0.6225 |
+| **full documents**, fixed windows | 6,439 | 0.4041 | 0.7497 |
+| **full documents**, paragraph splits | 6,439 | 0.4041 | 0.7497 |
+| **full documents**, tinyzchunk | 60,927 | 0.4287 | 0.7486 |
+| **full documents**, tinyzchunk + BM25 | 60,927 | 0.4801 | 0.8496 |
+| **full documents**, + CXM25 rerank | 60,927 | **0.5091** | 0.8102 |
+
+![Chunking a legal corpus beats truncating it by 3.4×](docs/ablation.png)
+
+## Against the leaderboard
+
+**On FaQuADIR, arara's best configuration outranks all 96 models on the board**
+— above `voyage-context-4`, `gemini-embedding-2` and `Qwen3-Embedding-8B` — on
+one CPU core. On BR-TaxQA-R it beats 90 of 95.
+
+The honest caveat: the leaderboard evaluates *embedding* models, and there is no
+BM25 entry on it. arara's strongest modes are lexical, and lexical retrieval is
+simply very good on short, high-overlap PT-BR documents — part of that gap is a
+missing baseline on their side, not a transformer-killing dense model on ours.
+
+![arara against the MTEB-BR leaderboard](docs/leaderboard.png)
+
+## Reranking
+
+MTEB-BR reranking hands you a fixed candidate list and scores only the order
+(MAP@1000), so `identity` is the baseline the benchmark ships with.
+
+| Task | identity | dense | lexical | hybrid | **CXM25** |
+|---|---|---|---|---|---|
+| QuatiReranking | 0.2839 | 0.2798 | 0.2939 | 0.3066 | **0.3100** |
+| JurisTCUReranking | 0.4150 | 0.3609 | 0.4279 | 0.4129 | **0.4845** |
+
+## Out-of-core, metadata, CRUD
+
+An index is read far more than it is written, so deletes are tombstones and
+freed slots are recycled on the next write.
+
+```python
+arara = Arara(path="./indice", max_chunk_chars=2000)
+
+arara.add_documents(docs, metadata={"ano": 2024})        # insert / replace
+arara.update_metadata("lei_1234", {"revisado": True})    # no re-embedding
+arara.delete_document("lei_1234")                        # tombstone + slot reuse
+arara.get_document("lei_1234")                           # (text, metadata)
+arara.compact()                                          # reclaim file space
+
+arara.search(q, where={"tipo": {"$in": ["lei", "decreto"]}, "ano": {"$gte": 2020}})
+arara.search(q, where={"$or": [{"uf": "SP"}, {"uf": "RJ"}]})
+```
+
+Supported per field: `$eq` (bare value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`,
+`$in`, `$nin`, `$exists`, `$contains`, `$startswith`, `$endswith`; plus
+top-level `$and` / `$or`. Field names are validated and values are bound as SQL
+parameters, so a filter cannot inject SQL.
+
+## Guarantees
+
+Enforced by 88 tests, not asserted in prose:
+
+- every chunk is an **exact substring** of the canonical document, ordered and
+  non-overlapping, with only whitespace between chunks — nothing is dropped;
+- **no chunk exceeds `max_chunk_chars`**, including on a 24,000-character line;
+- CRLF and LF inputs chunk **identically** and offsets still resolve;
+- the in-memory and on-disk paths return **identical rankings**;
+- importing the package never imports `torch` or `onnxruntime`.
+
+## Reproduce
+
+```bash
+python -m venv .venv && .venv/bin/pip install -e ".[bench,validate,dev]"
+python -m pytest tests/                 # 88 tests
+python bench/validate_metrics.py        # metrics vs pytrec_eval
+./bench/run_all.sh                      # every suite -> bench/results/
+python -m bench.profile                 # speed and memory -> docs/scaling.png
+python -m bench.charts                  # regenerate the figures
+python -m bench.leaderboard             # compare against MTEB-BR
+```
+
+## Layout
+
+```
+arara_rag/
+  chunk.py      chunking and the losslessness contract
+  dense.py      static encoder
+  lexical.py    BM25 inverted index + CXM25 reranker
+  store.py      memory-mapped vectors, SQLite catalog, filters
+  pipeline.py   Arara: add / search / rerank / CRUD
+bench/          task loaders, metrics, suites, profiling, charts
+tests/          88 contract and correctness tests
+space/          Gradio demo
+```
+
+## Limitations
+
+- **PT-BR and English.** The tokenizer, stemmer and stopwords are Portuguese.
+- **The dense model is small** and static; lexical retrieval carries the stack
+  on short, high-overlap documents.
+- **Per-chunk bookkeeping stays resident** (16 bytes/chunk); vectors, text and
+  metadata do not.
+- **CXM25 reranking is ~71 µs/document**, so it runs over a candidate set.
+- **Index build forks worker processes** to parallelise chunking and embedding.
+  Set `workers=1` where forking is unsafe or unavailable; the result is
+  byte-identical, only slower.
+
+## License
+
+Apache-2.0.

arara_rag-0.4.0/arara_rag/__init__.py

@@ -0,0 +1,36 @@
+"""arara-rag -- a Portuguese-first, CPU-only RAG retrieval stack.
+
+Everything runs on CPU with numpy. No PyTorch, no ONNX Runtime, no FAISS on the
+query path.
+
+    >>> from arara_rag import Arara
+    >>> a = Arara(path="./index")            # out-of-core, persistent
+    >>> a.add_documents({"doc": "texto ..."}, metadata={"ano": 2024})
+    >>> hits = a.search("pergunta", top_k=5, where={"ano": {"$gte": 2020}})
+"""
+
+from .chunk import Chunker
+from .dense import DEFAULT_DENSE_MODEL, DenseEncoder, DenseIndex
+from .fuse import rank_from_scores, reciprocal_rank_fusion
+from .lexical import BM25Index, CXM25Scorer
+from .pipeline import Arara
+from .text import Tokenizer
+from .types import Chunk, Hit
+
+__version__ = "0.4.0"
+
+__all__ = [
+    "Arara",
+    "BM25Index",
+    "CXM25Scorer",
+    "Chunk",
+    "Chunker",
+    "DEFAULT_DENSE_MODEL",
+    "DenseEncoder",
+    "DenseIndex",
+    "Hit",
+    "Tokenizer",
+    "rank_from_scores",
+    "reciprocal_rank_fusion",
+    "__version__",
+]

arara_rag-0.4.0/arara_rag/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

arara_rag-0.4.0/arara_rag/chunk.py

@@ -0,0 +1,177 @@
+"""Chunking strategies.
+
+``tinyzchunk`` is the default: a GPU-free, tokenizer-free chunker distilled
+from an LLM teacher. It is what lets arara index the 32k-1.1M character legal
+statutes that fixed-window embedders are forced to truncate.
+
+The contract below is enforced here rather than assumed, because the upstream
+chunker has three behaviours a pipeline cannot tolerate:
+
+1. it strips the whitespace between structural units, so consecutive chunks are
+   not byte-adjacent;
+2. it can return a chunk longer than ``max_chunk_chars`` on degenerate input
+   (a single very long line);
+3. it leaves CRLF line endings in the returned text, so "chunks identically" is
+   only true of the boundaries, not of the strings.
+
+arara therefore normalises line endings, splits oversized chunks at word
+boundaries, and defines the guarantee it can actually uphold:
+
+* every chunk is an exact substring of the **canonical** document
+  (line endings normalised to ``\\n``);
+* chunks are ordered and non-overlapping;
+* the gap between two consecutive chunks contains **only whitespace**, so no
+  non-whitespace character is ever dropped or duplicated;
+* no chunk exceeds ``max_chunk_chars``.
+
+``Arara.document_text(doc_id)`` returns the canonical text the offsets index
+into.
+"""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from .types import Chunk
+
+ChunkMode = Literal["tinyzchunk", "paragraph", "window", "document"]
+
+# Legacy alias: "none" never meant "one chunk per document" -- it means "no
+# structural chunker", i.e. fixed windows. Keeping the alias avoids silently
+# changing behaviour for callers while the honest name is "window".
+_ALIASES = {"none": "window"}
+
+
+def canonicalize(text: str) -> str:
+    """Normalise line endings so offsets are stable across platforms."""
+    return text.replace("\r\n", "\n").replace("\r", "\n")
+
+
+class Chunker:
+    """Splits documents into provably lossless spans."""
+
+    def __init__(
+        self,
+        mode: ChunkMode | str = "tinyzchunk",
+        max_chunk_chars: int = 2500,
+        min_chunk_chars: int = 100,
+        enforce_max_chars: bool = True,
+    ) -> None:
+        mode = _ALIASES.get(mode, mode)
+        if mode not in ("tinyzchunk", "paragraph", "window", "document"):
+            raise ValueError(f"unknown chunk mode: {mode!r}")
+        self.mode = mode
+        self.max_chunk_chars = max_chunk_chars
+        self.min_chunk_chars = min_chunk_chars
+        self.enforce_max_chars = enforce_max_chars
+        if mode == "tinyzchunk":
+            from tinyzchunk import Chunker as _TinyZChunker
+
+            self._impl = _TinyZChunker(
+                max_chunk_chars=max_chunk_chars,
+                min_chunk_chars=min_chunk_chars,
+            )
+        else:
+            self._impl = None
+
+    # -- public API ---------------------------------------------------------
+    def split(self, doc_id: str, text: str) -> list[Chunk]:
+        """Chunk ``text``; offsets index into ``canonicalize(text)``."""
+        canonical = canonicalize(text)
+        if not canonical:
+            return []
+        chunks: list[Chunk] = []
+        cursor = 0
+        for piece in self._pieces(canonical):
+            if not piece:
+                continue
+            start = canonical.find(piece, cursor)
+            if start < 0:
+                # Defensive: never emit a chunk whose offsets we cannot prove.
+                # Fall back to windowing the remainder rather than lying.
+                for win in self._window(canonical[cursor:]):
+                    s = canonical.find(win, cursor)
+                    if s < 0:
+                        continue
+                    chunks.append(
+                        Chunk(
+                            chunk_id=f"{doc_id}#{len(chunks)}",
+                            doc_id=doc_id,
+                            text=win,
+                            start=s,
+                            end=s + len(win),
+                        )
+                    )
+                    cursor = s + len(win)
+                continue
+            end = start + len(piece)
+            chunks.append(
+                Chunk(
+                    chunk_id=f"{doc_id}#{len(chunks)}",
+                    doc_id=doc_id,
+                    text=piece,
+                    start=start,
+                    end=end,
+                )
+            )
+            cursor = end
+        return chunks
+
+    # -- strategies ---------------------------------------------------------
+    def _pieces(self, text: str) -> list[str]:
+        if not text:
+            return []
+        if self.mode == "document":
+            # One vector per document, ceiling deliberately ignored. This is the
+            # operating point every fixed-window embedder is stuck at, and it is
+            # only sane for documents that fit the model's context.
+            stripped = text.strip()
+            return [stripped] if stripped else []
+        if self.mode == "tinyzchunk":
+            raw = self._impl.chunk(text)
+            raw = raw if raw else [text]
+        elif self.mode == "paragraph":
+            raw = self._paragraphs(text)
+        else:  # window
+            raw = [text]
+        if not self.enforce_max_chars:
+            return [p for piece in raw for p in self._window(piece) if p]
+        pieces: list[str] = []
+        for piece in raw:
+            # Word-aware splitting is applied to every mode, including the
+            # window baseline, so that a cut never lands mid-word.
+            pieces.extend(self._split_oversized(piece))
+        return pieces
+
+    def _window(self, text: str) -> list[str]:
+        """Hard character windows; the honest baseline every chunker beats."""
+        if len(text) <= self.max_chunk_chars:
+            return [text]
+        return [
+            text[i : i + self.max_chunk_chars]
+            for i in range(0, len(text), self.max_chunk_chars)
+        ]
+
+    def _paragraphs(self, text: str) -> list[str]:
+        raw = [p for p in text.split("\n\n") if p.strip()]
+        return raw if raw else [text]
+
+    def _split_oversized(self, piece: str) -> list[str]:
+        """Break a piece that exceeds the ceiling at the last word boundary."""
+        if len(piece) <= self.max_chunk_chars:
+            return [piece.strip()] if piece.strip() else []
+        out: list[str] = []
+        rest = piece
+        while len(rest) > self.max_chunk_chars:
+            window = rest[: self.max_chunk_chars]
+            cut = window.rfind(" ")
+            if cut < self.max_chunk_chars // 2:
+                cut = self.max_chunk_chars  # unbreakable run: hard cut
+            head = rest[:cut].strip()
+            if head:
+                out.append(head)
+            rest = rest[cut:]
+        tail = rest.strip()
+        if tail:
+            out.append(tail)
+        return out