worddael 0.1.0rc3__tar.gz

worddael-0.1.0rc3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 worddael contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

worddael-0.1.0rc3/PKG-INFO

@@ -0,0 +1,201 @@
+Metadata-Version: 2.4
+Name: worddael
+Version: 0.1.0rc3
+Summary: The Dale of Remembered Words (Worddael) | Chinese-aware RAG text chunking
+Author: worddael contributors
+License: MIT
+Project-URL: Homepage, https://github.com/cloudydreamland/TheDaleOfRememberedWords
+Project-URL: Repository, https://github.com/cloudydreamland/TheDaleOfRememberedWords
+Project-URL: Issues, https://github.com/cloudydreamland/TheDaleOfRememberedWords/issues
+Project-URL: Changelog, https://github.com/cloudydreamland/TheDaleOfRememberedWords/blob/main/CHANGELOG.md
+Project-URL: Security, https://github.com/cloudydreamland/TheDaleOfRememberedWords/security/policy
+Keywords: chinese,chunking,rag,nlp,text-splitting,retrieval
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: Chinese (Simplified)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: jieba
+Requires-Dist: jieba>=0.42; extra == "jieba"
+Provides-Extra: tiktoken
+Requires-Dist: tiktoken>=0.7; extra == "tiktoken"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Dynamic: license-file
+
+# The Dale of Remembered Words — Worddael
+
+简体中文 · [English](README.en.md)
+
+> 展示名 **The Dale of Remembered Words** 描绘一处安放文字的山谷；Worddael 是该项目的短名。
+
+**中文优先的 RAG 文本切分库。把中文文本切成好用的块，并保留精确到字符的原文偏移量。**
+
+[![CI](https://github.com/cloudydreamland/TheDaleOfRememberedWords/actions/workflows/ci.yml/badge.svg)](.github/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+## 为什么需要它 / Why
+
+通用文本切分器可用于多种语言，但中文句界、标点归属和回引源文等细节往往需要应用自行处理。`worddael` 提供中文规则、可组合策略和原文偏移，让这些行为能在管线中检查：
+
+- **中文句界原生正确**：`。！？；…` 加引号闭合格式感知；ASCII `.` 不切（保护 `3.14`、版本号、URL）
+- **字符级偏移不变量**：每个 chunk 保证 `chunk.text == source[chunk.start:chunk.end]`，检索结果可直接引用回原文（fuzz 测试覆盖）
+- **零必装依赖**：核心纯 Python；jieba / tiktoken / embedding 全部可选
+- **自带评测**：内置 BM25 检索召回评测 + 可插拔 LLM 判分器（支持干跑，零成本先跑通）
+- **CPU 即可用**：不需要 GPU，不需要下载模型
+
+## Quickstart (English)
+
+worddael (Worddael, "a cut-out chapter") chunks **Chinese** text for RAG, keeping exact
+character offsets so every retrieved chunk can be cited back to its source.
+
+```python
+from worddael import chunk
+
+chunks = chunk(long_text, strategy="recursive", max_chars=500, overlap_chars=50)
+assert all(c.text == long_text[c.start:c.end] for c in chunks)  # invariant
+```
+
+- Chinese-aware sentence boundaries (。！？；… with quote-closer attachment;
+  ASCII `.` never splits decimals/versions)
+- Markdown heading paths, atomic code fences, parent-child (small-to-big)
+  chunking, pluggable semantic chunking and token counters
+- Zero required dependencies, CPU-only; optional extras: `worddael[jieba]`,
+  `worddael[tiktoken]`
+- Built-in evaluation: BM25 recall/precision@k over an original 25-doc
+  benchmark, LLM answerability judging (dry-run needs no API key)
+
+See [docs/status_quo.md](docs/status_quo.md) for reproducible failure cases
+of default splitters on Chinese, and [benchmarks/results.md](benchmarks/results.md)
+for current numbers.
+
+## 安装 / Install
+
+> 当前尚未发布到 PyPI；下方给出从 GitHub 获取并本地安装的命令。
+
+```bash
+git clone https://github.com/cloudydreamland/TheDaleOfRememberedWords.git
+cd TheDaleOfRememberedWords
+python -m pip install .
+# PyPI 首发后：python -m pip install worddael
+python -m pip install ".[jieba]"
+```
+
+## 快速开始 / Quickstart
+
+```python
+from worddael import chunk
+
+text = open("manual.md", encoding="utf-8").read()
+chunks = chunk(text, strategy="recursive", max_chars=500, overlap_chars=50)
+
+for c in chunks:
+    print(c.seq, c.start, c.end, c.meta.get("headings"), c.text[:20])
+    assert c.text == text[c.start:c.end]   # 永远成立，可放心引用
+```
+
+```python
+from worddael import SemanticChunker, OpenAICompatibleEmbedder, HashingEmbedder
+
+# 语义切分：接任意 OpenAI 兼容 embedding 接口（key 从环境变量读取，批量+超时可配）
+
+embedder = OpenAICompatibleEmbedder(
+    base_url="https://open.bigmodel.cn/api/paas/v4",
+    model="embedding-3",
+    api_key_env="ZHIPUAI_API_KEY",
+)
+chunks = SemanticChunker(embed_fn=embedder, similarity_threshold=0.55,
+                         max_chars=500, min_chars=80).chunk(long_text)
+
+# 或用零依赖的 HashingEmbedder 先把管线跑通（玩具级，非质量结论）
+
+chunks = SemanticChunker(embed_fn=HashingEmbedder()).chunk(long_text)
+```
+
+命令行：
+
+```bash
+worddael doc.md --stats
+worddael manual.txt --strategy recursive --max-chars 400 --overlap-chars 50 --jsonl chunks.jsonl
+```
+
+评测（CPU、零 API 成本）：
+
+```bash
+python -m worddael.eval.report --k 3            # 内置语料上对比各策略的 recall@k
+```
+
+## 切分策略 / Strategies
+
+| strategy | 适用场景 | 说明 |
+|---|---|---|
+| `recursive` | 通用中文文本 | 段落→行→句号级→逗号级递归切分，中文标点优先 |
+| `markdown` | .md 文档 | 按标题结构切，heading path 写入 meta，代码块保持完整 |
+| `sentence` | 需要整句边界 | 按中文句界整句打包 |
+| `token` | 对齐模型 token 预算 | 可插拔计数器（启发式 / jieba / tiktoken） |
+| `semantic` | 主题边界敏感 | 自带 embedding 函数，相邻句相似度下降处切分，带防碎片最小块长 |
+| `parent-child` | 小-大检索（RAG 常见手搓模式） | 子块精确匹配、父块给上下文；`chunk_families()` 返回 (父, 子)，meta 双向链接，偏移不变量照旧 |
+
+## 与现有方案的关系 / Landscape
+
+| 方案 | 问题 |
+|---|---|
+| LangChain text splitters | 通用切分组件；Worddael 可作为中文切分选项，并提供自己的偏移与策略 API |
+| Chonkie 等 ingestion 库 | 面向更广的文档摄取与切分场景；选择时应按格式覆盖、语言和集成需求比较 |
+| RAGFlow 等 RAG 引擎 | 提供完整应用或管线；Worddael 是可嵌入现有 Python 项目的独立库 |
+| jieba | 中文分词工具；与按句界、结构和长度做文档切分的目标不同 |
+
+LangChain 默认参数切中文的**真实坏例子**（答案腰斩、标题孤立，可复现脚本）见 [docs/status_quo.md](docs/status_quo.md)；详细论证见 [GAP_PROOF.md](GAP_PROOF.md)。
+
+## 评测 / Evaluation
+
+- 内置基准（25 篇 / 75 问）真实结果：[benchmarks/results.md](benchmarks/results.md)
+  - 当前快照：recall@1 recursive 0.96 vs fixed-window 0.907；小-大检索（子块 150 检索 / 父块 600 判分）0.987 vs 平铺 0.96
+- 大规模 LLM 判分评测指南（拿到 key 后）：[docs/eval_guide.md](docs/eval_guide.md)
+- 选题论证（为什么这个缺口是真的）：[GAP_PROOF.md](GAP_PROOF.md)
+
+## 路线图 / Roadmap
+
+见 [ROADMAP.md](ROADMAP.md)。当前 v0.1.0rc3：六个策略（含父子块）+ 可解释切分 + 标点卫生保证 + 评测框架（测试数以 CI 为准）。真实语料实测见 [docs/dogfood.md](docs/dogfood.md)：90.6 万字《紅樓夢》，24.6 MB/s，卫生违例 0。
+
+## Non-goals（明确不做）
+
+- **模型/LLM 驱动切分**：红海方向且依赖 GPU/API，与零依赖纯 CPU 定位冲突；语义切分走可插拔 embedder
+- **通用 RAG 框架**：只做切分与切分评测，通过 [adapters](#安装--install) 进入 LangChain 等既有管线
+- **计费级 token 计数**：启发式计数只服务预算切分（误差数字见基准报告校准小节）
+- **流式超大文件**：单文档需可入内存；GB 级请自行分片
+
+架构与扩展指南见 [docs/architecture.md](docs/architecture.md)。
+
+## FAQ
+
+- **overlap 模式下拼接结果比原文长？** 设计如此：overlap 是上下文重复（上一块尾部回看）。`overlap_chars=0` 时拼接与原文逐字相等（有测试锁定）。
+- **为什么有的块以换行结尾？** 分隔符归属前一块是偏移精确的前提；判断块质量请对 `rstrip()` 后的文本判断（内置统计均如此）。
+- **预算为何偶尔超出 1-2 字？** 标点卫生会把下一块开头的悬垂标点吸收进前一块（上限 2 字），换来"块首无悬垂标点"的保证。
+- **测试数是固定的吗？** 以 CI 最新运行为准；文档不写死数字。
+
+## 开发 / Development
+
+```bash
+pip install -e ".[jieba,dev]"
+pytest
+ruff check src tests
+```
+
+## 反馈与参与
+
+使用问题和功能建议可以在 [Discussions](https://github.com/cloudydreamland/TheDaleOfRememberedWords/discussions) 交流；可复现缺陷请提交 [Issue](https://github.com/cloudydreamland/TheDaleOfRememberedWords/issues)。请只附合成或脱敏后的最小样例，不要上传真实个人信息、API key 或业务原文。安全问题请按 [SECURITY.md](SECURITY.md) 私下报告。
+
+## License
+
+MIT

worddael-0.1.0rc3/README.md

@@ -0,0 +1,167 @@
+# The Dale of Remembered Words — Worddael
+
+简体中文 · [English](README.en.md)
+
+> 展示名 **The Dale of Remembered Words** 描绘一处安放文字的山谷；Worddael 是该项目的短名。
+
+**中文优先的 RAG 文本切分库。把中文文本切成好用的块，并保留精确到字符的原文偏移量。**
+
+[![CI](https://github.com/cloudydreamland/TheDaleOfRememberedWords/actions/workflows/ci.yml/badge.svg)](.github/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+## 为什么需要它 / Why
+
+通用文本切分器可用于多种语言，但中文句界、标点归属和回引源文等细节往往需要应用自行处理。`worddael` 提供中文规则、可组合策略和原文偏移，让这些行为能在管线中检查：
+
+- **中文句界原生正确**：`。！？；…` 加引号闭合格式感知；ASCII `.` 不切（保护 `3.14`、版本号、URL）
+- **字符级偏移不变量**：每个 chunk 保证 `chunk.text == source[chunk.start:chunk.end]`，检索结果可直接引用回原文（fuzz 测试覆盖）
+- **零必装依赖**：核心纯 Python；jieba / tiktoken / embedding 全部可选
+- **自带评测**：内置 BM25 检索召回评测 + 可插拔 LLM 判分器（支持干跑，零成本先跑通）
+- **CPU 即可用**：不需要 GPU，不需要下载模型
+
+## Quickstart (English)
+
+worddael (Worddael, "a cut-out chapter") chunks **Chinese** text for RAG, keeping exact
+character offsets so every retrieved chunk can be cited back to its source.
+
+```python
+from worddael import chunk
+
+chunks = chunk(long_text, strategy="recursive", max_chars=500, overlap_chars=50)
+assert all(c.text == long_text[c.start:c.end] for c in chunks)  # invariant
+```
+
+- Chinese-aware sentence boundaries (。！？；… with quote-closer attachment;
+  ASCII `.` never splits decimals/versions)
+- Markdown heading paths, atomic code fences, parent-child (small-to-big)
+  chunking, pluggable semantic chunking and token counters
+- Zero required dependencies, CPU-only; optional extras: `worddael[jieba]`,
+  `worddael[tiktoken]`
+- Built-in evaluation: BM25 recall/precision@k over an original 25-doc
+  benchmark, LLM answerability judging (dry-run needs no API key)
+
+See [docs/status_quo.md](docs/status_quo.md) for reproducible failure cases
+of default splitters on Chinese, and [benchmarks/results.md](benchmarks/results.md)
+for current numbers.
+
+## 安装 / Install
+
+> 当前尚未发布到 PyPI；下方给出从 GitHub 获取并本地安装的命令。
+
+```bash
+git clone https://github.com/cloudydreamland/TheDaleOfRememberedWords.git
+cd TheDaleOfRememberedWords
+python -m pip install .
+# PyPI 首发后：python -m pip install worddael
+python -m pip install ".[jieba]"
+```
+
+## 快速开始 / Quickstart
+
+```python
+from worddael import chunk
+
+text = open("manual.md", encoding="utf-8").read()
+chunks = chunk(text, strategy="recursive", max_chars=500, overlap_chars=50)
+
+for c in chunks:
+    print(c.seq, c.start, c.end, c.meta.get("headings"), c.text[:20])
+    assert c.text == text[c.start:c.end]   # 永远成立，可放心引用
+```
+
+```python
+from worddael import SemanticChunker, OpenAICompatibleEmbedder, HashingEmbedder
+
+# 语义切分：接任意 OpenAI 兼容 embedding 接口（key 从环境变量读取，批量+超时可配）
+
+embedder = OpenAICompatibleEmbedder(
+    base_url="https://open.bigmodel.cn/api/paas/v4",
+    model="embedding-3",
+    api_key_env="ZHIPUAI_API_KEY",
+)
+chunks = SemanticChunker(embed_fn=embedder, similarity_threshold=0.55,
+                         max_chars=500, min_chars=80).chunk(long_text)
+
+# 或用零依赖的 HashingEmbedder 先把管线跑通（玩具级，非质量结论）
+
+chunks = SemanticChunker(embed_fn=HashingEmbedder()).chunk(long_text)
+```
+
+命令行：
+
+```bash
+worddael doc.md --stats
+worddael manual.txt --strategy recursive --max-chars 400 --overlap-chars 50 --jsonl chunks.jsonl
+```
+
+评测（CPU、零 API 成本）：
+
+```bash
+python -m worddael.eval.report --k 3            # 内置语料上对比各策略的 recall@k
+```
+
+## 切分策略 / Strategies
+
+| strategy | 适用场景 | 说明 |
+|---|---|---|
+| `recursive` | 通用中文文本 | 段落→行→句号级→逗号级递归切分，中文标点优先 |
+| `markdown` | .md 文档 | 按标题结构切，heading path 写入 meta，代码块保持完整 |
+| `sentence` | 需要整句边界 | 按中文句界整句打包 |
+| `token` | 对齐模型 token 预算 | 可插拔计数器（启发式 / jieba / tiktoken） |
+| `semantic` | 主题边界敏感 | 自带 embedding 函数，相邻句相似度下降处切分，带防碎片最小块长 |
+| `parent-child` | 小-大检索（RAG 常见手搓模式） | 子块精确匹配、父块给上下文；`chunk_families()` 返回 (父, 子)，meta 双向链接，偏移不变量照旧 |
+
+## 与现有方案的关系 / Landscape
+
+| 方案 | 问题 |
+|---|---|
+| LangChain text splitters | 通用切分组件；Worddael 可作为中文切分选项，并提供自己的偏移与策略 API |
+| Chonkie 等 ingestion 库 | 面向更广的文档摄取与切分场景；选择时应按格式覆盖、语言和集成需求比较 |
+| RAGFlow 等 RAG 引擎 | 提供完整应用或管线；Worddael 是可嵌入现有 Python 项目的独立库 |
+| jieba | 中文分词工具；与按句界、结构和长度做文档切分的目标不同 |
+
+LangChain 默认参数切中文的**真实坏例子**（答案腰斩、标题孤立，可复现脚本）见 [docs/status_quo.md](docs/status_quo.md)；详细论证见 [GAP_PROOF.md](GAP_PROOF.md)。
+
+## 评测 / Evaluation
+
+- 内置基准（25 篇 / 75 问）真实结果：[benchmarks/results.md](benchmarks/results.md)
+  - 当前快照：recall@1 recursive 0.96 vs fixed-window 0.907；小-大检索（子块 150 检索 / 父块 600 判分）0.987 vs 平铺 0.96
+- 大规模 LLM 判分评测指南（拿到 key 后）：[docs/eval_guide.md](docs/eval_guide.md)
+- 选题论证（为什么这个缺口是真的）：[GAP_PROOF.md](GAP_PROOF.md)
+
+## 路线图 / Roadmap
+
+见 [ROADMAP.md](ROADMAP.md)。当前 v0.1.0rc3：六个策略（含父子块）+ 可解释切分 + 标点卫生保证 + 评测框架（测试数以 CI 为准）。真实语料实测见 [docs/dogfood.md](docs/dogfood.md)：90.6 万字《紅樓夢》，24.6 MB/s，卫生违例 0。
+
+## Non-goals（明确不做）
+
+- **模型/LLM 驱动切分**：红海方向且依赖 GPU/API，与零依赖纯 CPU 定位冲突；语义切分走可插拔 embedder
+- **通用 RAG 框架**：只做切分与切分评测，通过 [adapters](#安装--install) 进入 LangChain 等既有管线
+- **计费级 token 计数**：启发式计数只服务预算切分（误差数字见基准报告校准小节）
+- **流式超大文件**：单文档需可入内存；GB 级请自行分片
+
+架构与扩展指南见 [docs/architecture.md](docs/architecture.md)。
+
+## FAQ
+
+- **overlap 模式下拼接结果比原文长？** 设计如此：overlap 是上下文重复（上一块尾部回看）。`overlap_chars=0` 时拼接与原文逐字相等（有测试锁定）。
+- **为什么有的块以换行结尾？** 分隔符归属前一块是偏移精确的前提；判断块质量请对 `rstrip()` 后的文本判断（内置统计均如此）。
+- **预算为何偶尔超出 1-2 字？** 标点卫生会把下一块开头的悬垂标点吸收进前一块（上限 2 字），换来"块首无悬垂标点"的保证。
+- **测试数是固定的吗？** 以 CI 最新运行为准；文档不写死数字。
+
+## 开发 / Development
+
+```bash
+pip install -e ".[jieba,dev]"
+pytest
+ruff check src tests
+```
+
+## 反馈与参与
+
+使用问题和功能建议可以在 [Discussions](https://github.com/cloudydreamland/TheDaleOfRememberedWords/discussions) 交流；可复现缺陷请提交 [Issue](https://github.com/cloudydreamland/TheDaleOfRememberedWords/issues)。请只附合成或脱敏后的最小样例，不要上传真实个人信息、API key 或业务原文。安全问题请按 [SECURITY.md](SECURITY.md) 私下报告。
+
+## License
+
+MIT

worddael-0.1.0rc3/pyproject.toml

@@ -0,0 +1,58 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "worddael"
+version = "0.1.0rc3"
+description = "The Dale of Remembered Words (Worddael) | Chinese-aware RAG text chunking"
+readme = "README.md"
+license = { text = "MIT" }
+authors = [{ name = "worddael contributors" }]
+requires-python = ">=3.10"
+keywords = ["chinese", "chunking", "rag", "nlp", "text-splitting", "retrieval"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Natural Language :: Chinese (Simplified)",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Text Processing :: Linguistic",
+]
+dependencies = []
+
+[project.optional-dependencies]
+jieba = ["jieba>=0.42"]
+tiktoken = ["tiktoken>=0.7"]
+dev = ["pytest>=8.0", "ruff>=0.6"]
+
+[project.scripts]
+worddael = "worddael.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/cloudydreamland/TheDaleOfRememberedWords"
+Repository = "https://github.com/cloudydreamland/TheDaleOfRememberedWords"
+Issues = "https://github.com/cloudydreamland/TheDaleOfRememberedWords/issues"
+Changelog = "https://github.com/cloudydreamland/TheDaleOfRememberedWords/blob/main/CHANGELOG.md"
+Security = "https://github.com/cloudydreamland/TheDaleOfRememberedWords/security/policy"
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+worddael = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "W"]
+ignore = ["E501"]

worddael-0.1.0rc3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

worddael-0.1.0rc3/src/worddael/__init__.py

@@ -0,0 +1,92 @@
+"""Worddael — Chinese-first text chunking for RAG.
+
+把中文文本切成好用的块，同时保留精确到字符的原文偏移量，
+让 RAG 的每一条检索结果都能被可靠引用。
+
+Quickstart::
+
+    from worddael import chunk
+
+    chunks = chunk(long_text, strategy="recursive", max_chars=500, overlap_chars=50)
+    for c in chunks:
+        print(c.seq, c.start, c.end, c.text[:20])
+"""
+
+from __future__ import annotations
+
+from .chunkers import (
+    STRATEGIES,
+    BaseChunker,
+    MarkdownChunker,
+    RecursiveChunker,
+    SemanticChunker,
+    SentenceChunker,
+    TokenChunker,
+)
+from .counters import CharCounter, get_counter
+from .embedders import (
+    Embedder,
+    EmbedderError,
+    HashingEmbedder,
+    OpenAICompatibleEmbedder,
+)
+from .io_utils import chunk_file, to_jsonl
+from .parent_child import ParentChildChunker
+from .types import Chunk
+
+__version__ = "0.1.0rc3"
+
+# parent-child families are produced by a dedicated module; register the
+# strategy here so get_chunker("parent-child") works like any other.
+STRATEGIES["parent-child"] = ParentChildChunker
+
+__all__ = [
+    "BaseChunker",
+    "CharCounter",
+    "Chunk",
+    "Embedder",
+    "EmbedderError",
+    "HashingEmbedder",
+    "MarkdownChunker",
+    "OpenAICompatibleEmbedder",
+    "ParentChildChunker",
+    "RecursiveChunker",
+    "SemanticChunker",
+    "SentenceChunker",
+    "STRATEGIES",
+    "TokenChunker",
+    "__version__",
+    "chunk",
+    "chunk_file",
+    "get_chunker",
+    "get_counter",
+    "to_jsonl",
+]
+
+
+def get_chunker(strategy: str = "recursive", **kwargs) -> BaseChunker:
+    """Build a chunker by name. See ``STRATEGIES`` for available names."""
+    try:
+        cls = STRATEGIES[strategy]
+    except KeyError:
+        raise ValueError(
+            f"unknown strategy {strategy!r}; expected one of {sorted(STRATEGIES)}"
+        ) from None
+    return cls(**kwargs)
+
+
+def chunk(text: str, strategy: str = "recursive", **kwargs) -> list[Chunk]:
+    """One-shot helper: ``chunk(text, strategy, **chunker_kwargs)``.
+
+    Uniform budget names: ``max_chars``/``overlap_chars`` are accepted by
+    every strategy. For ``strategy="token"`` they are interpreted as token
+    budgets (mapped to ``max_tokens``/``overlap_tokens``) — tokens are what
+    that strategy measures, and the char-named arguments keep call sites
+    strategy-agnostic.
+    """
+    if strategy == "token":
+        if "max_tokens" not in kwargs and "max_chars" in kwargs:
+            kwargs["max_tokens"] = kwargs.pop("max_chars")
+        if "overlap_tokens" not in kwargs and "overlap_chars" in kwargs:
+            kwargs["overlap_tokens"] = kwargs.pop("overlap_chars")
+    return get_chunker(strategy, **kwargs).chunk(text)

worddael-0.1.0rc3/src/worddael/adapters.py

@@ -0,0 +1,45 @@
+"""Ecosystem adapters: plug worddael into frameworks users already run.
+
+LangChain: :class:`LangChainChunker` subclasses
+``langchain_text_splitters.TextSplitter`` (optional import), so it works
+with ``split_documents`` / ``create_documents`` and drops into any existing
+RAG pipeline as a drop-in replacement for RecursiveCharacterTextSplitter.
+"""
+
+from __future__ import annotations
+
+from . import get_chunker
+from .chunkers import BaseChunker
+
+try:  # optional dependency — the adapter degrades to a plain object otherwise
+    from langchain_text_splitters import TextSplitter as _LCTextSplitter
+
+    _HAS_LANGCHAIN = True
+except ImportError:  # pragma: no cover
+    _LCTextSplitter = object  # type: ignore[assignment,misc]
+    _HAS_LANGCHAIN = False
+
+__all__ = ["LangChainChunker", "has_langchain"]
+
+
+def has_langchain() -> bool:
+    return _HAS_LANGCHAIN
+
+
+class LangChainChunker(_LCTextSplitter):
+    """worddael chunking behind the LangChain ``TextSplitter`` interface.
+
+    Example::
+
+        from worddael.adapters import LangChainChunker
+        splitter = LangChainChunker(strategy="recursive", max_chars=500, overlap_chars=50)
+        docs = splitter.create_documents([long_text])   # full LC Document flow
+    """
+
+    def __init__(self, strategy: str = "recursive", **chunker_kwargs) -> None:
+        if _HAS_LANGCHAIN:
+            super().__init__()
+        self._chunker: BaseChunker = get_chunker(strategy, **chunker_kwargs)
+
+    def split_text(self, text: str) -> list[str]:
+        return [c.text for c in self._chunker.chunk(text)]