rag-hygiene 0.1.0__tar.gz

rag_hygiene-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 rag-hygiene contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rag_hygiene-0.1.0/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.4
+Name: rag-hygiene
+Version: 0.1.0
+Summary: Find and automatically repair broken RAG chunks before you embed them. Zero dependencies.
+Author: OnProject
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/OnProject/rag-hygiene
+Project-URL: Repository, https://github.com/OnProject/rag-hygiene
+Project-URL: Issues, https://github.com/OnProject/rag-hygiene/issues
+Keywords: rag,chunking,retrieval,llm,embeddings,vector-database,linter,cleanup
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# rag-hygiene
+
+**Find and repair broken RAG chunks before you embed them.**
+
+Most RAG failures are decided before retrieval ever runs. A chunk that opens in
+the middle of a sentence embeds two unrelated claims. A table split from its
+header embeds a grid of numbers with no column names. A PDF that broke a word
+across lines embeds `docu` and `ment` as one token. None of this is visible until
+answers start being subtly wrong months later.
+
+`rag-hygiene` sits between your splitter and your vector database. It diagnoses
+nine classes of chunk damage and **repairs seven of them automatically** — no
+LLM calls, no embeddings, no eval set, no network, no dependencies.
+
+```bash
+pip install rag-hygiene
+```
+
+## Quick start
+
+```bash
+rag-hygiene scan chunks.json          # what is wrong?
+rag-hygiene fix  chunks.json          # repair it → chunks.clean.json
+rag-hygiene fix  chunks.json --dry-run -v   # preview the diff first
+```
+
+```
+2 chunks scanned
+
+mid-sentence-start  Chunk resumes a cut sentence
+  [high] c2  opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable
+
+2 finding(s)  1 high  1 low  · 1 auto-fixable
+run `rag-hygiene fix` to repair, or add --dry-run to preview
+```
+
+## The repair that matters
+
+```
+before                                    after
+────────────────────────────────────      ────────────────────────────────────
+[chunk 7]  …因此公司营收同比增长            [chunk 7]  …因此公司营收同比增长 40%，
+[chunk 8]  40%，超出市场预期。                        超出市场预期。
+                                          ✓ merged  ✓ heading "Q3 财报" injected
+```
+
+Every repair is **lossless or a rejoin of text that was already adjacent**.
+Nothing is summarised, paraphrased, or invented — which is what lets it run
+unattended in a build step.
+
+## Rules
+
+| Rule | Severity | Auto-fix | What it catches |
+|---|---|---|---|
+| `empty-chunk` | high | ✅ removed | Whitespace-only chunks; wasted embedding slots |
+| `mid-sentence-start` | high | ✅ merged | Chunk resumes a sentence the previous one cut |
+| `tiny-chunk` | medium | ✅ merged | Under 20 chars of prose — no meaning to embed |
+| `unclosed-fence` | medium | ✅ closed | Code fence opens, never closes — rest of prompt becomes code |
+| `orphan-table` | medium | ✅ header restored | Markdown table rows whose header landed in another chunk |
+| `near-duplicate` | medium | — | ≥72% word-sequence overlap; dupes split ranking mass |
+| `oversized-chunk` | low | — | Over 4000 chars — retrieval dilutes |
+| `pdf-noise` | low | ✅ normalised | Hyphenated line breaks, column padding, page gaps |
+| `missing-heading` | low | ✅ injected | A heading exists in the document but the chunk can't see it |
+
+`near-duplicate` and `oversized-chunk` are deliberately **not** auto-repaired:
+dropping a duplicate or truncating text changes what the retriever can find, and
+that is an editorial decision, not a mechanical one. The tool reports them and
+stays out of the way.
+
+## Python API
+
+```python
+from rag_hygiene import check, repair
+
+chunks = [
+    {"id": "a", "text": "Revenue grew by",        "source": "q3.md", "order": 0},
+    {"id": "b", "text": "40% year over year.",    "source": "q3.md", "order": 1},
+    {"id": "c", "text": "   ",                    "source": "q3.md", "order": 2},
+]
+
+report = check(chunks)
+report.count("high")          # 2
+report.by_rule()              # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
+report.fails("medium")        # True
+
+clean, report = repair(chunks)
+clean[0].text                 # 'Revenue grew by 40% year over year.'
+clean[0].metadata['merged_ids']   # ['b']  ← provenance is never lost
+```
+
+Accepts `Chunk` objects, dicts (`text` / `page_content` / `content`), or bare
+strings. Works with LangChain and LlamaIndex document exports unchanged.
+
+## CI gate
+
+The exit code follows the linter convention, so it drops into any pipeline:
+
+```yaml
+- run: pip install rag-hygiene
+- run: rag-hygiene scan chunks.json --fail-on high
+```
+
+| exit | meaning |
+|---|---|
+| 0 | clean, or the `--fail-on` gate was not tripped |
+| 1 | a finding at or above `--fail-on` remains |
+| 2 | bad input (missing file, unsupported format, unknown rule) |
+| 3 | internal error |
+
+## Formats
+
+Reads `.json`, `.jsonl`, and `.txt` / `.md`. A JSON payload may be a bare list or
+an object with a `chunks` / `documents` / `data` / `items` key. Output preserves
+the input format.
+
+## Design notes
+
+Three decisions shape the behaviour:
+
+**It joins only definite continuations.** A chunk opening with `40% …`, `, and
+then`, or a lowercase word is provably a continuation, so it gets merged. A chunk
+opening with `Revenue grew` *might* have lost its predecessor — that is reported
+at high severity but **not** auto-merged, because guessing would rewrite evidence.
+`rag-hygiene scan --fail-on high` still catches it; you decide.
+
+**Repairs are idempotent.** Running `fix` twice produces the same bytes as running
+it once. That is asserted in the test suite (`test_repair_converges`), because a
+formatter that keeps drifting is worse than no formatter.
+
+**`missing-heading` converges.** A document that carries no heading anywhere gets
+one `info`-level finding, not one per chunk — there is nothing to repair, and
+flooding the report would make the gate useless. A document with a heading but
+unlabelled chunks gets per-chunk findings that `fix` actually resolves.
+
+## Comparison
+
+`chunklint` (May 2026) performs overlapping static checks and depends on
+pydantic, rich, typer, rapidfuzz and pyyaml. `rag-hygiene` differs in three ways:
+it **repairs** rather than only reporting, it has **zero dependencies**, and it
+ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after
+`。`, so the usual whitespace heuristics silently do nothing on it).
+
+## License
+
+MIT

rag_hygiene-0.1.0/README.md

@@ -0,0 +1,145 @@
+# rag-hygiene
+
+**Find and repair broken RAG chunks before you embed them.**
+
+Most RAG failures are decided before retrieval ever runs. A chunk that opens in
+the middle of a sentence embeds two unrelated claims. A table split from its
+header embeds a grid of numbers with no column names. A PDF that broke a word
+across lines embeds `docu` and `ment` as one token. None of this is visible until
+answers start being subtly wrong months later.
+
+`rag-hygiene` sits between your splitter and your vector database. It diagnoses
+nine classes of chunk damage and **repairs seven of them automatically** — no
+LLM calls, no embeddings, no eval set, no network, no dependencies.
+
+```bash
+pip install rag-hygiene
+```
+
+## Quick start
+
+```bash
+rag-hygiene scan chunks.json          # what is wrong?
+rag-hygiene fix  chunks.json          # repair it → chunks.clean.json
+rag-hygiene fix  chunks.json --dry-run -v   # preview the diff first
+```
+
+```
+2 chunks scanned
+
+mid-sentence-start  Chunk resumes a cut sentence
+  [high] c2  opens mid-sentence ('40% year over year, beat'); its embedding mixes two unrelated claims ·fixable
+
+2 finding(s)  1 high  1 low  · 1 auto-fixable
+run `rag-hygiene fix` to repair, or add --dry-run to preview
+```
+
+## The repair that matters
+
+```
+before                                    after
+────────────────────────────────────      ────────────────────────────────────
+[chunk 7]  …因此公司营收同比增长            [chunk 7]  …因此公司营收同比增长 40%，
+[chunk 8]  40%，超出市场预期。                        超出市场预期。
+                                          ✓ merged  ✓ heading "Q3 财报" injected
+```
+
+Every repair is **lossless or a rejoin of text that was already adjacent**.
+Nothing is summarised, paraphrased, or invented — which is what lets it run
+unattended in a build step.
+
+## Rules
+
+| Rule | Severity | Auto-fix | What it catches |
+|---|---|---|---|
+| `empty-chunk` | high | ✅ removed | Whitespace-only chunks; wasted embedding slots |
+| `mid-sentence-start` | high | ✅ merged | Chunk resumes a sentence the previous one cut |
+| `tiny-chunk` | medium | ✅ merged | Under 20 chars of prose — no meaning to embed |
+| `unclosed-fence` | medium | ✅ closed | Code fence opens, never closes — rest of prompt becomes code |
+| `orphan-table` | medium | ✅ header restored | Markdown table rows whose header landed in another chunk |
+| `near-duplicate` | medium | — | ≥72% word-sequence overlap; dupes split ranking mass |
+| `oversized-chunk` | low | — | Over 4000 chars — retrieval dilutes |
+| `pdf-noise` | low | ✅ normalised | Hyphenated line breaks, column padding, page gaps |
+| `missing-heading` | low | ✅ injected | A heading exists in the document but the chunk can't see it |
+
+`near-duplicate` and `oversized-chunk` are deliberately **not** auto-repaired:
+dropping a duplicate or truncating text changes what the retriever can find, and
+that is an editorial decision, not a mechanical one. The tool reports them and
+stays out of the way.
+
+## Python API
+
+```python
+from rag_hygiene import check, repair
+
+chunks = [
+    {"id": "a", "text": "Revenue grew by",        "source": "q3.md", "order": 0},
+    {"id": "b", "text": "40% year over year.",    "source": "q3.md", "order": 1},
+    {"id": "c", "text": "   ",                    "source": "q3.md", "order": 2},
+]
+
+report = check(chunks)
+report.count("high")          # 2
+report.by_rule()              # {'empty-chunk': 1, 'mid-sentence-start': 1, ...}
+report.fails("medium")        # True
+
+clean, report = repair(chunks)
+clean[0].text                 # 'Revenue grew by 40% year over year.'
+clean[0].metadata['merged_ids']   # ['b']  ← provenance is never lost
+```
+
+Accepts `Chunk` objects, dicts (`text` / `page_content` / `content`), or bare
+strings. Works with LangChain and LlamaIndex document exports unchanged.
+
+## CI gate
+
+The exit code follows the linter convention, so it drops into any pipeline:
+
+```yaml
+- run: pip install rag-hygiene
+- run: rag-hygiene scan chunks.json --fail-on high
+```
+
+| exit | meaning |
+|---|---|
+| 0 | clean, or the `--fail-on` gate was not tripped |
+| 1 | a finding at or above `--fail-on` remains |
+| 2 | bad input (missing file, unsupported format, unknown rule) |
+| 3 | internal error |
+
+## Formats
+
+Reads `.json`, `.jsonl`, and `.txt` / `.md`. A JSON payload may be a bare list or
+an object with a `chunks` / `documents` / `data` / `items` key. Output preserves
+the input format.
+
+## Design notes
+
+Three decisions shape the behaviour:
+
+**It joins only definite continuations.** A chunk opening with `40% …`, `, and
+then`, or a lowercase word is provably a continuation, so it gets merged. A chunk
+opening with `Revenue grew` *might* have lost its predecessor — that is reported
+at high severity but **not** auto-merged, because guessing would rewrite evidence.
+`rag-hygiene scan --fail-on high` still catches it; you decide.
+
+**Repairs are idempotent.** Running `fix` twice produces the same bytes as running
+it once. That is asserted in the test suite (`test_repair_converges`), because a
+formatter that keeps drifting is worse than no formatter.
+
+**`missing-heading` converges.** A document that carries no heading anywhere gets
+one `info`-level finding, not one per chunk — there is nothing to repair, and
+flooding the report would make the gate useless. A document with a heading but
+unlabelled chunks gets per-chunk findings that `fix` actually resolves.
+
+## Comparison
+
+`chunklint` (May 2026) performs overlapping static checks and depends on
+pydantic, rich, typer, rapidfuzz and pyyaml. `rag-hygiene` differs in three ways:
+it **repairs** rather than only reporting, it has **zero dependencies**, and it
+ships CJK sentence segmentation (Chinese/Japanese/Korean text has no spaces after
+`。`, so the usual whitespace heuristics silently do nothing on it).
+
+## License
+
+MIT

rag_hygiene-0.1.0/pyproject.toml

@@ -0,0 +1,61 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rag-hygiene"
+version = "0.1.0"
+description = "Find and automatically repair broken RAG chunks before you embed them. Zero dependencies."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "OnProject" }]
+keywords = [
+    "rag",
+    "chunking",
+    "retrieval",
+    "llm",
+    "embeddings",
+    "vector-database",
+    "linter",
+    "cleanup",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Quality Assurance",
+    "Topic :: Text Processing :: Linguistic",
+    "Typing :: Typed",
+]
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "build", "twine"]
+
+[project.scripts]
+rag-hygiene = "rag_hygiene.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/OnProject/rag-hygiene"
+Repository = "https://github.com/OnProject/rag-hygiene"
+Issues = "https://github.com/OnProject/rag-hygiene/issues"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+rag_hygiene = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"

rag_hygiene-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

rag_hygiene-0.1.0/src/rag_hygiene/__init__.py

@@ -0,0 +1,87 @@
+"""rag-hygiene — find and repair broken RAG chunks before you embed them.
+
+    >>> from rag_hygiene import check, repair
+    >>> chunks = [{"id": "a", "text": "Revenue grew by"}, {"id": "b", "text": "40% last quarter."}]
+    >>> report = check(chunks)
+    >>> report.count("high")
+    1
+    >>> clean, fixes = repair(chunks)
+    >>> clean[0].text
+    'Revenue grew by 40% last quarter.'
+
+Zero dependencies, no network, no model calls — every rule is deterministic, so
+this can run in a pre-commit hook or a CI job.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Iterable
+
+from .fix import fix_chunks
+from .models import Chunk, Finding, Fix, Report
+from .rules import RULES
+
+__version__ = "0.1.0"
+__all__ = [
+    "Chunk",
+    "Finding",
+    "Fix",
+    "Report",
+    "RULES",
+    "check",
+    "repair",
+    "explain",
+    "__version__",
+]
+
+
+def _coerce(chunks: Iterable[Any]) -> list[Chunk]:
+    return [Chunk.from_obj(c, i) for i, c in enumerate(chunks)]
+
+
+def check(chunks: Iterable[Any], *, min_severity: str = "info") -> Report:
+    """Diagnose ``chunks`` without changing them.
+
+    Accepts a list of :class:`Chunk`, dicts with a ``text`` (or ``page_content``)
+    key, or bare strings.
+    """
+    from .rules import scan
+
+    items = _coerce(chunks)
+    report = Report(chunks_scanned=len(items), chunks_after=len(items))
+    report.findings = scan(items, min_severity=min_severity)
+    report.chars_before = report.chars_after = sum(len(c.text) for c in items)
+    return report
+
+
+def repair(
+    chunks: Iterable[Any],
+    *,
+    only: set[str] | None = None,
+    inject_heading: bool = True,
+) -> tuple[list[Chunk], Report]:
+    """Repair ``chunks`` and return ``(repaired_chunks, report)``.
+
+    The report carries both the repairs that were applied and the problems that
+    remain (duplicates, oversized chunks) because those need a human decision.
+    """
+    from .rules import scan
+
+    items = _coerce(chunks)
+    repaired, fix_report = fix_chunks(items, only=only, inject_heading=inject_heading)
+    findings = scan(repaired)
+    fix_report.findings = findings
+    return repaired, fix_report
+
+
+def explain(rule_id: str) -> str:
+    """Why a rule exists and whether it auto-repairs."""
+    from .rules import RULE_INDEX
+
+    if rule_id not in RULE_INDEX:
+        raise KeyError(f"unknown rule {rule_id!r}; run `rag-hygiene rules`")
+    rule = RULE_INDEX[rule_id]
+    return (
+        f"{rule.id} ({rule.severity}, {'auto-fixable' if rule.fixable else 'manual'})\n"
+        f"  {rule.title}"
+    )