boundary-smart-splitter 1.0.0__tar.gz

boundary_smart_splitter-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 boundary-smart-splitter contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

boundary_smart_splitter-1.0.0/PKG-INFO

@@ -0,0 +1,324 @@
+Metadata-Version: 2.4
+Name: boundary-smart-splitter
+Version: 1.0.0
+Summary: A context-aware, boundary-respecting text splitter for Python.
+Author-email: Jeet Mondal <JeetMondal777@users.noreply.github.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/JeetMondal777/boundary-smart-splitter
+Project-URL: Repository, https://github.com/JeetMondal777/boundary-smart-splitter
+Project-URL: Issues, https://github.com/JeetMondal777/boundary-smart-splitter/issues
+Keywords: text,splitter,nlp,chunking,langchain
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: langchain
+Requires-Dist: langchain-text-splitters>=0.0.1; extra == "langchain"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Dynamic: license-file
+
+# boundary-smart-splitter
+
+> A **boundary-first, size-second** text splitter for Python.
+> LangChain-compatible. Framework-agnostic core. No ML dependencies.
+
+```
+Boundary-smart-splitter respects your text structure.
+Word → Sentence → Paragraph → Section/Topic
+```
+
+Most text splitters hit a character count and slice — regardless of whether they cut through a sentence, a paragraph, or a heading. This library flips that: **chunk_size is measured in semantic units** (words, sentences, paragraphs, sections), and `max_chars` acts only as a hard safety ceiling to protect your embedding model's context window.
+
+---
+
+## Installation
+
+```bash
+pip install boundary-smart-splitter
+```
+
+For LangChain support, install the optional dependency:
+
+```bash
+pip install boundary-smart-splitter[langchain]
+```
+
+**No ML dependencies required.** Every splitter below is deterministic, regex-based, and works 100% offline.
+
+---
+
+## Design Philosophy
+
+| Principle | How it works |
+|---|---|
+| **Boundary-first, size-second** | Clean boundaries are more important than exact character counts |
+| **Semantic units** | `chunk_size` means words for V1, sentences for V2, paragraphs for V3, sections for V4 |
+| **max_chars as safety net** | Never exceeds your embedding model's context window, but only as a last resort |
+| **Graceful fallback chain** | V4 → V3 → V2 → V1 → hard char-cut |
+| **LangChain-compatible, not dependent** | The core never imports LangChain; wrappers are optional |
+
+---
+
+## Quick Start
+
+### V1 — Word-Boundary Splitter
+
+Never cuts a word in half. `chunk_size` = number of words.
+
+```python
+from boundary_smart_splitter import WordSplitter
+
+text = "The quick brown fox jumps over the lazy dog. " * 100
+splitter = WordSplitter(chunk_size=60, max_chars=500, tolerance=10)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+---
+
+### V2 — Sentence-Boundary Splitter
+
+Never cuts a sentence in half. Finds the nearest `.`, `?`, or `!` within tolerance.
+
+```python
+from boundary_smart_splitter import SentenceSplitter
+
+text = "Hello world. How are you? I am fine! " * 100
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+#### Custom Abbreviations
+
+Prevent mid-abbreviation splits (e.g. `Dr.`, `U.S.A.`, `e.g.`):
+
+```python
+splitter = SentenceSplitter(
+    chunk_size=5,
+    max_chars=500,
+    tolerance=2,
+    abbreviations={"Dr.", "U.S.A.", "e.g.", "i.e."},
+)
+```
+
+#### Boundary Preference
+
+```python
+# Forward-first (default): scan forward for sentence end, then backward
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="forward")
+
+# Backward-first: scan backward first, then forward
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="backward")
+```
+
+---
+
+### V3 — Paragraph-Boundary Splitter
+
+Never breaks a paragraph (`\n\n`). `chunk_size` = number of paragraphs.
+
+```python
+from boundary_smart_splitter import ParagraphSplitter
+
+text = "Para one.\n\nPara two.\n\nPara three."
+splitter = ParagraphSplitter(chunk_size=2, max_chars=500, tolerance=1)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+#### Markdown Mode
+
+Respects horizontal rules and headings as extra paragraph boundaries:
+
+```python
+splitter = ParagraphSplitter(
+    chunk_size=2,
+    max_chars=500,
+    tolerance=1,
+    use_markdown_mode=True,
+)
+```
+
+---
+
+### V4 — Structure & Topic-Aware Splitter
+
+Respects headings, numbered sections, and transition phrases. `chunk_size` = number of sections.
+
+```python
+from boundary_smart_splitter import StructureSplitter
+
+text = """
+# Intro
+Welcome to our proposal.
+
+1. Home Page
+The homepage will act as a strong first impression...
+
+2. About Us Page
+To build credibility and trust.
+
+However, we must also consider costs.
+
+3. Services Page
+A detailed breakdown of all services.
+"""
+
+splitter = StructureSplitter(
+    chunk_size=1,
+    max_chars=1500,
+    respect_headings=True,
+    respect_numbered_sections=True,
+    split_on_transitions=True,
+)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print("---")
+    print(chunk)
+```
+
+#### V4 Configuration
+
+| Parameter | Default | Description |
+|---|---|---|
+| `respect_headings` | `True` | Treat `# Heading` as a hard boundary |
+| `respect_numbered_sections` | `True` | Treat `1. Title` as a boundary |
+| `split_on_transitions` | `True` | Treat "However," / "In summary," etc. as boundaries |
+| `transition_phrases` | built-in list | Custom list of transition phrases |
+| `double_newline_as_boundary` | `True` | Treat `\n\n\n` as a boundary |
+
+---
+
+## Fallback Chain
+
+When a single unit exceeds `max_chars`, the library gracefully falls back to the next-smaller boundary:
+
+```
+StructureSplitter (V4)
+  → section too big? → ParagraphSplitter (V3)
+    → paragraph too big? → SentenceSplitter (V2)
+      → sentence too big? → WordSplitter (V1)
+        → word too big? → hard char-cut
+```
+
+This ensures you **never** get a chunk larger than `max_chars`, but you **always** get the cleanest possible boundary.
+
+---
+
+## LangChain Integration
+
+All splitters have LangChain-compatible wrappers. The core never imports LangChain — wrappers are optional.
+
+```python
+from boundary_smart_splitter.langchain import (
+    LangChainWordSplitter,
+    LangChainSentenceSplitter,
+    LangChainParagraphSplitter,
+    LangChainStructureSplitter,
+)
+
+splitter = LangChainStructureSplitter(chunk_size=1, max_chars=1500)
+chunks = splitter.split_text("Your long document here...")
+```
+
+---
+
+## API Comparison
+
+| Splitter | `chunk_size` unit | `max_chars` role | Fallback | Key Params |
+|---|---|---|---|---|
+| `WordSplitter` | words | hard ceiling, never exceeded | hard char-cut | `tolerance` |
+| `SentenceSplitter` | sentences | if sentence exceeds it, fall back to V1 | `WordSplitter` | `tolerance`, `boundary_preference`, `abbreviations` |
+| `ParagraphSplitter` | paragraphs | if paragraph exceeds it, fall back to V2 | `SentenceSplitter` | `tolerance`, `overlap`, `paragraph_separator`, `use_markdown_mode` |
+| `StructureSplitter` | sections / topics | if section exceeds it, fall back to V3 | `ParagraphSplitter` | `respect_headings`, `respect_numbered_sections`, `split_on_transitions` |
+
+---
+
+## Why boundary-first matters
+
+**Size-first** splitting (the common approach):
+```
+"...the quick brown fox jum" | "ps over the lazy dog..."
+# Bad: cuts through a word
+```
+
+**Boundary-first** splitting (this library):
+```
+"...the quick brown fox" | "jumps over the lazy dog..."
+# Good: clean word boundary
+```
+
+This becomes critical at sentence level:
+```
+"Please visit the U.S.A. for travel. Yes!"
+# sentence-aware: kept together
+# naive char-cut: "U.S.A." might get split
+```
+
+And at paragraph/section level for RAG:
+```
+"1. Home Page\n\nThe homepage..." | "2. About Us\n\nTo build..."
+# section-aware: one chunk per page/section
+# naive: cuts through the middle of a page description
+```
+
+---
+
+## Features
+
+- **Boundary-first, size-second**: clean boundaries always take priority over exact counts
+- **Semantic units for `chunk_size`**: words count words, sentences count sentences, paragraphs count paragraphs, sections count sections
+- **`max_chars` as universal safety net**: protects embedding model limits without breaking the semantic contract
+- **Graceful fallback chain**: V4 → V3 → V2 → V1, each level catches what the level above can't handle
+- **LangChain-compatible, not dependent**: the core never imports LangChain
+- **Each version is a superset**: V2 includes V1, V3 includes出门 include V2, V4 includes V3
+- **No ML dependencies**: fast, deterministic, offline-capable
+- **Abbreviation-aware**: configurable abbreviation list prevents false sentence breaks
+- **Markdown-aware**: optional Markdown paragraph/heading detection
+- **PDF-friendly**: handles leading whitespace from PDF text extraction
+
+---
+
+## Requirements
+
+- Python >= 3.9
+
+---
+
+## Contributing
+
+Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+---
+
+## License
+
+MIT © 2026

boundary_smart_splitter-1.0.0/README.md

@@ -0,0 +1,291 @@
+# boundary-smart-splitter
+
+> A **boundary-first, size-second** text splitter for Python.
+> LangChain-compatible. Framework-agnostic core. No ML dependencies.
+
+```
+Boundary-smart-splitter respects your text structure.
+Word → Sentence → Paragraph → Section/Topic
+```
+
+Most text splitters hit a character count and slice — regardless of whether they cut through a sentence, a paragraph, or a heading. This library flips that: **chunk_size is measured in semantic units** (words, sentences, paragraphs, sections), and `max_chars` acts only as a hard safety ceiling to protect your embedding model's context window.
+
+---
+
+## Installation
+
+```bash
+pip install boundary-smart-splitter
+```
+
+For LangChain support, install the optional dependency:
+
+```bash
+pip install boundary-smart-splitter[langchain]
+```
+
+**No ML dependencies required.** Every splitter below is deterministic, regex-based, and works 100% offline.
+
+---
+
+## Design Philosophy
+
+| Principle | How it works |
+|---|---|
+| **Boundary-first, size-second** | Clean boundaries are more important than exact character counts |
+| **Semantic units** | `chunk_size` means words for V1, sentences for V2, paragraphs for V3, sections for V4 |
+| **max_chars as safety net** | Never exceeds your embedding model's context window, but only as a last resort |
+| **Graceful fallback chain** | V4 → V3 → V2 → V1 → hard char-cut |
+| **LangChain-compatible, not dependent** | The core never imports LangChain; wrappers are optional |
+
+---
+
+## Quick Start
+
+### V1 — Word-Boundary Splitter
+
+Never cuts a word in half. `chunk_size` = number of words.
+
+```python
+from boundary_smart_splitter import WordSplitter
+
+text = "The quick brown fox jumps over the lazy dog. " * 100
+splitter = WordSplitter(chunk_size=60, max_chars=500, tolerance=10)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+---
+
+### V2 — Sentence-Boundary Splitter
+
+Never cuts a sentence in half. Finds the nearest `.`, `?`, or `!` within tolerance.
+
+```python
+from boundary_smart_splitter import SentenceSplitter
+
+text = "Hello world. How are you? I am fine! " * 100
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+#### Custom Abbreviations
+
+Prevent mid-abbreviation splits (e.g. `Dr.`, `U.S.A.`, `e.g.`):
+
+```python
+splitter = SentenceSplitter(
+    chunk_size=5,
+    max_chars=500,
+    tolerance=2,
+    abbreviations={"Dr.", "U.S.A.", "e.g.", "i.e."},
+)
+```
+
+#### Boundary Preference
+
+```python
+# Forward-first (default): scan forward for sentence end, then backward
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="forward")
+
+# Backward-first: scan backward first, then forward
+splitter = SentenceSplitter(chunk_size=5, max_chars=500, tolerance=2, boundary_preference="backward")
+```
+
+---
+
+### V3 — Paragraph-Boundary Splitter
+
+Never breaks a paragraph (`\n\n`). `chunk_size` = number of paragraphs.
+
+```python
+from boundary_smart_splitter import ParagraphSplitter
+
+text = "Para one.\n\nPara two.\n\nPara three."
+splitter = ParagraphSplitter(chunk_size=2, max_chars=500, tolerance=1)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print(chunk)
+```
+
+#### Markdown Mode
+
+Respects horizontal rules and headings as extra paragraph boundaries:
+
+```python
+splitter = ParagraphSplitter(
+    chunk_size=2,
+    max_chars=500,
+    tolerance=1,
+    use_markdown_mode=True,
+)
+```
+
+---
+
+### V4 — Structure & Topic-Aware Splitter
+
+Respects headings, numbered sections, and transition phrases. `chunk_size` = number of sections.
+
+```python
+from boundary_smart_splitter import StructureSplitter
+
+text = """
+# Intro
+Welcome to our proposal.
+
+1. Home Page
+The homepage will act as a strong first impression...
+
+2. About Us Page
+To build credibility and trust.
+
+However, we must also consider costs.
+
+3. Services Page
+A detailed breakdown of all services.
+"""
+
+splitter = StructureSplitter(
+    chunk_size=1,
+    max_chars=1500,
+    respect_headings=True,
+    respect_numbered_sections=True,
+    split_on_transitions=True,
+)
+chunks = splitter.split(text)
+
+for chunk in chunks:
+    print("---")
+    print(chunk)
+```
+
+#### V4 Configuration
+
+| Parameter | Default | Description |
+|---|---|---|
+| `respect_headings` | `True` | Treat `# Heading` as a hard boundary |
+| `respect_numbered_sections` | `True` | Treat `1. Title` as a boundary |
+| `split_on_transitions` | `True` | Treat "However," / "In summary," etc. as boundaries |
+| `transition_phrases` | built-in list | Custom list of transition phrases |
+| `double_newline_as_boundary` | `True` | Treat `\n\n\n` as a boundary |
+
+---
+
+## Fallback Chain
+
+When a single unit exceeds `max_chars`, the library gracefully falls back to the next-smaller boundary:
+
+```
+StructureSplitter (V4)
+  → section too big? → ParagraphSplitter (V3)
+    → paragraph too big? → SentenceSplitter (V2)
+      → sentence too big? → WordSplitter (V1)
+        → word too big? → hard char-cut
+```
+
+This ensures you **never** get a chunk larger than `max_chars`, but you **always** get the cleanest possible boundary.
+
+---
+
+## LangChain Integration
+
+All splitters have LangChain-compatible wrappers. The core never imports LangChain — wrappers are optional.
+
+```python
+from boundary_smart_splitter.langchain import (
+    LangChainWordSplitter,
+    LangChainSentenceSplitter,
+    LangChainParagraphSplitter,
+    LangChainStructureSplitter,
+)
+
+splitter = LangChainStructureSplitter(chunk_size=1, max_chars=1500)
+chunks = splitter.split_text("Your long document here...")
+```
+
+---
+
+## API Comparison
+
+| Splitter | `chunk_size` unit | `max_chars` role | Fallback | Key Params |
+|---|---|---|---|---|
+| `WordSplitter` | words | hard ceiling, never exceeded | hard char-cut | `tolerance` |
+| `SentenceSplitter` | sentences | if sentence exceeds it, fall back to V1 | `WordSplitter` | `tolerance`, `boundary_preference`, `abbreviations` |
+| `ParagraphSplitter` | paragraphs | if paragraph exceeds it, fall back to V2 | `SentenceSplitter` | `tolerance`, `overlap`, `paragraph_separator`, `use_markdown_mode` |
+| `StructureSplitter` | sections / topics | if section exceeds it, fall back to V3 | `ParagraphSplitter` | `respect_headings`, `respect_numbered_sections`, `split_on_transitions` |
+
+---
+
+## Why boundary-first matters
+
+**Size-first** splitting (the common approach):
+```
+"...the quick brown fox jum" | "ps over the lazy dog..."
+# Bad: cuts through a word
+```
+
+**Boundary-first** splitting (this library):
+```
+"...the quick brown fox" | "jumps over the lazy dog..."
+# Good: clean word boundary
+```
+
+This becomes critical at sentence level:
+```
+"Please visit the U.S.A. for travel. Yes!"
+# sentence-aware: kept together
+# naive char-cut: "U.S.A." might get split
+```
+
+And at paragraph/section level for RAG:
+```
+"1. Home Page\n\nThe homepage..." | "2. About Us\n\nTo build..."
+# section-aware: one chunk per page/section
+# naive: cuts through the middle of a page description
+```
+
+---
+
+## Features
+
+- **Boundary-first, size-second**: clean boundaries always take priority over exact counts
+- **Semantic units for `chunk_size`**: words count words, sentences count sentences, paragraphs count paragraphs, sections count sections
+- **`max_chars` as universal safety net**: protects embedding model limits without breaking the semantic contract
+- **Graceful fallback chain**: V4 → V3 → V2 → V1, each level catches what the level above can't handle
+- **LangChain-compatible, not dependent**: the core never imports LangChain
+- **Each version is a superset**: V2 includes V1, V3 includes出门 include V2, V4 includes V3
+- **No ML dependencies**: fast, deterministic, offline-capable
+- **Abbreviation-aware**: configurable abbreviation list prevents false sentence breaks
+- **Markdown-aware**: optional Markdown paragraph/heading detection
+- **PDF-friendly**: handles leading whitespace from PDF text extraction
+
+---
+
+## Requirements
+
+- Python >= 3.9
+
+---
+
+## Contributing
+
+Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+---
+
+## License
+
+MIT © 2026

boundary_smart_splitter-1.0.0/boundary_smart_splitter/__init__.py

@@ -0,0 +1,15 @@
+"""boundary-smart-splitter: A context-aware, boundary-respecting text splitter."""
+
+from .v1_word import WordSplitter
+from .v2_sentence import SentenceSplitter
+from .v3_paragraph import ParagraphSplitter
+from .v4_structure import StructureSplitter
+
+__version__ = "1.0.0"
+
+__all__ = [
+    "WordSplitter",
+    "SentenceSplitter",
+    "ParagraphSplitter",
+    "StructureSplitter",
+]

boundary_smart_splitter-1.0.0/boundary_smart_splitter/base.py

@@ -0,0 +1,50 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import List
+
+
+class BaseSplitter(ABC):
+    """Abstract base class for all splitters.
+
+    Each splitter measures ``chunk_size`` in its **own semantic unit** —
+    words, sentences, or paragraphs.  A ``max_chars`` ceiling protects
+    embedding-model context limits across all versions.
+
+    Parameters
+    ----------
+    chunk_size : int
+        Target count of semantic units (words / sentences / paragraphs)
+        per chunk.
+    max_chars : int
+        Hard character ceiling — no chunk ever exceeds this length.
+    tolerance : int
+        Number of semantic units to scan forward or backward when
+        adjusting a boundary so it lands on a clean break.
+    """
+
+    def __init__(
+        self,
+        *,
+        chunk_size: int,
+        max_chars: int,
+        tolerance: int,
+    ) -> None:
+        if chunk_size <= 0:
+            raise ValueError("chunk_size must be a positive integer")
+        if max_chars <= 0:
+            raise ValueError("max_chars must be a positive integer")
+        if tolerance < 0:
+            raise ValueError("tolerance must be a non-negative integer")
+        self.chunk_size = chunk_size
+        self.max_chars = max_chars
+        self.tolerance = tolerance
+
+    @abstractmethod
+    def split(self, text: str) -> List[str]:
+        """Split *text* into chunks and return them."""
+        ...
+
+    def _strip_and_filter(self, chunks: list[str]) -> list[str]:
+        """Strip whitespace and remove empty strings from a list of chunks."""
+        return [chunk.strip() for chunk in chunks if chunk.strip()]