chunklet-py 2.2.0__tar.gz → 2.3.0__tar.gz

{chunklet_py-2.2.0 → chunklet_py-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: chunklet-py
-Version: 2.2.0
+Version: 2.3.0
 Summary: High-fidelity context-aware chunking and interactive visualization for RAG. Advanced segmentation for code and documents, because your LLM is only as smart as the fragments you feed it.
 Author-email: speedyk_005 <speedy40115719@gmail.com>
 License-Expression: MIT
@@ -15,7 +15,6 @@ Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: Information Technology
 Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
@@ -23,21 +22,24 @@ Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Text Processing
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.10
+Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: loguru<1.0,>=0.7.3
 Requires-Dist: pysbd<1.0,>=0.3.4
 Requires-Dist: sentsplit<2.0,>=1.0.8
-Requires-Dist: sentencex<=0.6.1
+Requires-Dist: sentencex<2.0,>=1.0.0; platform_system != "Android"
+Requires-Dist: sentencex<=0.6.1; platform_system == "Android"
 Requires-Dist: indic-nlp-library<1.0,>=0.92
 Requires-Dist: py3langid<1.0,>=0.3.0
 Requires-Dist: mpire<3.0,>=2.10.2
-Requires-Dist: python-box<8.0,>=7.3.2
-Requires-Dist: more-itertools<11.0,>=10.6.0
+Requires-Dist: dotdict3<2.0,>=1.1.0
+Requires-Dist: more-itertools<12.0,>=10.6.0
 Requires-Dist: regex>=2025.7.29
-Requires-Dist: pydantic<3.0,>=2.11.6
+Requires-Dist: pydantic<3.0,>=2.11.0
 Requires-Dist: typer<1.0,>=0.19.0
 Requires-Dist: charset-normalizer<4.0,>=3.4.2
+Requires-Dist: setuptools<81
 Provides-Extra: structured-document
 Requires-Dist: pdfminer.six>=20250324; extra == "structured-document"
 Requires-Dist: python-docx<2.0,>=1.2.0; extra == "structured-document"
@@ -59,13 +61,17 @@ Provides-Extra: visualization
 Requires-Dist: uvicorn<1.0,>=0.35.0; extra == "visualization"
 Requires-Dist: fastapi<1.0,>=0.116.0; extra == "visualization"
 Requires-Dist: python-multipart<1.0,>=0.0.20; extra == "visualization"
+Requires-Dist: msgpack<2.0,>=1.1.0; extra == "visualization"
 Requires-Dist: aiofiles==25.1.0; extra == "visualization"
+Provides-Extra: viz
+Requires-Dist: chunklet-py[visualization]; extra == "viz"
 Provides-Extra: all
 Requires-Dist: chunklet-py[code,structured-document,visualization]; extra == "all"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.3.5; extra == "dev"
 Requires-Dist: pytest-cov>=6.2.1; extra == "dev"
 Requires-Dist: pytest-mock>=3.14.1; extra == "dev"
+Requires-Dist: pytest-timeout>=2.3.1; extra == "dev"
 Requires-Dist: ruff>=0.14.14; extra == "dev"
 Provides-Extra: docs
 Requires-Dist: mike>=2.1.2; extra == "docs"
@@ -80,17 +86,14 @@ Dynamic: license-file
 # 🧩 Chunklet-py
 
 <p align="center">
-  <img src="https://github.com/speedyk-005/chunklet-py/blob/main/logo_with_tagline.png?raw=true" alt="Chunklet-py Logo" width="300"/>
+  <img src="https://github.com/speedyk-005/chunklet-py/blob/main/logo_with_tagline.svg?raw=true" alt="Chunklet-py Logo" width="300"/>
 </p>
 
-“One library to split them all: Sentence, Code, Docs”
-
-> [!WARNING]
-> **Quick heads up!** Version 2 has some breaking changes. No worries though - check our [Migration Guide](https://speedyk-005.github.io/chunklet-py/latest/migration/) for a smooth upgrade!
-
-Hey! Welcome. Let's make some text chunking magic happen.
+<p align="center">
+  “One library to split them all: Sentence, Code, Docs”
+</p>
 
-[![Python Version](https://img.shields.io/badge/Python-3.10%20--%203.14-blue)](https://www.python.org/downloads/)
+[![Python Version](https://img.shields.io/badge/Python-3.11%20--%203.14-blue)](https://www.python.org/downloads/)
 [![PyPI](https://img.shields.io/pypi/v/chunklet-py)](https://pypi.org/project/chunklet-py)
 [![PyPI Downloads](https://static.pepy.tech/personalized-badge/chunklet-py?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=BLUE&left_text=downloads)](https://pepy.tech/projects/chunklet-py)
 [![Coverage Status](https://coveralls.io/repos/github/speedyk-005/chunklet-py/badge.svg?branch=main)](https://coveralls.io/github/speedyk-005/chunklet-py?branch=main)
@@ -100,6 +103,13 @@ Hey! Welcome. Let's make some text chunking magic happen.
 [![CodeFactor](https://www.codefactor.io/repository/github/speedyk-005/chunklet-py/badge)](https://www.codefactor.io/repository/github/speedyk-005/chunklet-py)
 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/speedyk-005/chunklet-py)
 
+> [!WARNING]
+> **Quick heads up!** Version 2 has some breaking changes. No worries though - check our [Migration Guide](https://speedyk-005.github.io/chunklet-py/latest/migration/) for a smooth upgrade!
+
+---
+
+Hey! Welcome. Let's make some text chunking magic happen.
+
 <p align="center">
   <a href="https://speedyk-005.github.io/chunklet-py/latest" target="_blank" rel="noopener noreferrer">
     -- documentation site --
@@ -168,7 +178,7 @@ Ready to get Chunklet-py running? Awesome! Let's get you set up quickly and pain
 
 > [!NOTE]
 > **chunklet-py (aka chunklet)** — The old `chunklet` package is no longer maintained. Use `chunklet-py` to get the latest version.
-    
+
 ### The Quick & Easy Way
 
 The simplest way to get started is with pip:
@@ -179,17 +189,28 @@ pip install chunklet-py
 chunklet --version
 ```
 
+> [!TIP]
+> <b>Termux (Android)</b>
+>
+> No rust toolchain on Termux (especially python 3.13)? Install pydantic-core pre-built wheels first then retry installing chunklet-py:
+>
+> ```bash
+> pip install typing-extensions
+> pip install pydantic-core --index-url https://termux-user-repository.github.io/pypi/
+> pip install "pydantic>=2.12.4,<2.13"
+> ```
+
 That's it! You're all set to start chunking.
 
 ### Extra Features (Optional)
 
 Want to unlock more Chunklet-py superpowers? Add these optional dependencies based on what you need:
 
-*   **Document Processing:** For handling `.pdf`, `.docx`, `.epub`, and other document formats:
+*   **Structured Documents:** For handling `.pdf`, `.docx`, `.epub`, and other document formats:
     ```bash
     pip install "chunklet-py[structured-document]"
     ```
-*   **Code Chunking:** For advanced code analysis and chunking features:
+*   **Code Chunking:** For Language-agnostic code chunking features:
     ```bash
     pip install "chunklet-py[code]"
     ```
@@ -197,7 +218,7 @@ Want to unlock more Chunklet-py superpowers? Add these optional dependencies bas
     ```bash
     pip install "chunklet-py[visualization]"
     ```
-*   **All Extras:** To install all optional dependencies:
+* **All Extras:** To install all optional dependencies:
     ```bash
     pip install "chunklet-py[all]"
     ```
@@ -237,7 +258,8 @@ Go forth and code! (And remember, good developers write tests. We appreciate exc
 
 ## Quick Reference 🛠️
 
-For the exhaustive details that I know you're probably avoiding, check the [official docs](https://speedyk-005.github.io/chunklet-py/latest/).
+> [!NOTE]
+> For the exhaustive details that I know you're probably avoiding, check the [official docs](https://speedyk-005.github.io/chunklet-py/latest/).
 
 ### The Constraint-Based Logic
 
@@ -245,9 +267,6 @@ Chunklet-py is basically a "choose your own adventure" for data. It's constraint
 
 **The Golden Rule:** You must provide at least one constraint, or the chunker has no idea when to stop.
 
-> [!NOTE]
-> A quick overview — for the complete picture, check the [detailed docs](https://speedyk-005.github.io/chunklet-py/latest/).
-
 ### Core Imports
 
 Pick your weapon based on whatever data mess you're currently cleaning up.

{chunklet_py-2.2.0 → chunklet_py-2.3.0}/README.md

@@ -1,17 +1,14 @@
 # 🧩 Chunklet-py
 
 <p align="center">
-  <img src="https://github.com/speedyk-005/chunklet-py/blob/main/logo_with_tagline.png?raw=true" alt="Chunklet-py Logo" width="300"/>
+  <img src="https://github.com/speedyk-005/chunklet-py/blob/main/logo_with_tagline.svg?raw=true" alt="Chunklet-py Logo" width="300"/>
 </p>
 
-“One library to split them all: Sentence, Code, Docs”
-
-> [!WARNING]
-> **Quick heads up!** Version 2 has some breaking changes. No worries though - check our [Migration Guide](https://speedyk-005.github.io/chunklet-py/latest/migration/) for a smooth upgrade!
-
-Hey! Welcome. Let's make some text chunking magic happen.
+<p align="center">
+  “One library to split them all: Sentence, Code, Docs”
+</p>
 
-[![Python Version](https://img.shields.io/badge/Python-3.10%20--%203.14-blue)](https://www.python.org/downloads/)
+[![Python Version](https://img.shields.io/badge/Python-3.11%20--%203.14-blue)](https://www.python.org/downloads/)
 [![PyPI](https://img.shields.io/pypi/v/chunklet-py)](https://pypi.org/project/chunklet-py)
 [![PyPI Downloads](https://static.pepy.tech/personalized-badge/chunklet-py?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=BLUE&left_text=downloads)](https://pepy.tech/projects/chunklet-py)
 [![Coverage Status](https://coveralls.io/repos/github/speedyk-005/chunklet-py/badge.svg?branch=main)](https://coveralls.io/github/speedyk-005/chunklet-py?branch=main)
@@ -21,6 +18,13 @@ Hey! Welcome. Let's make some text chunking magic happen.
 [![CodeFactor](https://www.codefactor.io/repository/github/speedyk-005/chunklet-py/badge)](https://www.codefactor.io/repository/github/speedyk-005/chunklet-py)
 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/speedyk-005/chunklet-py)
 
+> [!WARNING]
+> **Quick heads up!** Version 2 has some breaking changes. No worries though - check our [Migration Guide](https://speedyk-005.github.io/chunklet-py/latest/migration/) for a smooth upgrade!
+
+---
+
+Hey! Welcome. Let's make some text chunking magic happen.
+
 <p align="center">
   <a href="https://speedyk-005.github.io/chunklet-py/latest" target="_blank" rel="noopener noreferrer">
     -- documentation site --
@@ -89,7 +93,7 @@ Ready to get Chunklet-py running? Awesome! Let's get you set up quickly and pain
 
 > [!NOTE]
 > **chunklet-py (aka chunklet)** — The old `chunklet` package is no longer maintained. Use `chunklet-py` to get the latest version.
-    
+
 ### The Quick & Easy Way
 
 The simplest way to get started is with pip:
@@ -100,17 +104,28 @@ pip install chunklet-py
 chunklet --version
 ```
 
+> [!TIP]
+> <b>Termux (Android)</b>
+>
+> No rust toolchain on Termux (especially python 3.13)? Install pydantic-core pre-built wheels first then retry installing chunklet-py:
+>
+> ```bash
+> pip install typing-extensions
+> pip install pydantic-core --index-url https://termux-user-repository.github.io/pypi/
+> pip install "pydantic>=2.12.4,<2.13"
+> ```
+
 That's it! You're all set to start chunking.
 
 ### Extra Features (Optional)
 
 Want to unlock more Chunklet-py superpowers? Add these optional dependencies based on what you need:
 
-*   **Document Processing:** For handling `.pdf`, `.docx`, `.epub`, and other document formats:
+*   **Structured Documents:** For handling `.pdf`, `.docx`, `.epub`, and other document formats:
     ```bash
     pip install "chunklet-py[structured-document]"
     ```
-*   **Code Chunking:** For advanced code analysis and chunking features:
+*   **Code Chunking:** For Language-agnostic code chunking features:
     ```bash
     pip install "chunklet-py[code]"
     ```
@@ -118,7 +133,7 @@ Want to unlock more Chunklet-py superpowers? Add these optional dependencies bas
     ```bash
     pip install "chunklet-py[visualization]"
     ```
-*   **All Extras:** To install all optional dependencies:
+* **All Extras:** To install all optional dependencies:
     ```bash
     pip install "chunklet-py[all]"
     ```
@@ -158,7 +173,8 @@ Go forth and code! (And remember, good developers write tests. We appreciate exc
 
 ## Quick Reference 🛠️
 
-For the exhaustive details that I know you're probably avoiding, check the [official docs](https://speedyk-005.github.io/chunklet-py/latest/).
+> [!NOTE]
+> For the exhaustive details that I know you're probably avoiding, check the [official docs](https://speedyk-005.github.io/chunklet-py/latest/).
 
 ### The Constraint-Based Logic
 
@@ -166,9 +182,6 @@ Chunklet-py is basically a "choose your own adventure" for data. It's constraint
 
 **The Golden Rule:** You must provide at least one constraint, or the chunker has no idea when to stop.
 
-> [!NOTE]
-> A quick overview — for the complete picture, check the [detailed docs](https://speedyk-005.github.io/chunklet-py/latest/).
-
 ### Core Imports
 
 Pick your weapon based on whatever data mess you're currently cleaning up.

{chunklet_py-2.2.0 → chunklet_py-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "chunklet-py"
-version = "2.2.0"
+version = "2.3.0"
 description = "High-fidelity context-aware chunking and interactive visualization for RAG. Advanced segmentation for code and documents, because your LLM is only as smart as the fragments you feed it."
 authors = [
   { name = "speedyk_005", email = "speedy40115719@gmail.com" }
@@ -8,7 +8,7 @@ authors = [
 
 license = "MIT"
 readme = "README.md"
-requires-python = ">=3.10"
+requires-python = ">=3.11"
 
 classifiers = [
   "Development Status :: 5 - Production/Stable",
@@ -16,7 +16,6 @@ classifiers = [
   "Intended Audience :: Science/Research",
   "Intended Audience :: Information Technology",
   "Operating System :: OS Independent",
-  "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
@@ -35,22 +34,26 @@ keywords = [
 ]
 
 dependencies = [
+  "loguru>=0.7.3,<1.0",
   "pysbd>=0.3.4,<1.0",
   "sentsplit>=1.0.8,<2.0",
 
-  # Pinning to the last known working version with a reliable binary/build.
-  # The 1.0.x series has a broken sdist package structure.
-  "sentencex<=0.6.1",
+  # Rust binding (1.0+) for faster performance on supported platforms, legacy for Android.
+  "sentencex>=1.0.0,<2.0; platform_system != 'Android'",
+  "sentencex<=0.6.1; platform_system == 'Android'",
 
   "indic-nlp-library>=0.92,<1.0",
   "py3langid>=0.3.0,<1.0",
   "mpire>=2.10.2,<3.0",
-  "python-box>=7.3.2,<8.0",
-  "more-itertools>=10.6.0,<11.0",
+  "dotdict3>=1.1.0,<2.0",
+  "more-itertools>=10.6.0,<12.0",
   "regex>=2025.7.29",
-  "pydantic>=2.11.6,<3.0",
+  "pydantic>=2.11.0,<3.0",
   "typer>=0.19.0,<1.0",
   "charset-normalizer>=3.4.2,<4.0",
+
+  # sentsplit depends on deprecated pkg_resources removed in setuptools 81.0.0
+  "setuptools<81",
 ]
 
 [project.optional-dependencies]
@@ -83,17 +86,21 @@ visualization = [
   "uvicorn>=0.35.0,<1.0",
   "fastapi>=0.116.0,<1.0",
   "python-multipart>=0.0.20,<1.0",
+  "msgpack>=1.1.0,<2.0",
 
   # Major changes happens often
   "aiofiles==25.1.0",
 ]
 
+viz = ["chunklet-py[visualization]"]
+
 all = ["chunklet-py[structured-document,code,visualization]"]
 
 dev = [
   "pytest>=8.3.5",
   "pytest-cov>=6.2.1",
   "pytest-mock>=3.14.1",
+  "pytest-timeout>=2.3.1",
   "ruff>=0.14.14",
 ]
 
@@ -119,7 +126,7 @@ chunklet = "chunklet.cli:app"
 
 [build-system]
 requires = [
-    "setuptools>=64", 
+    "setuptools>=64",
     "wheel", # Essential for building dependencies that use legacy setup.py (like odfpy)
 ]
 build-backend = "setuptools.build_meta"
@@ -139,7 +146,7 @@ packages = [
 include-package-data = true
 
 [tool.setuptools.package-data]
-chunklet = ["visualizer/static/*", "visualizer/static/**/*"]
+chunklet = ["visualizer/static/**/*"]
 
 [tool.pytest.ini_options]
 minversion = "7.4"
@@ -150,6 +157,7 @@ filterwarnings = [
 
 # disable coverage report to prevent racing with multiprocessing
 # addopts = "--cov=chunklet --cov-report=term-missing:skip-covered --durations=10"
+addopts = "--timeout=60"
 
 testpaths = ["tests"]
 python_files = "test_*.py"
@@ -186,6 +194,9 @@ ignore = ["E501", "E203"]
 # Ignore warnings about File(...)
 "src/chunklet/visualizer/visualizer.py" = ["B008"]
 
+# Ignore E402 - warnings.filterwarnings() must be called before imports to suppress pkg_resources
+"src/chunklet/sentence_splitter/sentence_splitter.py" = ["E402"]
+
 [tool.ruff.format]
 # Replaces Black exclude logic
 exclude = [

{chunklet_py-2.2.0 → chunklet_py-2.3.0}/src/chunklet/__init__.py

@@ -60,9 +60,11 @@ __all__ = [
 
 
 # Map the public names to their sub-module locations
+_PLAIN_TEXT_CHUNKER_MODULE = "chunklet.plain_text_chunker"
+
 _LOOKUP = {
     "SentenceSplitter": "chunklet.sentence_splitter",
-    "PlainTextChunker": "chunklet.plain_text_chunker",
+    "PlainTextChunker": _PLAIN_TEXT_CHUNKER_MODULE,
     "DocumentChunker": "chunklet.document_chunker",
     "CodeChunker": "chunklet.code_chunker",
     "Visualizer": "chunklet.visualizer",
@@ -77,7 +79,7 @@ class _PlainTextChunkerModuleProxy(types.ModuleType):
     """
 
     def __init__(self):
-        super().__init__("chunklet.plain_text_chunker")
+        super().__init__(_PLAIN_TEXT_CHUNKER_MODULE)
         self._PlainTextChunker: typing.Any = None
 
     def _get_deprecated_class(self):
@@ -99,7 +101,7 @@ class _PlainTextChunkerModuleProxy(types.ModuleType):
 
 
 # Register proxy in sys.modules for backward compatibility
-sys.modules["chunklet.plain_text_chunker"] = _PlainTextChunkerModuleProxy()
+sys.modules[_PLAIN_TEXT_CHUNKER_MODULE] = _PlainTextChunkerModuleProxy()
 
 
 def __getattr__(name: str) -> typing.Any:

{chunklet_py-2.2.0 → chunklet_py-2.3.0}/src/chunklet/base_chunker.py

@@ -7,7 +7,7 @@ Defines the interface for chunkers.
 from abc import ABC, abstractmethod
 from collections.abc import Generator
 
-from box import Box
+from dotdict3 import DotDict
 from loguru import logger
 
 
@@ -22,41 +22,41 @@ class BaseChunker(ABC):
         self.verbose = verbose
 
     @abstractmethod
-    def chunk_text(self, *args, **kwargs) -> list[Box]:
+    def chunk_text(self, *args, **kwargs) -> list[DotDict]:
         """
         Extract chunks from text.
 
         Returns:
-            list[Box]: List of chunks with content and metadata.
+            List of chunks with content and metadata.
         """
         pass
 
     @abstractmethod
-    def chunk_texts(self, *args, **kwargs) -> list[list[Box]]:
+    def chunk_texts(self, *args, **kwargs) -> list[list[DotDict]]:
         """
         Process multiple texts.
 
         Returns:
-            list[list[Box]]: List of chunks for each input text.
+            List of chunks for each input text.
         """
         pass
 
     @abstractmethod
-    def chunk_file(self, *args, **kwargs) -> list[Box]:
+    def chunk_file(self, *args, **kwargs) -> list[DotDict]:
         """
         Read and chunk a file.
 
         Returns:
-            list[Box]: List of chunks with content and metadata.
+            List of chunks with content and metadata.
         """
         pass
 
     @abstractmethod
-    def chunk_files(self, *args, **kwargs) -> Generator[Box, None, None]:
+    def chunk_files(self, *args, **kwargs) -> Generator[DotDict, None, None]:
         """
         Process multiple files.
 
         Yields:
-            Box: `Box` object, representing a chunk with its content and metadata.
+            `DotDict` object, representing a chunk with its content and metadata.
         """
         pass