autochunks 0.0.8__tar.gz

autochunks-0.0.8/LICENSE

@@ -0,0 +1,15 @@
+Apache License 2.0
+
+Copyright 2026 Sumit Joshi
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

autochunks-0.0.8/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: autochunks
+Version: 0.0.8
+Summary: Autonomous Retrieval Optimization for RAG
+Author: Sumit Joshi
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/s8ilabs/AutoChunks
+Project-URL: Documentation, https://autochunks.readthedocs.io/
+Project-URL: Repository, https://github.com/s8ilabs/AutoChunks
+Project-URL: Issues, https://github.com/s8ilabs/AutoChunks/issues
+Keywords: rag,chunking,retrieval,nlp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: loguru
+Requires-Dist: arize-phoenix>=4.3.0
+Requires-Dist: opentelemetry-api
+Requires-Dist: opentelemetry-sdk
+Requires-Dist: opentelemetry-exporter-otlp
+Requires-Dist: nltk
+Requires-Dist: pymupdf4llm
+Requires-Dist: sentence-transformers
+Requires-Dist: torch
+Requires-Dist: fastapi
+Requires-Dist: uvicorn
+Requires-Dist: python-multipart
+Dynamic: license-file
+
+# AutoChunks
+### The Intelligent Data Optimization Layer for RAG Engineering
+
+[![Version](https://img.shields.io/badge/version-0.08--alpha-blue)](https://github.com/s8ilabs/AutoChunks)
+[![Documentation](https://img.shields.io/badge/docs-read--the--docs-teal)](https://autochunks.readthedocs.io/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-green)](LICENSE)
+
+![AutoChunks Hero](docs/assets/hero_banner.png)
+
+AutoChunks is a specialized engine designed to eliminate the guesswork from Retrieval-Augmented Generation (RAG). By treating chunking as an optimization problem rather than a set of heuristics, it empirically discovers the most performant data structures for your specific documents and retrieval models.
+
+---
+
+## From Heuristics to Evidence
+
+Most RAG pipelines today rely on arbitrary settings—like a 512-token chunk size with a 10% overlap. These values are often chosen without validation, leading to:
+
+*   **Fragmented Context**: Related information is split across multiple retrieval units.
+*   **Semantic Noise**: Poorly defined boundaries dilute the signal-to-noise ratio in LLM prompts.
+*   **Retrieval Gaps**: Critical information hidden in "dead zones" between chunks results in recall failure.
+
+**AutoChunks replaces trial-and-error with a data-driven tournament.** It generates adversarial synthetic ground truth from your documents and pits over 15+ chunking strategies against each other to find the mathematical optimum for your corpus.
+
+---
+
+## Core Pillars
+
+### The Vectorized Tournament
+AutoChunks runs an exhaustive parallel search across multiple strategy families—Recursive, Semantic, Layout-Aware, and Hybrid. Every candidate is evaluated in a high-speed NumPy-accelerated retrieval simulation, measuring performance across hundreds of queries in seconds.
+
+### Adversarial Synthetic QA
+The system performs a structural audit of your documents to generate "needle-in-a-haystack" question-answer pairs. This ensures that your chunking strategy is optimized against real-world search intent, not just random text splits.
+
+### Optimization Goals
+Align your data engineering with your business objectives. Choose from intent-based presets that guide the engine toward specific outcomes:
+*   **Balanced Ranking**: Optimizes for general-purpose retrieval quality.
+*   **Speed and Precision**: Minimizes LLM reading time by prioritizing Rank #1 hits.
+*   **Comprehensive Retrieval**: Prioritizes recall for compliance or legal use cases.
+*   **Cost Efficiency**: Minimizes vector storage and inference costs for massive datasets.
+
+---
+
+## Advanced Feature Set
+
+*   **Hybrid Semantic-Statistical Chunker**: Uses real-time embedding distance analysis to detect topic shifts while maintaining strict token limits.
+*   **Framework Bridges**: Native adapters for LangChain, LlamaIndex, and Haystack, allowing you to benchmark and optimize your existing framework code directly.
+*   **Layout-Aware Processing**: High-fidelity extraction that respects the nested structures of PDFs, HTML sections, and Markdown hierarchies.
+*   **Fidelity Inspector**: A visual debugging dashboard to qualitatively verify how different strategies fragment complex documents.
+*   **Enterprise Security**: Air-gap compatible. Supports local model deployment, SHA-256 binary fingerprinting for data privacy, and SecretStr protection for all cloud credentials.
+
+---
+
+## Quick Start
+
+### Installation
+```bash
+pip install -r requirements.txt
+```
+*Note: For GPU acceleration with Local Embeddings or Ragas, please refer to the [Getting Started guide](docs/getting_started.md).*
+
+### Launch the Dashboard
+The most effective way to optimize your data is through the visual interactive dashboard.
+```bash
+python -m autochunk.web.server
+```
+Navigate to `http://localhost:8000` to begin your first optimization run.
+
+### Python API
+```python
+from autochunk import AutoChunker
+
+# Initialize in Light Mode for rapid iteration
+optimizer = AutoChunker(mode="light")
+
+# Discover the optimal plan for your dataset
+plan, report = optimizer.optimize(
+    documents_path="./my_data_folder",
+    objective="balanced"
+)
+
+# Apply the winning strategy
+chunks = plan.apply("./new_documents", optimizer)
+```
+
+---
+
+## Documentation and Resources
+
+*   [Getting Started](docs/getting_started.md)
+*   [The Optimization Lifecycle](docs/core_concepts/eval_flow.md)
+*   [Metric Definitions and Scoring](docs/core_concepts/evaluation.md)
+*   [RAGAS Semantic Evaluation](docs/guides/ragas_evaluation.md)
+
+---
+
+Developed for the RAG and LLM Community.
+AutoChunks is released under the Apache License 2.0.

autochunks-0.0.8/README.md

@@ -0,0 +1,97 @@
+# AutoChunks
+### The Intelligent Data Optimization Layer for RAG Engineering
+
+[![Version](https://img.shields.io/badge/version-0.08--alpha-blue)](https://github.com/s8ilabs/AutoChunks)
+[![Documentation](https://img.shields.io/badge/docs-read--the--docs-teal)](https://autochunks.readthedocs.io/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-green)](LICENSE)
+
+![AutoChunks Hero](docs/assets/hero_banner.png)
+
+AutoChunks is a specialized engine designed to eliminate the guesswork from Retrieval-Augmented Generation (RAG). By treating chunking as an optimization problem rather than a set of heuristics, it empirically discovers the most performant data structures for your specific documents and retrieval models.
+
+---
+
+## From Heuristics to Evidence
+
+Most RAG pipelines today rely on arbitrary settings—like a 512-token chunk size with a 10% overlap. These values are often chosen without validation, leading to:
+
+*   **Fragmented Context**: Related information is split across multiple retrieval units.
+*   **Semantic Noise**: Poorly defined boundaries dilute the signal-to-noise ratio in LLM prompts.
+*   **Retrieval Gaps**: Critical information hidden in "dead zones" between chunks results in recall failure.
+
+**AutoChunks replaces trial-and-error with a data-driven tournament.** It generates adversarial synthetic ground truth from your documents and pits over 15+ chunking strategies against each other to find the mathematical optimum for your corpus.
+
+---
+
+## Core Pillars
+
+### The Vectorized Tournament
+AutoChunks runs an exhaustive parallel search across multiple strategy families—Recursive, Semantic, Layout-Aware, and Hybrid. Every candidate is evaluated in a high-speed NumPy-accelerated retrieval simulation, measuring performance across hundreds of queries in seconds.
+
+### Adversarial Synthetic QA
+The system performs a structural audit of your documents to generate "needle-in-a-haystack" question-answer pairs. This ensures that your chunking strategy is optimized against real-world search intent, not just random text splits.
+
+### Optimization Goals
+Align your data engineering with your business objectives. Choose from intent-based presets that guide the engine toward specific outcomes:
+*   **Balanced Ranking**: Optimizes for general-purpose retrieval quality.
+*   **Speed and Precision**: Minimizes LLM reading time by prioritizing Rank #1 hits.
+*   **Comprehensive Retrieval**: Prioritizes recall for compliance or legal use cases.
+*   **Cost Efficiency**: Minimizes vector storage and inference costs for massive datasets.
+
+---
+
+## Advanced Feature Set
+
+*   **Hybrid Semantic-Statistical Chunker**: Uses real-time embedding distance analysis to detect topic shifts while maintaining strict token limits.
+*   **Framework Bridges**: Native adapters for LangChain, LlamaIndex, and Haystack, allowing you to benchmark and optimize your existing framework code directly.
+*   **Layout-Aware Processing**: High-fidelity extraction that respects the nested structures of PDFs, HTML sections, and Markdown hierarchies.
+*   **Fidelity Inspector**: A visual debugging dashboard to qualitatively verify how different strategies fragment complex documents.
+*   **Enterprise Security**: Air-gap compatible. Supports local model deployment, SHA-256 binary fingerprinting for data privacy, and SecretStr protection for all cloud credentials.
+
+---
+
+## Quick Start
+
+### Installation
+```bash
+pip install -r requirements.txt
+```
+*Note: For GPU acceleration with Local Embeddings or Ragas, please refer to the [Getting Started guide](docs/getting_started.md).*
+
+### Launch the Dashboard
+The most effective way to optimize your data is through the visual interactive dashboard.
+```bash
+python -m autochunk.web.server
+```
+Navigate to `http://localhost:8000` to begin your first optimization run.
+
+### Python API
+```python
+from autochunk import AutoChunker
+
+# Initialize in Light Mode for rapid iteration
+optimizer = AutoChunker(mode="light")
+
+# Discover the optimal plan for your dataset
+plan, report = optimizer.optimize(
+    documents_path="./my_data_folder",
+    objective="balanced"
+)
+
+# Apply the winning strategy
+chunks = plan.apply("./new_documents", optimizer)
+```
+
+---
+
+## Documentation and Resources
+
+*   [Getting Started](docs/getting_started.md)
+*   [The Optimization Lifecycle](docs/core_concepts/eval_flow.md)
+*   [Metric Definitions and Scoring](docs/core_concepts/evaluation.md)
+*   [RAGAS Semantic Evaluation](docs/guides/ragas_evaluation.md)
+
+---
+
+Developed for the RAG and LLM Community.
+AutoChunks is released under the Apache License 2.0.

autochunks-0.0.8/autochunk/__init__.py

@@ -0,0 +1,9 @@
+import warnings
+# Suppress Pydantic v2 namespace conflicts common in docling models
+warnings.filterwarnings("ignore", message='.*conflict with protected namespace "model_".*', category=UserWarning)
+
+from .autochunker import AutoChunker
+from .embedding.adapter import EmbeddingFn
+from .config import AutoChunkConfig, EvalConfig, ProxyConfig, RetrievalStrategy, SafetyConstraints, ParallelConfig, TokenizerConfig, NetworkConfig, RagasConfig
+from .adapters import AutoChunkLangChainAdapter, AutoChunkLlamaIndexAdapter, AutoChunkHaystackAdapter
+from .storage.plan import Plan

autochunks-0.0.8/autochunk/__main__.py

@@ -0,0 +1,5 @@
+
+from .cli import main
+
+if __name__ == "__main__":
+    main()

autochunks-0.0.8/autochunk/adapters/__init__.py

@@ -0,0 +1,3 @@
+from .langchain import AutoChunkLangChainAdapter
+from .llamaindex import AutoChunkLlamaIndexAdapter
+from .haystack import AutoChunkHaystackAdapter

autochunks-0.0.8/autochunk/adapters/haystack.py

@@ -0,0 +1,68 @@
+
+from __future__ import annotations
+from typing import List, Dict, Any, Optional, Union
+from ..storage.plan import Plan
+from ..autochunker import AutoChunker
+
+try:
+    from haystack import component, Document
+    HAYSTACK_AVAILABLE = True
+except ImportError:
+    # Robust fallback for environment without Haystack
+    def component(cls): return cls
+    def output_types(**kwargs):
+        def decorator(func): return func
+        return decorator
+    component.output_types = output_types
+    class Document: pass
+    HAYSTACK_AVAILABLE = False
+
+@component
+class AutoChunkHaystackAdapter:
+    """
+    Official AutoChunks Adapter for Haystack 2.0.
+    Acts as a Pipeline Component for optimized document splitting.
+    """
+    def __init__(self, plan: Union[Plan, str]):
+        if isinstance(plan, str):
+            self.plan = Plan.read(plan)
+        else:
+            self.plan = plan
+        
+        # Initialize internal engine
+        self.chunker = AutoChunker(
+            embedding_provider=self.plan.embedding.get("name"),
+            embedding_model_or_path=self.plan.embedding.get("model")
+        )
+
+    @component.output_types(documents=List[Document])
+    def run(self, documents: List[Document]):
+        """
+        Implementation of the Haystack Component interface.
+        """
+        if not HAYSTACK_AVAILABLE:
+            raise ImportError("Please install haystack-ai: pip install haystack-ai")
+
+        # Convert Haystack docs to AutoChunks format
+        ac_docs = []
+        for d in documents:
+            ac_docs.append({
+                "id": str(getattr(d, "id", hash(d.content))),
+                "text": d.content,
+                "metadata": d.meta
+            })
+
+        # Process via  pipeline
+        gen_name = self.plan.generator_pipeline.get("name")
+        params = self.plan.generator_pipeline.get("params", {})
+        ac_chunks = self.chunker.apply_with_generator(ac_docs, gen_name, params)
+
+        # Re-wrap as Haystack Documents
+        return {
+            "documents": [
+                Document(
+                    content=ch["text"],
+                    meta={**ch.get("meta", {}), "autochunk_plan_id": self.plan.id}
+                ) for ch in ac_chunks
+            ]
+        }

autochunks-0.0.8/autochunk/adapters/langchain.py

@@ -0,0 +1,81 @@
+
+from __future__ import annotations
+from typing import List, Dict, Any, TYPE_CHECKING, Union
+from ..storage.plan import Plan
+from ..autochunker import AutoChunker, AutoChunkConfig
+
+if TYPE_CHECKING:
+    from langchain_core.documents import Document
+
+try:
+    from langchain_core.documents import BaseDocumentTransformer, Document
+    LANGCHAIN_AVAILABLE = True
+except ImportError:
+    class BaseDocumentTransformer: pass
+    LANGCHAIN_AVAILABLE = False
+
+class AutoChunkLangChainAdapter(BaseDocumentTransformer):
+    """
+    Official AutoChunks Adapter for LangChain.
+    Inherits from BaseDocumentTransformer for seamless integration 
+    into LangChain Indexing and LCEL pipelines.
+    """
+    def __init__(self, plan: Union[Plan, str], config: AutoChunkConfig = None):
+        if isinstance(plan, str):
+            self.plan = Plan.read(plan)
+        else:
+            self.plan = plan
+
+        # We use a configured AutoChunker to execute the plan
+        self.chunker = AutoChunker(
+            embedding_provider=self.plan.embedding.get("name"),
+            embedding_model_or_path=self.plan.embedding.get("model")
+        )
+
+    def transform_documents(self, documents: List[Document], **kwargs: Any) -> List[Document]:
+        """
+        Apply the optimized AutoChunks plan to a list of LangChain documents.
+        This processes ALL documents provided.
+        """
+        try:
+            from langchain_core.documents import Document
+        except ImportError:
+            raise ImportError("Please install langchain-core: pip install langchain-core")
+
+        # Convert LangChain docs to AutoChunks format
+        ac_docs = []
+        for d in documents:
+            # We use metadata.get('source', id(d)) as a unique doc_id
+            doc_id = str(d.metadata.get("source", id(d)))
+            ac_docs.append({
+                "id": doc_id,
+                "text": d.page_content,
+                "metadata": d.metadata
+            })
+
+        # Run the execution pipeline
+        gen_name = self.plan.generator_pipeline.get("name")
+        params = self.plan.generator_pipeline.get("params", {})
+        
+        ac_chunks = self.chunker.apply_with_generator(ac_docs, gen_name, params)
+
+        # Convert back to LangChain docs
+        lc_docs = []
+        for ch in ac_chunks:
+            # Preserve original metadata and add chunking metadata
+            meta = ch.get("meta", {}).copy()
+            # If original metadata was passed through, it might be nested or direct
+            # For now, we assume simple merger
+            lc_docs.append(Document(
+                page_content=ch["text"],
+                metadata={**meta, "autochunk_plan_id": self.plan.id}
+            ))
+            
+        return lc_docs
+
+    def split_documents(self, documents: List[Document]) -> List[Document]:
+        """Alias for transform_documents to match TextSplitter interface."""
+        return self.transform_documents(documents)
+
+    def __call__(self, documents: List[Document]) -> List[Document]:
+        return self.transform_documents(documents)

autochunks-0.0.8/autochunk/adapters/llamaindex.py

@@ -0,0 +1,94 @@
+
+from __future__ import annotations
+from typing import List, Dict, Any, TYPE_CHECKING, Union
+from ..storage.plan import Plan
+from ..autochunker import AutoChunker
+
+if TYPE_CHECKING:
+    from llama_index.core.schema import BaseNode, Document
+
+try:
+    from llama_index.core.node_parser import NodeParser, BaseNodeParser
+    from llama_index.core.schema import TextNode, BaseNode, Document
+    LLAMA_INDEX_AVAILABLE = True
+except ImportError:
+    class BaseNodeParser: pass
+    LLAMA_INDEX_AVAILABLE = False
+
+class AutoChunkLlamaIndexAdapter(BaseNodeParser):
+    """
+    Official AutoChunks Adapter for LlamaIndex.
+    Acts as a native NodeParser for seamless integration into IngestionPipelines.
+    """
+    def __init__(self, plan: Union[Plan, str]):
+        if isinstance(plan, str):
+            self.plan = Plan.read(plan)
+        else:
+            self.plan = plan
+
+        self.chunker = AutoChunker(
+            embedding_provider=self.plan.embedding.get("name"),
+            embedding_model_or_path=self.plan.embedding.get("model")
+        )
+
+    def _parse_nodes(self, nodes: List[BaseNode], show_progress: bool = False, **kwargs: Any) -> List[BaseNode]:
+        """
+        Internal implementation for LlamaIndex BaseNodeParser.
+        """
+        # Convert Nodes to AutoChunks format
+        ac_docs = []
+        for n in nodes:
+            ac_docs.append({
+                "id": n.node_id,
+                "text": n.get_content(),
+                "metadata": n.metadata
+            })
+        
+        # Run the execution pipeline
+        gen_name = self.plan.generator_pipeline.get("name")
+        params = self.plan.generator_pipeline.get("params", {})
+        
+        ac_chunks = self.chunker.apply_with_generator(ac_docs, gen_name, params)
+
+        # Convert back to LlamaIndex Nodes
+        final_nodes = []
+        for ch in ac_chunks:
+            node = TextNode(
+                text=ch["text"],
+                metadata={**ch.get("meta", {}), "autochunk_plan_id": self.plan.id}
+            )
+            final_nodes.append(node)
+            
+        return final_nodes
+
+    def get_nodes_from_documents(self, documents: List[Document], **kwargs: Any) -> List[BaseNode]:
+        try:
+            from llama_index.core.schema import TextNode
+        except ImportError:
+            raise ImportError("Please install llama-index-core: pip install llama-index-core")
+
+        # Convert LlamaIndex docs to AutoChunks format
+        ac_docs = []
+        for d in documents:
+            ac_docs.append({
+                "id": d.doc_id,
+                "text": d.get_content(),
+                "metadata": d.metadata
+            })
+
+        # Run the execution pipeline
+        gen_name = self.plan.generator_pipeline.get("name")
+        params = self.plan.generator_pipeline.get("params", {})
+        
+        ac_chunks = self.chunker.apply_with_generator(ac_docs, gen_name, params)
+
+        # Convert back to LlamaIndex Nodes
+        nodes = []
+        for ch in ac_chunks:
+            node = TextNode(
+                text=ch["text"],
+                metadata={**ch.get("meta", {}), "autochunk_plan_id": self.plan.id}
+            )
+            nodes.append(node)
+            
+        return nodes