adaptive-oci-chunking 0.1.0__tar.gz

adaptive_oci_chunking-0.1.0/.env.example

@@ -0,0 +1,8 @@
+OCI_CONFIG_FILE=~/.oci/config
+OCI_PROFILE=DEFAULT
+OCI_COMPARTMENT_ID=ocid1.compartment.oc1..example
+OCI_GENAI_ENDPOINT=https://inference.generativeai.us-chicago-1.oci.oraclecloud.com
+OCI_GENAI_EMBEDDING_MODEL=cohere.embed-english-v3.0
+OCI_OBJECT_STORAGE_NAMESPACE=my-namespace
+OCI_OBJECT_STORAGE_BUCKET=my-bucket
+

adaptive_oci_chunking-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: python -m pip install --upgrade pip
+      - run: pip install -e ".[dev]"
+      - run: ruff check .
+      - run: pytest

adaptive_oci_chunking-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+dist/
+build/
+*.egg-info/
+.env
+.DS_Store
+

adaptive_oci_chunking-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,105 @@
+# Contributing
+
+Thanks for helping improve Adaptive OCI Chunking. This project is intended to be useful for practitioners building RAG systems, researchers testing chunking strategies, and maintainers who want a clean place to compare document-splitting ideas.
+
+Maintainer: [Yash Shukla](https://www.linkedin.com/in/yashtechi/), focused on AI, cloud, and RAG systems.
+
+## Ways to Contribute
+
+- Add new chunkers for specific document structures or domains.
+- Improve intrinsic metrics or add new evaluation dimensions.
+- Add examples for LangChain, LlamaIndex, OCI, or other RAG workflows.
+- Improve tests, documentation, type hints, and packaging.
+- Report bugs with small reproducible examples.
+- Share benchmark results from real document collections.
+
+## Development Setup
+
+Clone the repo and install it in editable mode:
+
+```bash
+python -m venv .venv
+.venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+On macOS or Linux:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+If you only want optional integration support:
+
+```bash
+pip install -e ".[langchain,llama-index,oci,api]"
+```
+
+## Running Checks
+
+```bash
+ruff check .
+pytest
+python -m compileall src tests examples
+```
+
+If you add an example, make sure it either runs without optional credentials or clearly documents the required environment variables.
+
+## Adding a Chunker
+
+Chunkers live in `src/adaptive_chunking/chunkers.py` and implement `BaseChunker`.
+
+A good chunker should:
+
+- Preserve source order.
+- Return non-empty `Chunk` objects with stable `start_char` and `end_char` spans.
+- Avoid silently dropping text.
+- Fall back gracefully when its preferred structure is not present.
+- Include focused tests in `tests/test_adaptive_chunking.py`.
+- Be added to `default_chunkers()` only if it is broadly useful.
+
+## Adding a Metric
+
+Metrics live in `src/adaptive_chunking/metrics.py`.
+
+A good metric should:
+
+- Return a bounded score from `0.0` to `1.0`.
+- Be explainable from document and chunk structure alone.
+- Have a default weight in `MetricWeights`.
+- Include an explanation string in `IntrinsicMetricEvaluator.evaluate`.
+- Include tests for normal and edge cases.
+
+## Pull Request Checklist
+
+Before opening a PR:
+
+- Run `ruff check .`.
+- Run `pytest`.
+- Run `python -m compileall src tests examples`.
+- Update README or examples when behavior changes.
+- Add or update tests for code changes.
+- Keep changes focused on one concern where possible.
+
+## Design Principles
+
+- The core package should stay dependency-light.
+- Optional integrations should import their heavy dependencies only when used.
+- Chunking behavior should be inspectable and explainable.
+- Metrics should help users understand tradeoffs, not hide them behind a black box.
+- OCI support should remain optional.
+
+## Reporting Issues
+
+Please include:
+
+- Python version.
+- Installation command.
+- Minimal input text or document shape.
+- Expected chunking behavior.
+- Actual chunking behavior.
+- Any traceback or metric output.
+
+For private or sensitive documents, replace content with a synthetic example that preserves the relevant structure.

adaptive_oci_chunking-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yash Shukla
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

adaptive_oci_chunking-0.1.0/PKG-INFO

@@ -0,0 +1,343 @@
+Metadata-Version: 2.4
+Name: adaptive-oci-chunking
+Version: 0.1.0
+Summary: Adaptive document chunking for RAG with optional Oracle Cloud Infrastructure integrations.
+Project-URL: Repository, https://github.com/CaptnSalazar/adaptive-oci-chunking
+Project-URL: LinkedIn, https://www.linkedin.com/in/yashtechi/
+Author: Yash Shukla
+License-Expression: MIT
+License-File: LICENSE
+Keywords: adaptive-chunking,chunking,document-ai,genai,oci,rag
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.7
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Provides-Extra: api
+Requires-Dist: fastapi>=0.111; extra == 'api'
+Requires-Dist: uvicorn[standard]>=0.30; extra == 'api'
+Provides-Extra: dev
+Requires-Dist: fastapi>=0.111; extra == 'dev'
+Requires-Dist: langchain-text-splitters>=0.2; extra == 'dev'
+Requires-Dist: llama-index-core>=0.10; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: oci>=2.130; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: uvicorn[standard]>=0.30; extra == 'dev'
+Provides-Extra: integrations
+Requires-Dist: langchain-text-splitters>=0.2; extra == 'integrations'
+Requires-Dist: llama-index-core>=0.10; extra == 'integrations'
+Provides-Extra: langchain
+Requires-Dist: langchain-text-splitters>=0.2; extra == 'langchain'
+Provides-Extra: llama-index
+Requires-Dist: llama-index-core>=0.10; extra == 'llama-index'
+Provides-Extra: oci
+Requires-Dist: oci>=2.130; extra == 'oci'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# Adaptive OCI Chunking
+
+**Adaptive chunking toolkit for RAG with OCI, LangChain, and LlamaIndex support**
+
+[![CI](https://github.com/CaptnSalazar/adaptive-oci-chunking/actions/workflows/ci.yml/badge.svg)](https://github.com/CaptnSalazar/adaptive-oci-chunking/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
+[![arXiv](https://img.shields.io/badge/arXiv-2603.25333-b31b1b.svg)](https://arxiv.org/abs/2603.25333)
+
+</div>
+
+Adaptive OCI Chunking is an extensible Python implementation for document-aware chunk selection in Retrieval-Augmented Generation (RAG). It is inspired by Ekimetrics' `adaptive-chunking` repository and the paper _Adaptive Chunking: Optimizing Chunking-Method Selection for RAG_.
+
+The package evaluates several chunking strategies for each document, scores them with intrinsic metrics, and selects the best candidate before indexing or generation. Oracle Cloud Infrastructure (OCI) integrations are optional: the core chunking engine runs locally, while OCI Object Storage and Generative AI can be enabled when needed.
+
+## Architecture
+
+![Adaptive OCI Chunking architecture](Architecture.png)
+
+## What is Adaptive Chunking?
+
+No single chunking method works best for every document in a RAG pipeline. Adaptive chunking treats chunking as a selection problem: try multiple splitting strategies, score each result with intrinsic quality metrics, and choose the best candidate for the document at hand.
+
+This repo builds on that idea as a practical toolkit. It keeps the core dependency-light, adds extra production-oriented metrics, and includes optional adapters for OCI, LangChain, and LlamaIndex.
+
+## Features
+
+- Candidate chunkers:
+  - single-document
+  - fixed window with overlap
+  - recursive split
+  - split-then-merge
+  - section-aware
+  - delimiter-aware
+  - page-aware
+  - semantic lexical drift
+  - regex-guided section splitting
+- Metric-guided selection using paper-aligned intrinsic metrics:
+  - References Completeness (RC)
+  - Intrachunk Cohesion (ICC)
+  - Document Contextual Coherence (DCC)
+  - Block Integrity (BI)
+  - Size Compliance (SC)
+- Additional practical metrics:
+  - source coverage
+  - overlap control
+  - boundary quality
+  - semantic drift
+  - information density
+  - redundancy
+- Weighted strategy selection with explainable per-metric scores.
+- LangChain `TextSplitter` adapter.
+- LlamaIndex node conversion and parser-style adapter.
+- CLI for local text/Markdown files.
+- Optional OCI Object Storage loader and OCI Generative AI embedding adapter.
+- Small, dependency-light core for local document chunking workflows.
+
+## Contributing
+
+Contributions are welcome for new chunkers, metrics, examples, integrations, benchmarks, documentation, and bug fixes.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for setup instructions, PR expectations, and guidance for adding chunkers or metrics.
+
+Maintained by [Yash Shukla](https://www.linkedin.com/in/yashtechi/), focused on AI, cloud, and RAG systems.
+
+## Install
+
+```bash
+pip install -e ".[dev]"
+```
+
+With OCI support:
+
+```bash
+pip install -e ".[oci]"
+```
+
+With the API server:
+
+```bash
+pip install -e ".[api]"
+```
+
+With framework integrations:
+
+```bash
+pip install -e ".[langchain,llama-index]"
+```
+
+## Quick Start
+
+```bash
+adaptive-chunk chunk examples/sample.md --json
+```
+
+Python usage:
+
+```python
+from adaptive_chunking import AdaptiveChunker
+
+text = "## Introduction\nAdaptive chunking chooses a splitter per document.\n\n## Details\n..."
+chunker = AdaptiveChunker()
+result = chunker.chunk(text, document_id="demo")
+
+print(result.strategy_name)
+for chunk in result.chunks:
+    print(chunk.text)
+```
+
+## Examples
+
+Runnable examples live in `examples/`:
+
+- `basic_adaptive_chunking.py`: end-to-end adaptive selection with metric output.
+- `custom_selector.py`: custom chunker list and metric weights.
+- `langchain_integration.py`: LangChain `TextSplitter` usage.
+- `llama_index_integration.py`: LlamaIndex `TextNode` conversion.
+- `oci_object_storage.py`: loading source text from OCI Object Storage.
+
+## Chunker Options
+
+```python
+from adaptive_chunking.chunkers import (
+    DelimiterChunker,
+    PageChunker,
+    SectionAwareChunker,
+    SemanticChunker,
+)
+from adaptive_chunking.selector import AdaptiveSelector
+from adaptive_chunking import AdaptiveChunker
+
+selector = AdaptiveSelector(
+    chunkers=[
+        SectionAwareChunker(max_size=1800),
+        DelimiterChunker(delimiter="\n---\n"),
+        PageChunker(page_delimiter="\f"),
+        SemanticChunker(max_size=1400, similarity_threshold=0.08),
+    ]
+)
+
+result = AdaptiveChunker(selector=selector).chunk(text)
+```
+
+## Metrics
+
+The selector ranks every candidate by a weighted average of intrinsic scores. The first five metrics follow the paper's evaluation dimensions; the additional metrics make the implementation more practical for production RAG systems where dropped text, excessive overlap, and duplicated chunks are common failure modes.
+
+Weights can be tuned:
+
+```python
+from adaptive_chunking.metrics import IntrinsicMetricEvaluator, MetricConfig, MetricWeights
+from adaptive_chunking.selector import AdaptiveSelector
+
+weights = MetricWeights(
+    block_integrity=1.4,
+    coverage=1.5,
+    redundancy=0.8,
+)
+evaluator = IntrinsicMetricEvaluator(MetricConfig(weights=weights))
+selector = AdaptiveSelector(evaluator=evaluator)
+```
+
+## Adaptive Scoring
+
+For each document, the selector runs every candidate chunker and evaluates the chunks it produces. Each candidate receives a normalized weighted score:
+
+```text
+score(candidate) = sum(metric_value_i * metric_weight_i) / sum(metric_weight_i)
+```
+
+Where:
+
+- `metric_value_i` is the metric score for a candidate, normalized from `0.0` to `1.0`.
+- `metric_weight_i` controls how important that metric is for selection.
+- Higher scores are better.
+- Candidates are ranked from highest score to lowest score.
+
+For example, a domain that cares about preserving source text and section boundaries might emphasize `coverage` and `block_integrity`:
+
+| Metric | Value | Weight | Weighted value |
+|--------|------:|-------:|---------------:|
+| coverage | 1.00 | 1.50 | 1.50 |
+| block_integrity | 0.90 | 1.40 | 1.26 |
+| redundancy | 0.80 | 0.80 | 0.64 |
+
+```text
+score = (1.50 + 1.26 + 0.64) / (1.50 + 1.40 + 0.80)
+      = 3.40 / 3.70
+      = 0.919
+```
+
+You can inspect every candidate, not just the winner:
+
+```python
+from adaptive_chunking import AdaptiveChunker
+
+result = AdaptiveChunker().chunk(text, document_id="demo")
+
+for candidate in result.candidates:
+    print(candidate.strategy_name, round(candidate.score, 3), len(candidate.chunks))
+    for metric in candidate.metrics:
+        print(" ", metric.name, metric.value, "weight=", metric.weight)
+```
+
+This makes the selection process explainable: if a chunker loses, you can see whether it dropped content, produced excessive overlap, cut through structure, or failed a size constraint.
+
+## LangChain
+
+```python
+from adaptive_chunking.langchain import LangChainAdaptiveTextSplitter
+
+splitter = LangChainAdaptiveTextSplitter()
+documents = splitter.create_documents([text])
+```
+
+## LlamaIndex
+
+```python
+from adaptive_chunking import AdaptiveChunker
+from adaptive_chunking.llama_index import result_to_llama_nodes
+
+result = AdaptiveChunker().chunk(text, document_id="policy")
+nodes = result_to_llama_nodes(result)
+```
+
+## OCI Usage
+
+Copy `.env.example` and set the values for your tenancy and compartment. The core library does not require OCI credentials unless you instantiate an OCI adapter.
+
+```python
+from adaptive_chunking.oci import OCIObjectStorageTextLoader
+
+loader = OCIObjectStorageTextLoader(
+    namespace="my-namespace",
+    bucket_name="documents",
+)
+text = loader.load_text("policies/example.md")
+```
+
+## API Server
+
+```bash
+uvicorn adaptive_chunking.api:app --reload
+```
+
+Then post:
+
+```bash
+curl -X POST http://127.0.0.1:8000/chunk \
+  -H "Content-Type: application/json" \
+  -d "{\"text\":\"# Title\nBody text\", \"document_id\":\"demo\"}"
+```
+
+## Project Layout
+
+```text
+src/adaptive_chunking/
+  chunkers.py      # candidate splitting strategies
+  metrics.py       # intrinsic metric implementations
+  selector.py      # weighted adaptive strategy selection
+  pipeline.py      # high-level AdaptiveChunker
+  langchain.py     # optional LangChain TextSplitter adapter
+  llama_index.py   # optional LlamaIndex node helpers
+  oci.py           # optional OCI adapters
+  api.py           # optional FastAPI app
+  cli.py           # command line interface
+tests/
+examples/
+```
+
+## Notes
+
+This repo is designed as a clean, extensible foundation rather than a verbatim copy of the reference implementation. The metric implementations are practical approximations intended for engineering use and experimentation. Production RAG deployments should calibrate weights, chunk sizes, and embedding models against their document domains.
+
+## References
+
+- Ekimetrics reference implementation: [ekimetrics/adaptive-chunking](https://github.com/ekimetrics/adaptive-chunking)
+- Paper: [Adaptive Chunking: Optimizing Chunking-Method Selection for RAG](https://arxiv.org/abs/2603.25333)
+
+## Citation
+
+If this project helps your work, please cite the original adaptive chunking paper:
+
+```bibtex
+@inproceedings{demoura2026adaptive,
+    title={Adaptive Chunking: Optimizing Chunking-Method Selection for RAG},
+    author={de Moura Junior, Paulo Roberto and Lelong, Jean and Blangero, Annabelle},
+    booktitle={Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026)},
+    year={2026},
+    url={https://arxiv.org/abs/2603.25333},
+}
+```
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).