mdkeychunker 0.3.0__tar.gz

mdkeychunker-0.3.0/.env.sample

@@ -0,0 +1,25 @@
+# === MDKeyChunker v2 Configuration ===
+
+# LLM Provider: openai | anthropic | openai_compatible
+LLM_PROVIDER=openai
+
+# API key (not needed for local Ollama)
+LLM_API_KEY=your_api_key_here
+
+# Base URL — set for Ollama/vLLM/LM Studio (leave empty for OpenAI/Anthropic)
+# LLM_BASE_URL=http://localhost:11434/v1
+
+# Model name
+LLM_MODEL=gpt-4o-mini
+
+# Chunking
+MIN_CHUNK_SIZE=100       # Minimum characters per chunk
+MAX_CHUNK_SIZE=1500      # Soft max characters per chunk
+
+# Key-based restructuring
+MERGE_BY_KEYS=true       # Merge chunks sharing the same key
+MAX_MERGED_SIZE=3000     # Max combined size after merging
+MIN_ORPHAN_SIZE=200      # Below this, orphan chunks get context enrichment
+
+# Logging
+LOG_LEVEL=INFO

mdkeychunker-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,238 @@
+# Contributing to MDKeyChunker
+
+Thank you for considering contributing to MDKeyChunker! This document provides guidelines and instructions for contributing.
+
+## Development Setup
+
+1. **Fork and clone the repository**
+
+```bash
+git clone https://github.com/bhavik-mangla/MDKeyChunker.git
+cd MDKeyChunker
+```
+
+2. **Create a virtual environment**
+
+```bash
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+3. **Install in development mode**
+
+```bash
+pip install -e ".[dev]"
+```
+
+4. **Run tests to verify setup**
+
+```bash
+pytest
+```
+
+## Code Standards
+
+### Style Guidelines
+
+- **PEP 8**: Follow Python style guide
+- **Type hints**: All functions must have type annotations
+- **Docstrings**: Use Google-style docstrings
+- **Line length**: 100 characters (configured in pyproject.toml)
+
+### Tools
+
+We use automated tools for code quality:
+
+```bash
+# Format code
+ruff format mdkeychunker/
+
+# Lint
+ruff check mdkeychunker/
+
+# Type checking
+mypy mdkeychunker/
+```
+
+### Example Function
+
+```python
+def process_chunks(text: str, config: Config) -> list[Chunk]:
+    """
+    Process markdown text into enriched chunks.
+
+    Args:
+        text: Input markdown text to process
+        config: Configuration object
+
+    Returns:
+        List of enriched Chunk objects
+
+    Raises:
+        ValueError: If text is empty
+    """
+    if not text:
+        raise ValueError("Text cannot be empty")
+
+    pipeline = Pipeline(config)
+    return pipeline.process_text(text)
+```
+
+## Testing
+
+### Writing Tests
+
+- Place tests in `tests/` directory
+- Name test files `test_*.py`
+- Name test functions `test_*`
+- Use fixtures for common setup
+- Aim for >80% code coverage
+
+### Test Structure
+
+```python
+import pytest
+from mdkeychunker.module import Function
+
+@pytest.fixture
+def config():
+    """Create test configuration."""
+    return Config(param1=value1)
+
+def test_basic_functionality(config):
+    """Test basic behavior."""
+    result = Function(config)
+    assert result == expected
+
+def test_edge_case():
+    """Test edge case handling."""
+    with pytest.raises(ValueError):
+        Function(invalid_input)
+```
+
+### Running Tests
+
+```bash
+# All tests
+pytest
+
+# Specific file
+pytest tests/test_chunker.py
+
+# With coverage
+pytest --cov=mdkeychunker --cov-report=html
+
+# Verbose output
+pytest -v
+```
+
+## Pull Request Process
+
+1. **Create a feature branch**
+
+```bash
+git checkout -b feature/your-feature-name
+```
+
+2. **Make your changes**
+   - Write code following style guidelines
+   - Add tests for new functionality
+   - Update documentation as needed
+
+3. **Verify tests pass**
+
+```bash
+pytest
+mypy mdkeychunker/
+ruff format --check mdkeychunker/
+```
+
+4. **Commit with clear messages**
+
+```bash
+git commit -m "Add feature: description of what was added"
+```
+
+5. **Push and create PR**
+
+```bash
+git push origin feature/your-feature-name
+```
+
+Then create a pull request on GitHub with:
+
+- Clear title describing the change
+- Description of what was changed and why
+- Link to any related issues
+- Screenshots/examples if applicable
+
+## Areas for Contribution
+
+### High Priority
+
+- **Performance optimization**: Profile and optimize bottlenecks for large documents
+- **Additional LLM providers**: Add support for Cohere, Mistral, etc.
+- **Documentation**: Improve examples, tutorials, API docs
+- **Streaming support**: Process very large documents chunk-by-chunk
+
+### Medium Priority
+
+- **Adaptive chunking**: Dynamic chunk sizing based on content
+- **Streaming API**: Process large documents in chunks
+- **Export formats**: Neo4j graph export, custom formats
+- **Visualization**: Generate chunk relationship graphs
+
+### Nice to Have
+
+- **Web UI**: Simple interface for testing
+- **Benchmarking suite**: Standardized performance tests
+- **Integration examples**: Sample RAG pipeline implementations
+- **Docker support**: Containerized deployment
+
+## Documentation
+
+When adding new features:
+
+1. **Update README.md** if user-facing
+2. **Add docstrings** to all functions/classes
+3. **Update .env.sample** for new config options
+4. **Add examples** in `examples.py` if applicable
+
+## Code Review Process
+
+All submissions require review. We look for:
+
+- **Functionality**: Does it work as intended?
+- **Tests**: Are there adequate tests?
+- **Code quality**: Is it readable and maintainable?
+- **Documentation**: Is it well-documented?
+- **Performance**: Any performance implications?
+
+## Reporting Bugs
+
+Use GitHub Issues with:
+
+- **Clear title**: Concise description of the bug
+- **Environment**: OS, Python version, package version
+- **Reproduction steps**: Minimal code to reproduce
+- **Expected vs. actual behavior**
+- **Logs/errors**: Full error messages
+
+## Feature Requests
+
+Use GitHub Issues with:
+
+- **Use case**: Why is this feature needed?
+- **Proposed solution**: How should it work?
+- **Alternatives**: Other approaches considered
+- **Impact**: Who would benefit?
+
+## Questions?
+
+- Open a GitHub Discussion for general questions
+- Check existing issues/PRs for similar topics
+- Tag maintainers for urgent items
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the MIT License.

mdkeychunker-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bhavik Mangla
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mdkeychunker-0.3.0/MANIFEST.in

@@ -0,0 +1,14 @@
+# Manifest file for package distribution
+
+include README.md
+include LICENSE
+include CONTRIBUTING.md
+include .env.sample
+include demo.md
+include examples.py
+
+recursive-include mdkeychunker *.py
+recursive-include tests *.py
+
+exclude .gitignore
+exclude setup.sh

mdkeychunker-0.3.0/PKG-INFO

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: mdkeychunker
+Version: 0.3.0
+Summary: Markdown chunking with LLM enrichment for RAG pipelines
+Author: Bhavik Mangla
+License: MIT
+Project-URL: Homepage, https://github.com/bhavik-mangla/MDKeyChunker
+Project-URL: Repository, https://github.com/bhavik-mangla/MDKeyChunker
+Project-URL: Issues, https://github.com/bhavik-mangla/MDKeyChunker/issues
+Project-URL: Changelog, https://github.com/bhavik-mangla/MDKeyChunker/releases
+Keywords: markdown,chunking,rag,llm,retrieval-augmented-generation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: tiktoken>=0.5.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.18.0; extra == "anthropic"
+Provides-Extra: spacy
+Requires-Dist: spacy>=3.0.0; extra == "spacy"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: mypy>=1.5.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Provides-Extra: benchmark
+Requires-Dist: sentence-transformers>=2.2.0; extra == "benchmark"
+Requires-Dist: datasets>=2.14.0; extra == "benchmark"
+Requires-Dist: beir>=2.0.0; extra == "benchmark"
+Requires-Dist: torch>=2.0.0; extra == "benchmark"
+Requires-Dist: tqdm>=4.66.0; extra == "benchmark"
+Requires-Dist: matplotlib>=3.7.0; extra == "benchmark"
+Dynamic: license-file
+
+# MDKeyChunker
+
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Research Paper](https://img.shields.io/badge/Research-ArXiv-red.svg)](https://arxiv.org/abs/2603.23533)
+
+Markdown chunking with single-call LLM enrichment or ultra-fast spaCy processing for RAG pipelines. 
+Paper: **[MDKeyChunker: Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAG](https://arxiv.org/abs/2603.23533)** (arXiv:2603.23533)
+
+## What It Does
+
+1. **Chunks** Markdown into structural units (headers, code blocks, tables, lists)
+2. **Enriches** each chunk with ONE call → title, summary, keywords, entities, questions, semantic key
+    - **LLM Mode**: High-fidelity metadata using GPT/Claude/Ollama
+    - **spaCy Mode**: Fast, local, and free semantic key extraction
+3. **Passes rolling keys** forward so the enricher has context about prior topics
+4. **Restructures** by merging chunks that share the same specific-subtopic key
+
+## Quick Start
+
+```bash
+git clone https://github.com/bhavik-mangla/MDKeyChunker.git
+cd MDKeyChunker
+pip install -e .              # add [anthropic] or [spacy] extras as needed
+cp .env.sample .env           # then edit .env with your API key
+mdkeychunker demo.md
+```
+
+spaCy mode needs the extra and a model: `pip install -e ".[spacy]" && python -m spacy download en_core_web_md`.
+
+Or programmatically:
+
+```python
+from mdkeychunker import Pipeline, Config
+
+# High-fidelity mode (GPT-4o-mini)
+config = Config.from_env()
+pipeline = Pipeline(config, enricher_mode="llm")
+chunks = pipeline.process_file("document.md")
+
+# Ultra-fast / Free mode (spaCy)
+pipeline_fast = Pipeline(config, enricher_mode="spacy")
+chunks_fast = pipeline_fast.process_file("document.md")
+```
+
+## CLI
+
+```bash
+# Basic usage
+mdkeychunker document.md
+
+# Save to specific output
+mdkeychunker document.md -o chunks.jsonl
+
+# With summary file and stats
+mdkeychunker document.md --summary summary.txt --stats
+
+# Disable merging
+mdkeychunker document.md --no-merge
+
+# Override LLM provider (e.g., local Ollama)
+mdkeychunker document.md --provider openai_compatible --base-url http://localhost:11434/v1 --model llama3
+```
+
+## Configuration
+
+Set via `.env` file or environment variables:
+
+| Variable | Default | Description |
+|---|---|---|
+| `LLM_PROVIDER` | `openai` | `openai` \| `anthropic` \| `openai_compatible` |
+| `LLM_API_KEY` | — | API key (not needed for local Ollama) |
+| `LLM_BASE_URL` | — | Base URL for Ollama/vLLM/LM Studio |
+| `LLM_MODEL` | `gpt-4o-mini` | Model name |
+| `MIN_CHUNK_SIZE` | `100` | Minimum characters per chunk |
+| `MAX_CHUNK_SIZE` | `1500` | Soft max characters per chunk |
+| `MERGE_BY_KEYS` | `true` | Merge chunks sharing the same key |
+| `MAX_MERGED_SIZE` | `3000` | Max combined size after merging |
+| `MIN_ORPHAN_SIZE` | `200` | Below this, orphan chunks get context enrichment |
+| `LOG_LEVEL` | `INFO` | Logging verbosity |
+
+## LLM Providers
+
+```bash
+# OpenAI
+LLM_PROVIDER=openai
+LLM_API_KEY=sk-...
+LLM_MODEL=gpt-4o-mini
+
+# Anthropic
+LLM_PROVIDER=anthropic
+LLM_API_KEY=sk-ant-...
+LLM_MODEL=claude-3-haiku-20240307
+
+# Ollama (local)
+LLM_PROVIDER=openai_compatible
+LLM_BASE_URL=http://localhost:11434/v1
+LLM_MODEL=llama3
+```
+
+## How It Works
+
+```
+Markdown → Chunker → Enricher (1 LLM call/chunk) → Restructurer → Enriched Chunks
+                          ↑
+                    Rolling Keys
+                    (context from prior chunks)
+```
+
+**Key design**: The `key` field means the *specific subtopic* that distinguishes a chunk — e.g. `"admissions process"`, `"oauth token flow"`, `"gradient descent optimization"` — never the broad document topic. Chunks with matching keys are merged globally, regardless of position.
+
+## Chunk Schema
+
+```json
+{
+  "chunk_id": "a3f2b1c4d5e6f7a8",
+  "text": "The admissions process begins in March...",
+  "section_title": "Admissions",
+  "title": "Spring Admissions Timeline",
+  "summary": "Describes the March start of the admissions process...",
+  "keywords": ["admissions", "application deadline", "March intake"],
+  "entities": [{"name": "March", "type": "EVENT"}],
+  "questions": ["When does the admissions process begin?"],
+  "key": "admissions process",
+  "related_keys": ["curriculum framework"],
+  "content_types": ["paragraph"],
+  "position_index": 2,
+  "previous_chunk_id": "9b8c7d6e5f4a3b2c",
+  "next_chunk_id": "1a2b3c4d5e6f7a8b",
+  "token_count": 187,
+  "start_line": 12,
+  "end_line": 28
+}
+```
+
+## Benchmarks
+
+The paper's evaluation (30 queries over an 18-document Markdown corpus, Configs A–D) is described in [arXiv:2603.23533](https://arxiv.org/abs/2603.23533). The code version used in the paper is commit `e3e1b86`.
+
+`benchmarks/scifact.py` runs a SciFact sanity check in spaCy mode (`pip install -e ".[benchmark,spacy]"`, then `python benchmarks/scifact.py`). Embeddings are [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) with exact cosine search.
+
+![SciFact Benchmark Results](figures/scifact_benchmark.png)
+
+| Strategy | Library | Recall@5 | nDCG@5 | Chunks/Doc |
+| :--- | :--- | :--- | :--- | :--- |
+| MDKeyChunker (spaCy mode) | — | 0.762 | 0.681 | 1.00 |
+| [Recursive Character](https://github.com/langchain-ai/langchain) | LangChain | 0.760 | 0.678 | 3.43 |
+| [Semantic Chunker](https://github.com/langchain-ai/langchain-experimental) | LangChain | 0.775 | 0.681 | 2.03 |
+| Fixed Token (512) | Standard | 0.762 | 0.681 | 1.05 |
+
+**Read this table with care.** SciFact abstracts are short single-paragraph plain text, so MDKeyChunker emits one chunk per abstract and this run effectively measures whole-abstract retrieval. It does not exercise Markdown structure, LLM enrichment, or key-based restructuring, and it is not evidence for those stages. The baseline rows were produced with scripts that are not yet in this repository.
+
+## API
+
+```python
+from mdkeychunker import Pipeline, Config, Chunk
+
+# Config
+config = Config(llm_provider="openai", llm_model="gpt-4o-mini", merge_by_keys=True)
+
+# Pipeline
+pipeline = Pipeline(config)
+chunks: list[Chunk] = pipeline.process_text(markdown_text)
+chunks: list[Chunk] = pipeline.process_file("path/to/file.md")
+
+# Save output
+pipeline.save_jsonl(chunks, "output.jsonl")
+pipeline.save_summary(chunks, "summary.txt")
+```
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+## Citation
+
+If you use MDKeyChunker, please cite:
+
+```bibtex
+@misc{mangla2026mdkeychunker,
+  title         = {{MDKeyChunker}: Single-Call {LLM} Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy {RAG}},
+  author        = {Mangla, Bhavik},
+  year          = {2026},
+  eprint        = {2603.23533},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CL},
+  url           = {https://arxiv.org/abs/2603.23533}
+}
+```
+
+GitHub's "Cite this repository" button uses [`CITATION.cff`](CITATION.cff).
+
+## License
+
+MIT