agent-context-packager 0.1.0__tar.gz

agent_context_packager-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: agent-context-packager
+Version: 0.1.0
+Summary: A document-to-agent-context compiler.
+Author-email: Your Name <your.email@example.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pymupdf>=1.23.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pyinstaller>=6.0.0
+Requires-Dist: tabulate>=0.9.0
+Requires-Dist: fastembed>=0.2.7
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: datasets>=2.0.0
+Requires-Dist: requests>=2.0.0
+Requires-Dist: pathspec>=0.11.0
+Requires-Dist: detect-secrets>=1.4.0
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-mock>=3.10.0; extra == "test"
+
+# AgentPack
+
+**AgentPack** improves the context pipeline for document-grounded agents.
+
+Instead of forcing AI agents to parse messy, disparate file formats (PDFs, CSVs, Markdown, text) at runtime, AgentPack is an offline **document-to-agent-context compiler**. It takes unstructured knowledge bases, turns them into clean semantic chunks with citations, retrieves the right evidence, and sends only high-signal context to the model.
+
+## The Benchmark
+**Given the same LLM, AgentPack provides better context than raw document stuffing or naive RAG.**
+
+I benchmarked AgentPack against standard RAG baselines on 42 complex financial queries from [Patronus AI FinanceBench](https://github.com/patronus-ai/financebench). The results prove that AgentPack reduces context bloat, improves evidence retrieval, preserves citations, and helps the exact same LLM produce more grounded answers.
+
+**Benchmark Highlights:**
+* **161x Reduction in Token Cost:** Cut context token usage from 424k to 2.6k, saving ~$0.10 per query.
+* **2x Context Relevance:** Vastly outperformed naive chunking in retrieving semantically complete financial tables.
+* **"Lost in the Middle" Prevention:** Outperformed raw document stuffing in correctness by preventing the LLM from drowning in noise.
+
+Read the full scientific methodology and results in [BENCHMARK.md](https://github.com/Vedant1202/agentpack/blob/main/BENCHMARK.md).
+
+## Installation
+
+You can install AgentPack via pip or npm.
+
+**Option 1: Using pip (Python)**
+```bash
+pip install agent-context-packager
+```
+
+**Option 2: Using npm (Node.js/CLI binary)**
+```bash
+npm install -g agent-context-packager
+```
+
+**Option 3: From Source**
+```bash
+git clone https://github.com/Vedant1202/agentpack.git
+cd agentpack
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+## Quick Start
+
+### 1. Scan for Secrets (Recommended)
+Before compiling a pack, ensure you aren't accidentally leaking API keys or secrets into the LLM context window. AgentPack automatically installs Yelp's `detect-secrets`.
+```bash
+detect-secrets scan > .secrets.baseline
+```
+
+### 2. Compile a Pack
+Point AgentPack at any folder containing your documents (`.txt`, `.md`, `.csv`, `.pdf`).
+
+```bash
+agentpack pack ./my_docs --out ./agentpack-output
+```
+
+**Key Compilation Options:**
+- `--include "*.md,*.txt"`: Only pack specific files or extensions.
+- `--ignore "tests/,drafts/"`: Exclude specific directories or files.
+- `--remove-empty-lines`: Compress text files to save LLM tokens.
+- `--no-gitignore`: Ignore `.gitignore` rules and pack everything.
+
+### 2. Retrieve
+AgentPack comes with a built-in hybrid search engine (SQLite FTS5 + FastEmbed vector search) to test your chunks instantly.
+
+```bash
+agentpack retrieve ./agentpack-output "eligibility criteria" --top-k 5
+```
+
+### 3. V1 Deterministic Eval
+Benchmark AgentPack against naive chunking using our offline evaluation harness.
+
+```bash
+agentpack eval ./benchmarks/my_dataset
+```
+
+## Comprehensive CLI Documentation
+
+AgentPack provides a rich CLI for auditing, validating, and testing your context packs (including Generative QA evaluations). 
+
+**[📖 Read the full CLI Reference](https://github.com/Vedant1202/agentpack/blob/main/docs/cli-reference.md)**
+
+## Supported Parsers
+- **TXT**: Paragraph-aware splitting.
+- **Markdown**: Semantic heading-aware section path tracking.
+- **CSV**: Uses Pandas & Tabulate to convert tabular data into Markdown tables.
+- **PDF**: Accurate page-by-page PyMuPDF extraction.
+
+## Architecture Overview
+
+```mermaid
+flowchart LR
+    Docs[Raw Docs] --> Parsers[Parsers]
+    Parsers --> Chunker[Chunker]
+    Chunker --> Pack[Context Pack]
+    Pack --> Agent[LLM Agent]
+```
+
+For a deep dive into how AgentPack parses, chunks, and indexes data, see [Architecture & Internals](https://github.com/Vedant1202/agentpack/blob/main/docs/architecture.md).
+
+## Current Limitations & Roadmap
+AgentPack is currently focused on text-based semantic extraction. The following features are on the roadmap but **not yet implemented**:
+- **Image Understanding / Vision**: AgentPack does not currently run OCR or vision models on images embedded within PDFs or Markdown files. Images are currently ignored during the parsing phase.
+- **Complex Table Structures**: While basic CSVs are supported, highly nested or merged-cell tables within PDFs are not perfectly reconstructed yet.
+- **Web Crawling**: You currently need to provide local files. Direct URL scraping is planned.
+- **Cloud Vector DB Integration**: Retrieval currently runs locally using SQLite FTS5 and FastEmbed. Connectors for Pinecone, Weaviate, or Qdrant are planned.
+
+---
+*Built with ❤️ for Agents.*

agent_context_packager-0.1.0/README.md

@@ -0,0 +1,109 @@
+# AgentPack
+
+**AgentPack** improves the context pipeline for document-grounded agents.
+
+Instead of forcing AI agents to parse messy, disparate file formats (PDFs, CSVs, Markdown, text) at runtime, AgentPack is an offline **document-to-agent-context compiler**. It takes unstructured knowledge bases, turns them into clean semantic chunks with citations, retrieves the right evidence, and sends only high-signal context to the model.
+
+## The Benchmark
+**Given the same LLM, AgentPack provides better context than raw document stuffing or naive RAG.**
+
+I benchmarked AgentPack against standard RAG baselines on 42 complex financial queries from [Patronus AI FinanceBench](https://github.com/patronus-ai/financebench). The results prove that AgentPack reduces context bloat, improves evidence retrieval, preserves citations, and helps the exact same LLM produce more grounded answers.
+
+**Benchmark Highlights:**
+* **161x Reduction in Token Cost:** Cut context token usage from 424k to 2.6k, saving ~$0.10 per query.
+* **2x Context Relevance:** Vastly outperformed naive chunking in retrieving semantically complete financial tables.
+* **"Lost in the Middle" Prevention:** Outperformed raw document stuffing in correctness by preventing the LLM from drowning in noise.
+
+Read the full scientific methodology and results in [BENCHMARK.md](https://github.com/Vedant1202/agentpack/blob/main/BENCHMARK.md).
+
+## Installation
+
+You can install AgentPack via pip or npm.
+
+**Option 1: Using pip (Python)**
+```bash
+pip install agent-context-packager
+```
+
+**Option 2: Using npm (Node.js/CLI binary)**
+```bash
+npm install -g agent-context-packager
+```
+
+**Option 3: From Source**
+```bash
+git clone https://github.com/Vedant1202/agentpack.git
+cd agentpack
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+## Quick Start
+
+### 1. Scan for Secrets (Recommended)
+Before compiling a pack, ensure you aren't accidentally leaking API keys or secrets into the LLM context window. AgentPack automatically installs Yelp's `detect-secrets`.
+```bash
+detect-secrets scan > .secrets.baseline
+```
+
+### 2. Compile a Pack
+Point AgentPack at any folder containing your documents (`.txt`, `.md`, `.csv`, `.pdf`).
+
+```bash
+agentpack pack ./my_docs --out ./agentpack-output
+```
+
+**Key Compilation Options:**
+- `--include "*.md,*.txt"`: Only pack specific files or extensions.
+- `--ignore "tests/,drafts/"`: Exclude specific directories or files.
+- `--remove-empty-lines`: Compress text files to save LLM tokens.
+- `--no-gitignore`: Ignore `.gitignore` rules and pack everything.
+
+### 2. Retrieve
+AgentPack comes with a built-in hybrid search engine (SQLite FTS5 + FastEmbed vector search) to test your chunks instantly.
+
+```bash
+agentpack retrieve ./agentpack-output "eligibility criteria" --top-k 5
+```
+
+### 3. V1 Deterministic Eval
+Benchmark AgentPack against naive chunking using our offline evaluation harness.
+
+```bash
+agentpack eval ./benchmarks/my_dataset
+```
+
+## Comprehensive CLI Documentation
+
+AgentPack provides a rich CLI for auditing, validating, and testing your context packs (including Generative QA evaluations). 
+
+**[📖 Read the full CLI Reference](https://github.com/Vedant1202/agentpack/blob/main/docs/cli-reference.md)**
+
+## Supported Parsers
+- **TXT**: Paragraph-aware splitting.
+- **Markdown**: Semantic heading-aware section path tracking.
+- **CSV**: Uses Pandas & Tabulate to convert tabular data into Markdown tables.
+- **PDF**: Accurate page-by-page PyMuPDF extraction.
+
+## Architecture Overview
+
+```mermaid
+flowchart LR
+    Docs[Raw Docs] --> Parsers[Parsers]
+    Parsers --> Chunker[Chunker]
+    Chunker --> Pack[Context Pack]
+    Pack --> Agent[LLM Agent]
+```
+
+For a deep dive into how AgentPack parses, chunks, and indexes data, see [Architecture & Internals](https://github.com/Vedant1202/agentpack/blob/main/docs/architecture.md).
+
+## Current Limitations & Roadmap
+AgentPack is currently focused on text-based semantic extraction. The following features are on the roadmap but **not yet implemented**:
+- **Image Understanding / Vision**: AgentPack does not currently run OCR or vision models on images embedded within PDFs or Markdown files. Images are currently ignored during the parsing phase.
+- **Complex Table Structures**: While basic CSVs are supported, highly nested or merged-cell tables within PDFs are not perfectly reconstructed yet.
+- **Web Crawling**: You currently need to provide local files. Direct URL scraping is planned.
+- **Cloud Vector DB Integration**: Retrieval currently runs locally using SQLite FTS5 and FastEmbed. Connectors for Pinecone, Weaviate, or Qdrant are planned.
+
+---
+*Built with ❤️ for Agents.*

agent_context_packager-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "agent-context-packager"
+version = "0.1.0"
+description = "A document-to-agent-context compiler."
+authors = [
+    { name = "Your Name", email = "your.email@example.com" }
+]
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = [
+    "typer>=0.9.0",
+    "pydantic>=2.0.0",
+    "pymupdf>=1.23.0",
+    "pandas>=2.0.0",
+    "tiktoken>=0.5.0",
+    "pyyaml>=6.0",
+    "pyinstaller>=6.0.0",
+    "tabulate>=0.9.0",
+    "fastembed>=0.2.7",
+    "numpy>=1.24.0",
+    "datasets>=2.0.0",
+    "requests>=2.0.0",
+    "pathspec>=0.11.0",
+    "detect-secrets>=1.4.0"
+]
+
+[project.scripts]
+agentpack = "agentpack.cli:app"
+
+[project.optional-dependencies]
+docs = [
+    "mkdocs-material>=9.0.0",
+    "mkdocstrings[python]>=0.24.0"
+]
+test = [
+    "pytest>=7.0.0",
+    "pytest-mock>=3.10.0"
+]

agent_context_packager-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: agent-context-packager
+Version: 0.1.0
+Summary: A document-to-agent-context compiler.
+Author-email: Your Name <your.email@example.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pymupdf>=1.23.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pyinstaller>=6.0.0
+Requires-Dist: tabulate>=0.9.0
+Requires-Dist: fastembed>=0.2.7
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: datasets>=2.0.0
+Requires-Dist: requests>=2.0.0
+Requires-Dist: pathspec>=0.11.0
+Requires-Dist: detect-secrets>=1.4.0
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-mock>=3.10.0; extra == "test"
+
+# AgentPack
+
+**AgentPack** improves the context pipeline for document-grounded agents.
+
+Instead of forcing AI agents to parse messy, disparate file formats (PDFs, CSVs, Markdown, text) at runtime, AgentPack is an offline **document-to-agent-context compiler**. It takes unstructured knowledge bases, turns them into clean semantic chunks with citations, retrieves the right evidence, and sends only high-signal context to the model.
+
+## The Benchmark
+**Given the same LLM, AgentPack provides better context than raw document stuffing or naive RAG.**
+
+I benchmarked AgentPack against standard RAG baselines on 42 complex financial queries from [Patronus AI FinanceBench](https://github.com/patronus-ai/financebench). The results prove that AgentPack reduces context bloat, improves evidence retrieval, preserves citations, and helps the exact same LLM produce more grounded answers.
+
+**Benchmark Highlights:**
+* **161x Reduction in Token Cost:** Cut context token usage from 424k to 2.6k, saving ~$0.10 per query.
+* **2x Context Relevance:** Vastly outperformed naive chunking in retrieving semantically complete financial tables.
+* **"Lost in the Middle" Prevention:** Outperformed raw document stuffing in correctness by preventing the LLM from drowning in noise.
+
+Read the full scientific methodology and results in [BENCHMARK.md](https://github.com/Vedant1202/agentpack/blob/main/BENCHMARK.md).
+
+## Installation
+
+You can install AgentPack via pip or npm.
+
+**Option 1: Using pip (Python)**
+```bash
+pip install agent-context-packager
+```
+
+**Option 2: Using npm (Node.js/CLI binary)**
+```bash
+npm install -g agent-context-packager
+```
+
+**Option 3: From Source**
+```bash
+git clone https://github.com/Vedant1202/agentpack.git
+cd agentpack
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+## Quick Start
+
+### 1. Scan for Secrets (Recommended)
+Before compiling a pack, ensure you aren't accidentally leaking API keys or secrets into the LLM context window. AgentPack automatically installs Yelp's `detect-secrets`.
+```bash
+detect-secrets scan > .secrets.baseline
+```
+
+### 2. Compile a Pack
+Point AgentPack at any folder containing your documents (`.txt`, `.md`, `.csv`, `.pdf`).
+
+```bash
+agentpack pack ./my_docs --out ./agentpack-output
+```
+
+**Key Compilation Options:**
+- `--include "*.md,*.txt"`: Only pack specific files or extensions.
+- `--ignore "tests/,drafts/"`: Exclude specific directories or files.
+- `--remove-empty-lines`: Compress text files to save LLM tokens.
+- `--no-gitignore`: Ignore `.gitignore` rules and pack everything.
+
+### 2. Retrieve
+AgentPack comes with a built-in hybrid search engine (SQLite FTS5 + FastEmbed vector search) to test your chunks instantly.
+
+```bash
+agentpack retrieve ./agentpack-output "eligibility criteria" --top-k 5
+```
+
+### 3. V1 Deterministic Eval
+Benchmark AgentPack against naive chunking using our offline evaluation harness.
+
+```bash
+agentpack eval ./benchmarks/my_dataset
+```
+
+## Comprehensive CLI Documentation
+
+AgentPack provides a rich CLI for auditing, validating, and testing your context packs (including Generative QA evaluations). 
+
+**[📖 Read the full CLI Reference](https://github.com/Vedant1202/agentpack/blob/main/docs/cli-reference.md)**
+
+## Supported Parsers
+- **TXT**: Paragraph-aware splitting.
+- **Markdown**: Semantic heading-aware section path tracking.
+- **CSV**: Uses Pandas & Tabulate to convert tabular data into Markdown tables.
+- **PDF**: Accurate page-by-page PyMuPDF extraction.
+
+## Architecture Overview
+
+```mermaid
+flowchart LR
+    Docs[Raw Docs] --> Parsers[Parsers]
+    Parsers --> Chunker[Chunker]
+    Chunker --> Pack[Context Pack]
+    Pack --> Agent[LLM Agent]
+```
+
+For a deep dive into how AgentPack parses, chunks, and indexes data, see [Architecture & Internals](https://github.com/Vedant1202/agentpack/blob/main/docs/architecture.md).
+
+## Current Limitations & Roadmap
+AgentPack is currently focused on text-based semantic extraction. The following features are on the roadmap but **not yet implemented**:
+- **Image Understanding / Vision**: AgentPack does not currently run OCR or vision models on images embedded within PDFs or Markdown files. Images are currently ignored during the parsing phase.
+- **Complex Table Structures**: While basic CSVs are supported, highly nested or merged-cell tables within PDFs are not perfectly reconstructed yet.
+- **Web Crawling**: You currently need to provide local files. Direct URL scraping is planned.
+- **Cloud Vector DB Integration**: Retrieval currently runs locally using SQLite FTS5 and FastEmbed. Connectors for Pinecone, Weaviate, or Qdrant are planned.
+
+---
+*Built with ❤️ for Agents.*

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/SOURCES.txt

@@ -0,0 +1,36 @@
+README.md
+pyproject.toml
+src/agent_context_packager.egg-info/PKG-INFO
+src/agent_context_packager.egg-info/SOURCES.txt
+src/agent_context_packager.egg-info/dependency_links.txt
+src/agent_context_packager.egg-info/entry_points.txt
+src/agent_context_packager.egg-info/requires.txt
+src/agent_context_packager.egg-info/top_level.txt
+src/agentpack/__init__.py
+src/agentpack/__main__.py
+src/agentpack/audit.py
+src/agentpack/chunker.py
+src/agentpack/cli.py
+src/agentpack/models.py
+src/agentpack/pack.py
+src/agentpack/retrieve.py
+src/agentpack/scanner.py
+src/agentpack/validate.py
+src/agentpack/eval/__init__.py
+src/agentpack/eval/baselines.py
+src/agentpack/eval/benchmarks.py
+src/agentpack/eval/generation.py
+src/agentpack/eval/metrics.py
+src/agentpack/eval/runner.py
+src/agentpack/parsers/__init__.py
+src/agentpack/parsers/base.py
+src/agentpack/parsers/csv_parser.py
+src/agentpack/parsers/markdown_parser.py
+src/agentpack/parsers/pdf_parser.py
+src/agentpack/parsers/text_parser.py
+tests/test_chunker.py
+tests/test_cli.py
+tests/test_pack.py
+tests/test_parsers.py
+tests/test_retrieve.py
+tests/test_scanner.py

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+agentpack = agentpack.cli:app

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/requires.txt

@@ -0,0 +1,22 @@
+typer>=0.9.0
+pydantic>=2.0.0
+pymupdf>=1.23.0
+pandas>=2.0.0
+tiktoken>=0.5.0
+pyyaml>=6.0
+pyinstaller>=6.0.0
+tabulate>=0.9.0
+fastembed>=0.2.7
+numpy>=1.24.0
+datasets>=2.0.0
+requests>=2.0.0
+pathspec>=0.11.0
+detect-secrets>=1.4.0
+
+[docs]
+mkdocs-material>=9.0.0
+mkdocstrings[python]>=0.24.0
+
+[test]
+pytest>=7.0.0
+pytest-mock>=3.10.0

agent_context_packager-0.1.0/src/agent_context_packager.egg-info/top_level.txt

@@ -0,0 +1 @@
+agentpack

agent_context_packager-0.1.0/src/agentpack/__init__.py

@@ -0,0 +1 @@
+"""AgentPack compiler package."""

agent_context_packager-0.1.0/src/agentpack/__main__.py

@@ -0,0 +1,4 @@
+from agentpack.cli import app
+
+if __name__ == "__main__":
+    app()

agent_context_packager-0.1.0/src/agentpack/audit.py

@@ -0,0 +1,65 @@
+import yaml
+from pathlib import Path
+
+def audit_pack(pack_dir: str) -> str:
+    """Generates an audit report for an agentpack output directory."""
+    base_path = Path(pack_dir)
+    manifest_path = base_path / "manifest.yml"
+    
+    if not manifest_path.exists():
+        return f"Error: Manifest not found at {manifest_path}"
+        
+    try:
+        with open(manifest_path, "r", encoding="utf-8") as f:
+            manifest = yaml.safe_load(f)
+    except Exception as e:
+        return f"Error: Failed to parse manifest YAML: {e}"
+        
+    sources = manifest.get("sources", [])
+    chunks = manifest.get("chunks", [])
+    tables = manifest.get("tables", [])
+    
+    files_processed = len(sources)
+    total_chunks = len(chunks)
+    total_tables = len(tables)
+    total_tokens = sum(chunk.get("token_count", 0) for chunk in chunks)
+    
+    max_chunk_size = 0
+    largest_chunk_id = None
+    for chunk in chunks:
+        if chunk.get("token_count", 0) > max_chunk_size:
+            max_chunk_size = chunk.get("token_count", 0)
+            largest_chunk_id = chunk.get("id")
+            
+    warnings = []
+    for source in sources:
+        for warning in source.get("warnings", []):
+            warnings.append(f"Source {source.get('id')}: [{warning.get('type')}] {warning.get('message')}")
+            
+    # Format report
+    report = [
+        f"# AgentPack Audit Report for '{manifest.get('pack', {}).get('name', 'Unknown')}'",
+        f"Generated at: {manifest.get('pack', {}).get('generated_at', 'Unknown')}\n",
+        "## Statistics",
+        f"- **Files Processed:** {files_processed}",
+        f"- **Total Chunks:** {total_chunks}",
+        f"- **Total Tables:** {total_tables}",
+        f"- **Total Tokens:** {total_tokens}",
+        f"- **Largest Chunk:** {max_chunk_size} tokens (ID: {largest_chunk_id})\n",
+        "## Extraction Warnings",
+    ]
+    
+    if warnings:
+        for warning in warnings:
+            report.append(f"- {warning}")
+    else:
+        report.append("- No extraction warnings.")
+        
+    # Write report
+    report_text = "\n".join(report)
+    report_path = base_path / "reports" / "validation_report.md"
+    report_path.parent.mkdir(exist_ok=True)
+    with open(report_path, "w", encoding="utf-8") as f:
+        f.write(report_text)
+        
+    return report_text

agent_context_packager-0.1.0/src/agentpack/chunker.py

@@ -0,0 +1,80 @@
+import tiktoken
+from typing import List
+from agentpack.models import SourceDocument
+
+class Chunk:
+    def __init__(self, chunk_id: str, source_id: str, path: str, token_count: int, content: str, metadata: dict):
+        self.chunk_id = chunk_id
+        self.source_id = source_id
+        self.path = path
+        self.token_count = token_count
+        self.content = content
+        self.metadata = metadata
+
+def chunk_document(doc: SourceDocument, max_tokens: int = 800, overlap_percent: float = 0.15) -> List[Chunk]:
+    encoder = tiktoken.get_encoding("cl100k_base")
+    chunks = []
+    
+    current_blocks = []
+    current_tokens = 0
+    current_metadata = {"source_path": doc.path}
+    chunk_index = 0
+    
+    overlap_tokens_target = int(max_tokens * overlap_percent)
+    
+    def create_chunk():
+        nonlocal current_blocks, current_tokens, chunk_index, current_metadata
+        if not current_blocks:
+            return
+        content_str = "\n\n".join([b["text"] for b in current_blocks])
+        chunk_id = f"{doc.source_id}_chunk_{chunk_index:03d}"
+        chunks.append(Chunk(
+            chunk_id=chunk_id,
+            source_id=doc.source_id,
+            path=f"chunks/{chunk_id}.md",
+            token_count=current_tokens,
+            content=content_str,
+            metadata=current_metadata.copy()
+        ))
+        chunk_index += 1
+        
+        # Keep blocks for overlap
+        overlap_blocks = []
+        overlap_toks = 0
+        for b in reversed(current_blocks):
+            if overlap_toks + b["tokens"] > overlap_tokens_target:
+                break
+            # don't overlap tables to avoid duplicating massive tables
+            if b["type"] == "table" and overlap_toks > 0:
+                break
+            overlap_blocks.insert(0, b)
+            overlap_toks += b["tokens"]
+            
+        current_blocks = overlap_blocks
+        current_tokens = overlap_toks
+
+    for block in doc.blocks:
+        if not block.text:
+            continue
+            
+        block_tokens = len(encoder.encode(block.text))
+        
+        # Update metadata
+        if block.section_path:
+            current_metadata["section"] = block.section_path[-1]
+        if block.page:
+            current_metadata["page"] = block.page
+            
+        if block.row_range:
+            current_metadata["row_range"] = list(block.row_range)
+            
+        # Very large blocks might exceed max_tokens, in a full implementation we'd split the block itself.
+        # For this MVP, we will just start a new chunk if adding this block pushes us over, unless we are empty.
+        if current_tokens + block_tokens > max_tokens and current_tokens > 0:
+            create_chunk()
+            
+        current_blocks.append({"text": block.text, "tokens": block_tokens, "type": block.type})
+        current_tokens += block_tokens
+        
+    create_chunk()
+    return chunks