agent-context-packager 0.1.0__py3-none-any.whl

agent_context_packager-0.1.0.dist-info/METADATA

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: agent-context-packager
+Version: 0.1.0
+Summary: A document-to-agent-context compiler.
+Author-email: Your Name <your.email@example.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pymupdf>=1.23.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: tiktoken>=0.5.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pyinstaller>=6.0.0
+Requires-Dist: tabulate>=0.9.0
+Requires-Dist: fastembed>=0.2.7
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: datasets>=2.0.0
+Requires-Dist: requests>=2.0.0
+Requires-Dist: pathspec>=0.11.0
+Requires-Dist: detect-secrets>=1.4.0
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-mock>=3.10.0; extra == "test"
+
+# AgentPack
+
+**AgentPack** improves the context pipeline for document-grounded agents.
+
+Instead of forcing AI agents to parse messy, disparate file formats (PDFs, CSVs, Markdown, text) at runtime, AgentPack is an offline **document-to-agent-context compiler**. It takes unstructured knowledge bases, turns them into clean semantic chunks with citations, retrieves the right evidence, and sends only high-signal context to the model.
+
+## The Benchmark
+**Given the same LLM, AgentPack provides better context than raw document stuffing or naive RAG.**
+
+I benchmarked AgentPack against standard RAG baselines on 42 complex financial queries from [Patronus AI FinanceBench](https://github.com/patronus-ai/financebench). The results prove that AgentPack reduces context bloat, improves evidence retrieval, preserves citations, and helps the exact same LLM produce more grounded answers.
+
+**Benchmark Highlights:**
+* **161x Reduction in Token Cost:** Cut context token usage from 424k to 2.6k, saving ~$0.10 per query.
+* **2x Context Relevance:** Vastly outperformed naive chunking in retrieving semantically complete financial tables.
+* **"Lost in the Middle" Prevention:** Outperformed raw document stuffing in correctness by preventing the LLM from drowning in noise.
+
+Read the full scientific methodology and results in [BENCHMARK.md](https://github.com/Vedant1202/agentpack/blob/main/BENCHMARK.md).
+
+## Installation
+
+You can install AgentPack via pip or npm.
+
+**Option 1: Using pip (Python)**
+```bash
+pip install agent-context-packager
+```
+
+**Option 2: Using npm (Node.js/CLI binary)**
+```bash
+npm install -g agent-context-packager
+```
+
+**Option 3: From Source**
+```bash
+git clone https://github.com/Vedant1202/agentpack.git
+cd agentpack
+python3 -m venv venv
+source venv/bin/activate
+pip install -e .
+```
+
+## Quick Start
+
+### 1. Scan for Secrets (Recommended)
+Before compiling a pack, ensure you aren't accidentally leaking API keys or secrets into the LLM context window. AgentPack automatically installs Yelp's `detect-secrets`.
+```bash
+detect-secrets scan > .secrets.baseline
+```
+
+### 2. Compile a Pack
+Point AgentPack at any folder containing your documents (`.txt`, `.md`, `.csv`, `.pdf`).
+
+```bash
+agentpack pack ./my_docs --out ./agentpack-output
+```
+
+**Key Compilation Options:**
+- `--include "*.md,*.txt"`: Only pack specific files or extensions.
+- `--ignore "tests/,drafts/"`: Exclude specific directories or files.
+- `--remove-empty-lines`: Compress text files to save LLM tokens.
+- `--no-gitignore`: Ignore `.gitignore` rules and pack everything.
+
+### 2. Retrieve
+AgentPack comes with a built-in hybrid search engine (SQLite FTS5 + FastEmbed vector search) to test your chunks instantly.
+
+```bash
+agentpack retrieve ./agentpack-output "eligibility criteria" --top-k 5
+```
+
+### 3. V1 Deterministic Eval
+Benchmark AgentPack against naive chunking using our offline evaluation harness.
+
+```bash
+agentpack eval ./benchmarks/my_dataset
+```
+
+## Comprehensive CLI Documentation
+
+AgentPack provides a rich CLI for auditing, validating, and testing your context packs (including Generative QA evaluations). 
+
+**[📖 Read the full CLI Reference](https://github.com/Vedant1202/agentpack/blob/main/docs/cli-reference.md)**
+
+## Supported Parsers
+- **TXT**: Paragraph-aware splitting.
+- **Markdown**: Semantic heading-aware section path tracking.
+- **CSV**: Uses Pandas & Tabulate to convert tabular data into Markdown tables.
+- **PDF**: Accurate page-by-page PyMuPDF extraction.
+
+## Architecture Overview
+
+```mermaid
+flowchart LR
+    Docs[Raw Docs] --> Parsers[Parsers]
+    Parsers --> Chunker[Chunker]
+    Chunker --> Pack[Context Pack]
+    Pack --> Agent[LLM Agent]
+```
+
+For a deep dive into how AgentPack parses, chunks, and indexes data, see [Architecture & Internals](https://github.com/Vedant1202/agentpack/blob/main/docs/architecture.md).
+
+## Current Limitations & Roadmap
+AgentPack is currently focused on text-based semantic extraction. The following features are on the roadmap but **not yet implemented**:
+- **Image Understanding / Vision**: AgentPack does not currently run OCR or vision models on images embedded within PDFs or Markdown files. Images are currently ignored during the parsing phase.
+- **Complex Table Structures**: While basic CSVs are supported, highly nested or merged-cell tables within PDFs are not perfectly reconstructed yet.
+- **Web Crawling**: You currently need to provide local files. Direct URL scraping is planned.
+- **Cloud Vector DB Integration**: Retrieval currently runs locally using SQLite FTS5 and FastEmbed. Connectors for Pinecone, Weaviate, or Qdrant are planned.
+
+---
+*Built with ❤️ for Agents.*

agent_context_packager-0.1.0.dist-info/RECORD

@@ -0,0 +1,27 @@
+agentpack/__init__.py,sha256=loQ6gSLeLIQi-sqFzcAff-FimPgv3F1bli-IvFLw-HQ,34
+agentpack/__main__.py,sha256=1GwAogovmCk7NUCapkbVWitA8WkrIFPmHOXK0cMhxKg,68
+agentpack/audit.py,sha256=ic5aZaF1seYyQJED3ubhFNn6Wf9YmUssbGc18ODIi3w,2318
+agentpack/chunker.py,sha256=ynqdp7b--RQxwg63LN9aSWV7rvv_6mesLY1lAASp6yM,2897
+agentpack/cli.py,sha256=kPBEPN_NQ47Mi5KFyylro7c-vySkuKwB022qTs0KVxk,5996
+agentpack/models.py,sha256=qg7o-77dKW995VXdyBkke9SwLDLL49vk2q6Xt6EVEXc,945
+agentpack/pack.py,sha256=Yu7Ti03cv_iwwtMq7vt1zGpkAKBFE9QG9sHgMzJOmX0,5700
+agentpack/retrieve.py,sha256=FYkOTJzG-jTgcifVCKfGm5Y6-yiAmBT2rjHXHbAc1wk,8941
+agentpack/scanner.py,sha256=hzuAJPzTP9OjfQMLkVL_3-5SPyXd9K47WETCcSG1ink,3857
+agentpack/validate.py,sha256=NB-zhHHNO8WNvV2E8bos7cLmjbQ0TDGyco-pdEiw0bc,2411
+agentpack/eval/__init__.py,sha256=F3Mg_nrhyGVdckambYD5gtGh5rUxz1J3ALJ7We3jd6c,52
+agentpack/eval/baselines.py,sha256=JVBkEBfBK0mzDIFkObtECeR39bSIwwaU_XSxspoM_vs,4768
+agentpack/eval/benchmarks.py,sha256=_OfBnQ5QJJMqMDRc_UcXZVE0QyGPEajA8uxzE5cJlBI,3158
+agentpack/eval/generation.py,sha256=v1bieidtafr943WfxXQxX70GLnNUTIIwTc6X806OgAs,6926
+agentpack/eval/metrics.py,sha256=d1YBt7Mv2AhZswCZ4NOUw36KkeUDIL--X3h4qsP9fHU,1762
+agentpack/eval/runner.py,sha256=7Hsnmw9NRJaFt7q9rbqztep4h37ZutlFIrRWMikJTk0,4366
+agentpack/parsers/__init__.py,sha256=MO8UQ8t-gK4SyIS8FXglAHxeDJ4MeCix_l_OOX8HjkM,44
+agentpack/parsers/base.py,sha256=xE71FFm_m5tgAjHp9HfQOQIsYSAf1vQXWqBmm4CEQZw,298
+agentpack/parsers/csv_parser.py,sha256=q-99N2VkSCaWqTLi_fqbj--0_6Hk29KvzfMOE_bWCvc,1917
+agentpack/parsers/markdown_parser.py,sha256=_KcMeMUTEuEUzUuTH-AG0WtXV2EcnGuHvg8PKbxE6yY,2984
+agentpack/parsers/pdf_parser.py,sha256=4XR1Q21G3Fnq9NQ_c23QNUuKlQifu4K6Y2FkDn16PEg,1989
+agentpack/parsers/text_parser.py,sha256=BOcOLYu_izB4Ry0FomXJ_zNihf8b8LKC6WBKgGw4AKg,1439
+agent_context_packager-0.1.0.dist-info/METADATA,sha256=to06R0gMTdb5--FuigRkYojjtC3c-TyhT4uUavnl4nI,5567
+agent_context_packager-0.1.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+agent_context_packager-0.1.0.dist-info/entry_points.txt,sha256=QEmFmah-r2yma_MKbIvKAYxe0A9pKQGnsruvL8_g9oc,48
+agent_context_packager-0.1.0.dist-info/top_level.txt,sha256=kKYX3EZl6xeaJl0FJUcQz3OKeqPjHkJCJCjUVcvXYTU,10
+agent_context_packager-0.1.0.dist-info/RECORD,,

agent_context_packager-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

agent_context_packager-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+agentpack = agentpack.cli:app

agent_context_packager-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+agentpack

agentpack/__init__.py

@@ -0,0 +1 @@
+"""AgentPack compiler package."""

agentpack/__main__.py

@@ -0,0 +1,4 @@
+from agentpack.cli import app
+
+if __name__ == "__main__":
+    app()

agentpack/audit.py

@@ -0,0 +1,65 @@
+import yaml
+from pathlib import Path
+
+def audit_pack(pack_dir: str) -> str:
+    """Generates an audit report for an agentpack output directory."""
+    base_path = Path(pack_dir)
+    manifest_path = base_path / "manifest.yml"
+    
+    if not manifest_path.exists():
+        return f"Error: Manifest not found at {manifest_path}"
+        
+    try:
+        with open(manifest_path, "r", encoding="utf-8") as f:
+            manifest = yaml.safe_load(f)
+    except Exception as e:
+        return f"Error: Failed to parse manifest YAML: {e}"
+        
+    sources = manifest.get("sources", [])
+    chunks = manifest.get("chunks", [])
+    tables = manifest.get("tables", [])
+    
+    files_processed = len(sources)
+    total_chunks = len(chunks)
+    total_tables = len(tables)
+    total_tokens = sum(chunk.get("token_count", 0) for chunk in chunks)
+    
+    max_chunk_size = 0
+    largest_chunk_id = None
+    for chunk in chunks:
+        if chunk.get("token_count", 0) > max_chunk_size:
+            max_chunk_size = chunk.get("token_count", 0)
+            largest_chunk_id = chunk.get("id")
+            
+    warnings = []
+    for source in sources:
+        for warning in source.get("warnings", []):
+            warnings.append(f"Source {source.get('id')}: [{warning.get('type')}] {warning.get('message')}")
+            
+    # Format report
+    report = [
+        f"# AgentPack Audit Report for '{manifest.get('pack', {}).get('name', 'Unknown')}'",
+        f"Generated at: {manifest.get('pack', {}).get('generated_at', 'Unknown')}\n",
+        "## Statistics",
+        f"- **Files Processed:** {files_processed}",
+        f"- **Total Chunks:** {total_chunks}",
+        f"- **Total Tables:** {total_tables}",
+        f"- **Total Tokens:** {total_tokens}",
+        f"- **Largest Chunk:** {max_chunk_size} tokens (ID: {largest_chunk_id})\n",
+        "## Extraction Warnings",
+    ]
+    
+    if warnings:
+        for warning in warnings:
+            report.append(f"- {warning}")
+    else:
+        report.append("- No extraction warnings.")
+        
+    # Write report
+    report_text = "\n".join(report)
+    report_path = base_path / "reports" / "validation_report.md"
+    report_path.parent.mkdir(exist_ok=True)
+    with open(report_path, "w", encoding="utf-8") as f:
+        f.write(report_text)
+        
+    return report_text

agentpack/chunker.py

@@ -0,0 +1,80 @@
+import tiktoken
+from typing import List
+from agentpack.models import SourceDocument
+
+class Chunk:
+    def __init__(self, chunk_id: str, source_id: str, path: str, token_count: int, content: str, metadata: dict):
+        self.chunk_id = chunk_id
+        self.source_id = source_id
+        self.path = path
+        self.token_count = token_count
+        self.content = content
+        self.metadata = metadata
+
+def chunk_document(doc: SourceDocument, max_tokens: int = 800, overlap_percent: float = 0.15) -> List[Chunk]:
+    encoder = tiktoken.get_encoding("cl100k_base")
+    chunks = []
+    
+    current_blocks = []
+    current_tokens = 0
+    current_metadata = {"source_path": doc.path}
+    chunk_index = 0
+    
+    overlap_tokens_target = int(max_tokens * overlap_percent)
+    
+    def create_chunk():
+        nonlocal current_blocks, current_tokens, chunk_index, current_metadata
+        if not current_blocks:
+            return
+        content_str = "\n\n".join([b["text"] for b in current_blocks])
+        chunk_id = f"{doc.source_id}_chunk_{chunk_index:03d}"
+        chunks.append(Chunk(
+            chunk_id=chunk_id,
+            source_id=doc.source_id,
+            path=f"chunks/{chunk_id}.md",
+            token_count=current_tokens,
+            content=content_str,
+            metadata=current_metadata.copy()
+        ))
+        chunk_index += 1
+        
+        # Keep blocks for overlap
+        overlap_blocks = []
+        overlap_toks = 0
+        for b in reversed(current_blocks):
+            if overlap_toks + b["tokens"] > overlap_tokens_target:
+                break
+            # don't overlap tables to avoid duplicating massive tables
+            if b["type"] == "table" and overlap_toks > 0:
+                break
+            overlap_blocks.insert(0, b)
+            overlap_toks += b["tokens"]
+            
+        current_blocks = overlap_blocks
+        current_tokens = overlap_toks
+
+    for block in doc.blocks:
+        if not block.text:
+            continue
+            
+        block_tokens = len(encoder.encode(block.text))
+        
+        # Update metadata
+        if block.section_path:
+            current_metadata["section"] = block.section_path[-1]
+        if block.page:
+            current_metadata["page"] = block.page
+            
+        if block.row_range:
+            current_metadata["row_range"] = list(block.row_range)
+            
+        # Very large blocks might exceed max_tokens, in a full implementation we'd split the block itself.
+        # For this MVP, we will just start a new chunk if adding this block pushes us over, unless we are empty.
+        if current_tokens + block_tokens > max_tokens and current_tokens > 0:
+            create_chunk()
+            
+        current_blocks.append({"text": block.text, "tokens": block_tokens, "type": block.type})
+        current_tokens += block_tokens
+        
+    create_chunk()
+    return chunks

agentpack/cli.py

@@ -0,0 +1,148 @@
+import typer
+from agentpack.pack import write_pack
+from agentpack.validate import validate_pack
+from agentpack.audit import audit_pack
+from agentpack.retrieve import search_pack
+from agentpack.eval.runner import run_eval
+
+app = typer.Typer(help="AgentPack CLI", no_args_is_help=True)
+
+@app.command()
+def pack(
+    input_dir: str, 
+    out: str = typer.Option(..., help="Output directory"),
+    include: str = typer.Option(None, help="Include only files matching these glob patterns (comma-separated)"),
+    exclude: str = typer.Option(None, "-i", "--ignore", help="Additional patterns to exclude (comma-separated)"),
+    no_gitignore: bool = typer.Option(False, help="Don't use .gitignore rules for filtering files"),
+    no_default_patterns: bool = typer.Option(False, help="Don't apply built-in ignore patterns"),
+    include_hidden: bool = typer.Option(False, help="Include hidden directories"),
+    verbose: bool = typer.Option(False, help="Enable detailed debug logging"),
+    quiet: bool = typer.Option(False, help="Suppress all console output except errors"),
+    remove_empty_lines: bool = typer.Option(False, help="Remove blank lines from all text files")
+):
+    """Pack documents into an agent-friendly context pack."""
+    if not quiet:
+        typer.echo(f"Packing {input_dir} into {out}...")
+        
+    include_patterns = include.split(",") if include else None
+    exclude_patterns = exclude.split(",") if exclude else None
+
+    write_pack(
+        input_dir=input_dir, 
+        output_dir=out,
+        include_patterns=include_patterns,
+        exclude_patterns=exclude_patterns,
+        no_gitignore=no_gitignore,
+        no_default_patterns=no_default_patterns,
+        include_hidden=include_hidden,
+        verbose=verbose,
+        quiet=quiet,
+        remove_empty_lines=remove_empty_lines
+    )
+    
+    if not quiet:
+        typer.echo("Done.")
+
+@app.command()
+def validate(pack_dir: str):
+    """Validates the structural integrity of a context pack."""
+    typer.echo(f"Validating pack at {pack_dir}...")
+    errors = validate_pack(pack_dir)
+    if errors:
+        typer.secho("Validation failed with errors:", fg=typer.colors.RED)
+        for err in errors:
+            typer.echo(f"- {err}")
+        raise typer.Exit(code=1)
+    else:
+        typer.secho("Pack validation successful.", fg=typer.colors.GREEN)
+
+@app.command()
+def audit(pack_dir: str):
+    """Generates an audit report for a context pack."""
+    typer.echo(f"Auditing pack at {pack_dir}...")
+    report = audit_pack(pack_dir)
+    if report.startswith("Error:"):
+        typer.secho(report, fg=typer.colors.RED)
+        raise typer.Exit(code=1)
+    else:
+        typer.echo(report)
+        typer.secho("\nAudit report generated.", fg=typer.colors.GREEN)
+
+@app.command()
+def retrieve(
+    pack_dir: str, 
+    query: str, 
+    top_k: int = typer.Option(5, help="Number of results to return"),
+    mode: str = typer.Option("hybrid", help="Search mode: hybrid, vector, or fts")
+):
+    """Retrieves top-k evidence chunks from a pack."""
+    typer.echo(f"Searching for '{query}' in {pack_dir} using {mode} mode...")
+    results = search_pack(pack_dir, query, top_k, mode=mode)
+    
+    if not results:
+        typer.secho("No results found.", fg=typer.colors.YELLOW)
+        return
+        
+    for i, res in enumerate(results):
+        source = res['citation'].get('source_path', res['source_id'])
+        section = res['citation'].get('section', '')
+        page = res['citation'].get('page', '')
+        
+        cite_str = source
+        if page:
+            cite_str += f", page {page}"
+        if section:
+            cite_str += f", {section}"
+            
+        typer.secho(f"\n{i+1}. {cite_str}", fg=typer.colors.CYAN, bold=True)
+        typer.echo(f"   chunk: {res['path']}")
+        typer.echo(f"   tokens: {res['token_count']}")
+        typer.echo(f"   score: {res['score']:.2f}")
+
+@app.command(name="eval")
+def evaluate(benchmark_dir: str):
+    """Runs a deterministic evaluation benchmark."""
+    typer.echo(f"Running evaluation on {benchmark_dir}...")
+    report = run_eval(benchmark_dir)
+    if report.startswith("Error:"):
+        typer.secho(report, fg=typer.colors.RED)
+        raise typer.Exit(code=1)
+    else:
+        typer.echo(report)
+        typer.secho("\nEvaluation complete.", fg=typer.colors.GREEN)
+
+@app.command(name="gen-eval")
+def gen_eval(benchmark_dir: str, gen_model: str = "gemini-1.5-flash", judge_model: str = "gemini-1.5-pro", limit: int = typer.Option(None, help="Limit number of queries for a smoke test")):
+    """Evaluate Generative QA using AgentPack"""
+    from agentpack.eval.generation import run_generation_eval
+    
+    typer.echo(f"Running generative evaluation on {benchmark_dir}...")
+    report = run_generation_eval(benchmark_dir, gen_model, judge_model, limit)
+    if report.startswith("Error"):
+        typer.secho(report, fg=typer.colors.RED)
+    else:
+        typer.echo(report)
+        typer.secho("Generation evaluation complete.", fg=typer.colors.GREEN)
+
+@app.command(name="prep-benchmark")
+def prep_benchmark(dataset: str = typer.Option("financebench", help="Dataset to slice"), sample_size: int = typer.Option(10, help="Number of documents to sample")):
+    """Downloads and slices a public dataset into a benchmark format."""
+    from agentpack.eval.benchmarks import slice_financebench, slice_tatqa, slice_qasper
+    
+    out_dir = f"benchmarks/{dataset}_sample"
+    typer.echo(f"Preparing {dataset} into {out_dir} with {sample_size} samples...")
+    
+    if dataset == "financebench":
+        slice_financebench(out_dir, sample_size=sample_size)
+    elif dataset == "tatqa":
+        slice_tatqa(out_dir, sample_size=sample_size)
+    elif dataset == "qasper":
+        slice_qasper(out_dir, sample_size=sample_size)
+    else:
+        typer.secho(f"Unsupported dataset: {dataset}", fg=typer.colors.RED)
+        raise typer.Exit(code=1)
+        
+    typer.secho("Preparation complete.", fg=typer.colors.GREEN)
+
+if __name__ == "__main__":
+    app()

agentpack/eval/__init__.py

@@ -0,0 +1 @@
+"""Evaluation module for benchmarking AgentPack."""

agentpack/eval/baselines.py

@@ -0,0 +1,134 @@
+import sqlite3
+import os
+import tiktoken
+from pathlib import Path
+from typing import List, Dict
+
+def _extract_text(file_path: Path) -> str:
+    if file_path.suffix.lower() == ".pdf":
+        try:
+            import fitz
+            doc = fitz.open(file_path)
+            text = []
+            for page in doc:
+                text.append(page.get_text("text"))
+            return "\n".join(text)
+        except Exception:
+            return ""
+    else:
+        try:
+            with open(file_path, "r", encoding="utf-8", errors="replace") as f:
+                return f.read()
+        except Exception:
+            return ""
+
+_raw_conn_cache = None
+_naive_conn_cache = None
+_last_corpus_dir = None
+
+def _get_raw_conn(corpus_dir: Path):
+    global _raw_conn_cache, _last_corpus_dir
+    if _raw_conn_cache is not None and _last_corpus_dir == corpus_dir:
+        return _raw_conn_cache
+        
+    conn = sqlite3.connect(":memory:")
+    cur = conn.cursor()
+    cur.execute('CREATE VIRTUAL TABLE files_fts USING fts5(path UNINDEXED, content)')
+    
+    from tqdm import tqdm
+    all_files = []
+    for root, _, files in os.walk(corpus_dir):
+        for file in files:
+            p = Path(root) / file
+            if p.suffix in [".txt", ".md", ".csv", ".pdf"]:
+                all_files.append(p)
+                
+    for p in tqdm(all_files, desc="Building Raw File Baseline Index"):
+        content = _extract_text(p)
+        cur.execute("INSERT INTO files_fts (path, content) VALUES (?, ?)", (p.name, content))
+        
+    _raw_conn_cache = conn
+    _last_corpus_dir = corpus_dir
+    return conn
+
+def _get_naive_conn(corpus_dir: Path, chunk_size: int = 4000):
+    global _naive_conn_cache
+    if _naive_conn_cache is not None and _last_corpus_dir == corpus_dir:
+        return _naive_conn_cache
+        
+    conn = sqlite3.connect(":memory:")
+    cur = conn.cursor()
+    cur.execute('CREATE VIRTUAL TABLE chunks_fts USING fts5(path UNINDEXED, chunk_id UNINDEXED, content)')
+    
+    chunk_id = 0
+    
+    from tqdm import tqdm
+    all_files = []
+    for root, _, files in os.walk(corpus_dir):
+        for file in files:
+            p = Path(root) / file
+            if p.suffix in [".txt", ".md", ".csv", ".pdf"]:
+                all_files.append(p)
+                
+    for p in tqdm(all_files, desc="Building Naive Chunk Baseline Index"):
+        content = _extract_text(p)
+        for i in range(0, len(content), chunk_size):
+            chunk_text = content[i:i+chunk_size]
+            cur.execute("INSERT INTO chunks_fts (path, chunk_id, content) VALUES (?, ?, ?)", (p.name, chunk_id, chunk_text))
+            chunk_id += 1
+            
+    _naive_conn_cache = conn
+    return conn
+
+def raw_file_search(corpus_dir: Path, query: str, top_k: int = 5) -> List[Dict]:
+    conn = _get_raw_conn(corpus_dir)
+    cur = conn.cursor()
+    
+    clean_query = " OR ".join([f'"{w}"' for w in query.replace('"', '').split() if w])
+    if not clean_query: return []
+    
+    try:
+        cur.execute('SELECT path, rank, content FROM files_fts WHERE files_fts MATCH ? ORDER BY rank LIMIT ?', (clean_query, top_k))
+    except sqlite3.OperationalError:
+        try:
+            cur.execute('SELECT path, rank, content FROM files_fts WHERE files_fts MATCH ? ORDER BY rank LIMIT ?', (query, top_k))
+        except sqlite3.OperationalError:
+            return []
+            
+    encoder = tiktoken.get_encoding("cl100k_base")
+    results = []
+    for path, rank, content in cur.fetchall():
+        results.append({
+            "path": path,
+            "token_count": len(encoder.encode(content)),
+            "score": rank,
+            "content": content
+        })
+    return results
+
+def naive_chunk_search(corpus_dir: Path, query: str, top_k: int = 5, chunk_size: int = 4000) -> List[Dict]:
+    conn = _get_naive_conn(corpus_dir, chunk_size)
+    cur = conn.cursor()
+    
+    clean_query = " OR ".join([f'"{w}"' for w in query.replace('"', '').split() if w])
+    if not clean_query: return []
+    
+    try:
+        cur.execute('SELECT path, chunk_id, rank, content FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?', (clean_query, top_k))
+    except sqlite3.OperationalError:
+        try:
+            cur.execute('SELECT path, chunk_id, rank, content FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?', (query, top_k))
+        except sqlite3.OperationalError:
+            return []
+            
+    encoder = tiktoken.get_encoding("cl100k_base")
+    results = []
+    for path, chunk_id, rank, content in cur.fetchall():
+        results.append({
+            "chunk_id": chunk_id,
+            "path": path,
+            "token_count": len(encoder.encode(content)),
+            "score": rank,
+            "content": content
+        })
+    return results