aigroup-workflow 2.2.1 → 2.2.2

package/skills/ai-ml/rag-architect/references/chunking-strategies.md

@@ -1,878 +1,878 @@
-# Chunking Strategies
-
----
-
-## Strategy Comparison Matrix
-
-| Strategy | Best For | Chunk Quality | Implementation Complexity |
-|----------|----------|---------------|---------------------------|
-| **Fixed-size** | Simple documents, logs | Low-Medium | Simple |
-| **Recursive character** | General text, articles | Medium | Simple |
-| **Sentence-based** | Conversational, Q&A | Medium-High | Medium |
-| **Semantic** | Technical docs, manuals | High | Medium |
-| **Document-aware** | Structured content (MD, HTML) | High | Medium |
-| **Agentic/Contextual** | Complex documents | Very High | Complex |
-| **Late chunking** | Long-context embeddings | High | Medium |
-
----
-
-## When to Use Each Strategy
-
-### Fixed-Size Chunking
-```
-Best For:
-- Log files and structured data
-- Quick prototyping
-- When content has no natural structure
-- Baseline comparison
-
-When to Avoid:
-- Technical documentation
-- Content with semantic units (paragraphs, sections)
-- When context preservation matters
-```
-
-### Recursive Character Splitting
-```
-Best For:
-- General articles and blog posts
-- Mixed content types
-- Default starting point for most RAG
-- LangChain/LlamaIndex default
-
-When to Avoid:
-- Highly structured documents
-- Code-heavy content
-- Tables and lists
-```
-
-### Semantic Chunking
-```
-Best For:
-- Technical documentation
-- Research papers
-- Content with natural topic boundaries
-- When retrieval precision is critical
-
-When to Avoid:
-- Real-time ingestion (slower)
-- Very short documents
-- Cost-sensitive pipelines (requires embeddings)
-```
-
-### Document-Aware Chunking
-```
-Best For:
-- Markdown documentation
-- HTML pages
-- LaTeX papers
-- Code files
-
-When to Avoid:
-- Plain text without structure
-- Inconsistent formatting
-```
-
----
-
-## Fixed-Size Chunking
-
-```python
-def fixed_size_chunk(
-    text: str,
-    chunk_size: int = 500,
-    overlap: int = 50
-) -> list[str]:
-    """Simple fixed-size chunking with overlap."""
-    chunks = []
-    start = 0
-
-    while start < len(text):
-        end = start + chunk_size
-        chunk = text[start:end]
-
-        # Try to break at word boundary
-        if end < len(text):
-            last_space = chunk.rfind(' ')
-            if last_space > chunk_size * 0.8:  # Only if reasonably far in
-                chunk = chunk[:last_space]
-                end = start + last_space
-
-        chunks.append(chunk.strip())
-        start = end - overlap
-
-    return chunks
-
-# Usage
-chunks = fixed_size_chunk(document_text, chunk_size=500, overlap=50)
-```
-
----
-
-## Recursive Character Splitting (LangChain Style)
-
-```python
-from typing import Callable
-
-class RecursiveCharacterSplitter:
-    """Split text recursively using multiple separators."""
-
-    def __init__(
-        self,
-        chunk_size: int = 1000,
-        chunk_overlap: int = 200,
-        separators: list[str] | None = None,
-        length_function: Callable[[str], int] = len
-    ):
-        self.chunk_size = chunk_size
-        self.chunk_overlap = chunk_overlap
-        self.separators = separators or ["\n\n", "\n", ". ", " ", ""]
-        self.length_function = length_function
-
-    def split_text(self, text: str) -> list[str]:
-        """Split text into chunks."""
-        return self._split_text(text, self.separators)
-
-    def _split_text(self, text: str, separators: list[str]) -> list[str]:
-        final_chunks = []
-        separator = separators[-1]
-
-        for i, sep in enumerate(separators):
-            if sep == "":
-                separator = sep
-                break
-            if sep in text:
-                separator = sep
-                break
-
-        splits = text.split(separator) if separator else list(text)
-
-        good_splits = []
-        for split in splits:
-            if self.length_function(split) < self.chunk_size:
-                good_splits.append(split)
-            else:
-                if good_splits:
-                    merged = self._merge_splits(good_splits, separator)
-                    final_chunks.extend(merged)
-                    good_splits = []
-                # Recursively split large chunks
-                other_chunks = self._split_text(split, separators[separators.index(separator) + 1:])
-                final_chunks.extend(other_chunks)
-
-        if good_splits:
-            merged = self._merge_splits(good_splits, separator)
-            final_chunks.extend(merged)
-
-        return final_chunks
-
-    def _merge_splits(self, splits: list[str], separator: str) -> list[str]:
-        """Merge splits into chunks respecting size limits."""
-        chunks = []
-        current_chunk = []
-        current_length = 0
-
-        for split in splits:
-            split_length = self.length_function(split)
-
-            if current_length + split_length > self.chunk_size:
-                if current_chunk:
-                    chunks.append(separator.join(current_chunk))
-                    # Keep overlap
-                    while current_length > self.chunk_overlap and current_chunk:
-                        current_length -= self.length_function(current_chunk[0])
-                        current_chunk = current_chunk[1:]
-
-            current_chunk.append(split)
-            current_length += split_length
-
-        if current_chunk:
-            chunks.append(separator.join(current_chunk))
-
-        return chunks
-
-# Usage
-splitter = RecursiveCharacterSplitter(
-    chunk_size=1000,
-    chunk_overlap=200,
-    separators=["\n\n", "\n", ". ", " "]
-)
-chunks = splitter.split_text(document_text)
-```
-
-### Token-Based Splitting
-
-```python
-import tiktoken
-
-def create_token_splitter(
-    model: str = "gpt-4",
-    chunk_size: int = 500,
-    chunk_overlap: int = 50
-):
-    """Create splitter that counts tokens instead of characters."""
-    encoding = tiktoken.encoding_for_model(model)
-
-    def token_length(text: str) -> int:
-        return len(encoding.encode(text))
-
-    return RecursiveCharacterSplitter(
-        chunk_size=chunk_size,
-        chunk_overlap=chunk_overlap,
-        length_function=token_length
-    )
-
-# Usage
-token_splitter = create_token_splitter(chunk_size=500, chunk_overlap=50)
-chunks = token_splitter.split_text(document_text)
-```
-
----
-
-## Sentence-Based Chunking
-
-```python
-import re
-from dataclasses import dataclass
-
-@dataclass
-class SentenceChunk:
-    text: str
-    sentences: list[str]
-    start_sentence: int
-    end_sentence: int
-
-def sentence_chunk(
-    text: str,
-    sentences_per_chunk: int = 5,
-    overlap_sentences: int = 1
-) -> list[SentenceChunk]:
-    """Chunk by sentence count with overlap."""
-    # Split into sentences
-    sentence_pattern = r'(?<=[.!?])\s+'
-    sentences = re.split(sentence_pattern, text)
-    sentences = [s.strip() for s in sentences if s.strip()]
-
-    chunks = []
-    i = 0
-
-    while i < len(sentences):
-        end = min(i + sentences_per_chunk, len(sentences))
-        chunk_sentences = sentences[i:end]
-
-        chunks.append(SentenceChunk(
-            text=" ".join(chunk_sentences),
-            sentences=chunk_sentences,
-            start_sentence=i,
-            end_sentence=end - 1
-        ))
-
-        i += sentences_per_chunk - overlap_sentences
-
-    return chunks
-
-# Better sentence splitting with NLTK
-import nltk
-nltk.download('punkt')
-from nltk.tokenize import sent_tokenize
-
-def sentence_chunk_nltk(
-    text: str,
-    max_chunk_size: int = 1000,
-    overlap_sentences: int = 2
-) -> list[str]:
-    """Chunk by sentences up to max size."""
-    sentences = sent_tokenize(text)
-    chunks = []
-    current_chunk = []
-    current_size = 0
-
-    for sentence in sentences:
-        sentence_size = len(sentence)
-
-        if current_size + sentence_size > max_chunk_size and current_chunk:
-            chunks.append(" ".join(current_chunk))
-            # Keep overlap sentences
-            current_chunk = current_chunk[-overlap_sentences:] if overlap_sentences else []
-            current_size = sum(len(s) for s in current_chunk)
-
-        current_chunk.append(sentence)
-        current_size += sentence_size
-
-    if current_chunk:
-        chunks.append(" ".join(current_chunk))
-
-    return chunks
-```
-
----
-
-## Semantic Chunking
-
-```python
-import numpy as np
-from sentence_transformers import SentenceTransformer
-from sklearn.metrics.pairwise import cosine_similarity
-
-class SemanticChunker:
-    """Chunk based on semantic similarity between sentences."""
-
-    def __init__(
-        self,
-        model_name: str = "all-MiniLM-L6-v2",
-        similarity_threshold: float = 0.5,
-        min_chunk_size: int = 100,
-        max_chunk_size: int = 1500
-    ):
-        self.model = SentenceTransformer(model_name)
-        self.similarity_threshold = similarity_threshold
-        self.min_chunk_size = min_chunk_size
-        self.max_chunk_size = max_chunk_size
-
-    def chunk(self, text: str) -> list[str]:
-        """Split text at semantic boundaries."""
-        # Split into sentences
-        sentences = self._split_sentences(text)
-        if len(sentences) <= 1:
-            return [text]
-
-        # Get embeddings
-        embeddings = self.model.encode(sentences)
-
-        # Find breakpoints based on similarity drops
-        breakpoints = self._find_breakpoints(embeddings)
-
-        # Create chunks
-        chunks = []
-        start = 0
-
-        for bp in breakpoints:
-            chunk_text = " ".join(sentences[start:bp])
-
-            # Handle size constraints
-            if len(chunk_text) > self.max_chunk_size:
-                # Split large chunks
-                sub_chunks = self._split_large_chunk(sentences[start:bp])
-                chunks.extend(sub_chunks)
-            elif len(chunk_text) >= self.min_chunk_size:
-                chunks.append(chunk_text)
-            elif chunks:
-                # Merge small chunk with previous
-                chunks[-1] += " " + chunk_text
-            else:
-                chunks.append(chunk_text)
-
-            start = bp
-
-        # Handle remaining sentences
-        if start < len(sentences):
-            remaining = " ".join(sentences[start:])
-            if chunks and len(remaining) < self.min_chunk_size:
-                chunks[-1] += " " + remaining
-            else:
-                chunks.append(remaining)
-
-        return chunks
-
-    def _split_sentences(self, text: str) -> list[str]:
-        """Split text into sentences."""
-        import re
-        sentences = re.split(r'(?<=[.!?])\s+', text)
-        return [s.strip() for s in sentences if s.strip()]
-
-    def _find_breakpoints(self, embeddings: np.ndarray) -> list[int]:
-        """Find semantic breakpoints using similarity drops."""
-        breakpoints = []
-
-        for i in range(1, len(embeddings)):
-            similarity = cosine_similarity(
-                embeddings[i-1:i],
-                embeddings[i:i+1]
-            )[0][0]
-
-            if similarity < self.similarity_threshold:
-                breakpoints.append(i)
-
-        return breakpoints
-
-    def _split_large_chunk(self, sentences: list[str]) -> list[str]:
-        """Split oversized chunk at midpoint."""
-        mid = len(sentences) // 2
-        return [
-            " ".join(sentences[:mid]),
-            " ".join(sentences[mid:])
-        ]
-
-# Usage
-chunker = SemanticChunker(
-    similarity_threshold=0.5,
-    min_chunk_size=200,
-    max_chunk_size=1000
-)
-semantic_chunks = chunker.chunk(document_text)
-```
-
-### Percentile-Based Breakpoints
-
-```python
-def find_breakpoints_percentile(
-    embeddings: np.ndarray,
-    percentile: int = 25
-) -> list[int]:
-    """Find breakpoints at similarity drops below percentile threshold."""
-    similarities = []
-
-    for i in range(1, len(embeddings)):
-        sim = cosine_similarity(
-            embeddings[i-1:i],
-            embeddings[i:i+1]
-        )[0][0]
-        similarities.append((i, sim))
-
-    # Dynamic threshold based on distribution
-    sim_values = [s[1] for s in similarities]
-    threshold = np.percentile(sim_values, percentile)
-
-    return [i for i, sim in similarities if sim < threshold]
-```
-
----
-
-## Document-Aware Chunking
-
-### Markdown Chunking
-
-```python
-import re
-from dataclasses import dataclass
-
-@dataclass
-class MarkdownChunk:
-    text: str
-    heading: str | None
-    heading_level: int
-    metadata: dict
-
-def chunk_markdown(
-    text: str,
-    max_chunk_size: int = 1500,
-    include_heading_in_chunk: bool = True
-) -> list[MarkdownChunk]:
-    """Chunk markdown by headers while respecting structure."""
-    # Pattern to match headers
-    header_pattern = r'^(#{1,6})\s+(.+)$'
-
-    lines = text.split('\n')
-    chunks = []
-    current_chunk_lines = []
-    current_heading = None
-    current_level = 0
-    heading_stack = []  # For breadcrumb context
-
-    for line in lines:
-        header_match = re.match(header_pattern, line)
-
-        if header_match:
-            # Save current chunk if exists
-            if current_chunk_lines:
-                chunk_text = '\n'.join(current_chunk_lines)
-                if len(chunk_text.strip()) > 0:
-                    prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
-                    chunks.append(MarkdownChunk(
-                        text=prefix + chunk_text,
-                        heading=current_heading,
-                        heading_level=current_level,
-                        metadata={"breadcrumb": " > ".join(heading_stack)}
-                    ))
-
-            # Update heading context
-            level = len(header_match.group(1))
-            heading = header_match.group(2).strip()
-
-            # Maintain heading stack for breadcrumbs
-            while heading_stack and current_level >= level:
-                heading_stack.pop()
-                current_level -= 1
-
-            heading_stack.append(heading)
-            current_heading = heading
-            current_level = level
-            current_chunk_lines = []
-
-        else:
-            current_chunk_lines.append(line)
-
-            # Check chunk size
-            current_text = '\n'.join(current_chunk_lines)
-            if len(current_text) > max_chunk_size:
-                # Split at paragraph boundary
-                paragraphs = current_text.split('\n\n')
-                if len(paragraphs) > 1:
-                    split_point = len('\n\n'.join(paragraphs[:-1]))
-                    chunk_text = current_text[:split_point]
-                    prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
-                    chunks.append(MarkdownChunk(
-                        text=prefix + chunk_text,
-                        heading=current_heading,
-                        heading_level=current_level,
-                        metadata={"breadcrumb": " > ".join(heading_stack)}
-                    ))
-                    current_chunk_lines = [current_text[split_point:].strip()]
-
-    # Don't forget the last chunk
-    if current_chunk_lines:
-        chunk_text = '\n'.join(current_chunk_lines)
-        if len(chunk_text.strip()) > 0:
-            prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
-            chunks.append(MarkdownChunk(
-                text=prefix + chunk_text,
-                heading=current_heading,
-                heading_level=current_level,
-                metadata={"breadcrumb": " > ".join(heading_stack)}
-            ))
-
-    return chunks
-```
-
-### Code-Aware Chunking
-
-```python
-import re
-from dataclasses import dataclass
-
-@dataclass
-class CodeChunk:
-    text: str
-    language: str | None
-    chunk_type: str  # "code", "text", "mixed"
-
-def chunk_with_code_blocks(
-    text: str,
-    max_chunk_size: int = 1500
-) -> list[CodeChunk]:
-    """Chunk text while keeping code blocks intact."""
-    # Pattern to match code blocks
-    code_block_pattern = r'```(\w+)?\n(.*?)```'
-
-    chunks = []
-    last_end = 0
-
-    for match in re.finditer(code_block_pattern, text, re.DOTALL):
-        # Text before code block
-        text_before = text[last_end:match.start()].strip()
-        if text_before:
-            # Chunk the text portion
-            text_chunks = recursive_chunk(text_before, max_chunk_size)
-            chunks.extend([
-                CodeChunk(text=t, language=None, chunk_type="text")
-                for t in text_chunks
-            ])
-
-        # Code block (keep intact if possible)
-        language = match.group(1)
-        code_content = match.group(2)
-        full_block = match.group(0)
-
-        if len(full_block) <= max_chunk_size:
-            chunks.append(CodeChunk(
-                text=full_block,
-                language=language,
-                chunk_type="code"
-            ))
-        else:
-            # Split large code blocks by function/class
-            code_chunks = split_code_block(code_content, language, max_chunk_size)
-            chunks.extend(code_chunks)
-
-        last_end = match.end()
-
-    # Remaining text after last code block
-    remaining = text[last_end:].strip()
-    if remaining:
-        text_chunks = recursive_chunk(remaining, max_chunk_size)
-        chunks.extend([
-            CodeChunk(text=t, language=None, chunk_type="text")
-            for t in text_chunks
-        ])
-
-    return chunks
-
-def split_code_block(code: str, language: str, max_size: int) -> list[CodeChunk]:
-    """Split code block at logical boundaries."""
-    # Simple function/class boundary splitting for Python
-    if language == "python":
-        pattern = r'\n(?=def |class |async def )'
-    elif language in ["javascript", "typescript"]:
-        pattern = r'\n(?=function |class |const |export )'
-    else:
-        pattern = r'\n\n'
-
-    parts = re.split(pattern, code)
-    chunks = []
-    current = ""
-
-    for part in parts:
-        if len(current) + len(part) > max_size and current:
-            chunks.append(CodeChunk(
-                text=f"```{language}\n{current}```",
-                language=language,
-                chunk_type="code"
-            ))
-            current = part
-        else:
-            current += part
-
-    if current:
-        chunks.append(CodeChunk(
-            text=f"```{language}\n{current}```",
-            language=language,
-            chunk_type="code"
-        ))
-
-    return chunks
-```
-
----
-
-## Contextual/Agentic Chunking
-
-```python
-from openai import OpenAI
-
-def contextual_chunk(
-    document: str,
-    max_chunk_size: int = 1500
-) -> list[dict]:
-    """Use LLM to add context to each chunk."""
-    # First, do structural chunking
-    base_chunks = recursive_chunk(document, max_chunk_size)
-
-    client = OpenAI()
-    contextualized_chunks = []
-
-    for chunk in base_chunks:
-        # Generate contextual summary
-        response = client.chat.completions.create(
-            model="gpt-4o-mini",
-            messages=[
-                {
-                    "role": "system",
-                    "content": """Provide a brief context for this document chunk.
-                    Include: what topic it covers, how it relates to the broader document,
-                    and key concepts mentioned. Keep it under 100 words."""
-                },
-                {
-                    "role": "user",
-                    "content": f"Document excerpt:\n\n{chunk}"
-                }
-            ],
-            max_tokens=150
-        )
-
-        context = response.choices[0].message.content
-
-        contextualized_chunks.append({
-            "text": chunk,
-            "context": context,
-            "text_with_context": f"Context: {context}\n\nContent: {chunk}"
-        })
-
-    return contextualized_chunks
-```
-
-### Propositions-Based Chunking
-
-```python
-def extract_propositions(text: str) -> list[str]:
-    """Extract atomic propositions from text using LLM."""
-    client = OpenAI()
-
-    response = client.chat.completions.create(
-        model="gpt-4o-mini",
-        messages=[
-            {
-                "role": "system",
-                "content": """Extract atomic propositions from the text.
-                Each proposition should:
-                - Be a single, complete fact
-                - Be self-contained (understandable without context)
-                - Include necessary entity references
-
-                Return as a JSON array of strings."""
-            },
-            {
-                "role": "user",
-                "content": text
-            }
-        ],
-        response_format={"type": "json_object"}
-    )
-
-    import json
-    result = json.loads(response.choices[0].message.content)
-    return result.get("propositions", [])
-
-# Usage: For very fine-grained retrieval
-propositions = extract_propositions(document_text)
-# Each proposition becomes its own retrievable unit
-```
-
----
-
-## Late Chunking (for Long-Context Embeddings)
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-class LateChunker:
-    """
-    Late chunking: embed full document, then pool token embeddings into chunks.
-    Preserves full document context while creating retrievable chunks.
-    """
-
-    def __init__(self, model_name: str = "jinaai/jina-embeddings-v2-base-en"):
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
-        self.model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
-        self.model.eval()
-
-    def chunk_and_embed(
-        self,
-        text: str,
-        chunk_size: int = 512,
-        overlap: int = 64
-    ) -> list[dict]:
-        """
-        Embed full document, then create chunk embeddings via mean pooling.
-        """
-        # Tokenize full document
-        inputs = self.tokenizer(
-            text,
-            return_tensors="pt",
-            truncation=True,
-            max_length=8192  # Model's max context
-        )
-
-        # Get token-level embeddings
-        with torch.no_grad():
-            outputs = self.model(**inputs)
-            token_embeddings = outputs.last_hidden_state[0]  # [seq_len, hidden_dim]
-
-        # Get token-to-text mapping
-        tokens = self.tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
-
-        # Create chunks from token embeddings
-        chunks = []
-        seq_len = token_embeddings.shape[0]
-        start = 0
-
-        while start < seq_len:
-            end = min(start + chunk_size, seq_len)
-
-            # Mean pool token embeddings for this chunk
-            chunk_embedding = token_embeddings[start:end].mean(dim=0).numpy()
-
-            # Reconstruct text for this chunk
-            chunk_token_ids = inputs["input_ids"][0][start:end]
-            chunk_text = self.tokenizer.decode(chunk_token_ids, skip_special_tokens=True)
-
-            chunks.append({
-                "text": chunk_text,
-                "embedding": chunk_embedding,
-                "start_token": start,
-                "end_token": end
-            })
-
-            start = end - overlap
-
-        return chunks
-
-# Usage
-late_chunker = LateChunker()
-chunks_with_embeddings = late_chunker.chunk_and_embed(
-    long_document,
-    chunk_size=512,
-    overlap=64
-)
-```
-
----
-
-## Metadata Enrichment
-
-```python
-from dataclasses import dataclass
-from datetime import datetime
-import hashlib
-
-@dataclass
-class EnrichedChunk:
-    text: str
-    embedding: list[float] | None
-    metadata: dict
-
-def enrich_chunk(
-    text: str,
-    source_file: str,
-    chunk_index: int,
-    total_chunks: int,
-    additional_metadata: dict | None = None
-) -> EnrichedChunk:
-    """Add comprehensive metadata to chunk."""
-    metadata = {
-        # Source tracking
-        "source": source_file,
-        "chunk_index": chunk_index,
-        "total_chunks": total_chunks,
-
-        # Content characteristics
-        "char_count": len(text),
-        "word_count": len(text.split()),
-        "content_hash": hashlib.md5(text.encode()).hexdigest()[:12],
-
-        # Temporal
-        "indexed_at": datetime.utcnow().isoformat(),
-
-        # Position context
-        "position": "start" if chunk_index == 0 else (
-            "end" if chunk_index == total_chunks - 1 else "middle"
-        )
-    }
-
-    if additional_metadata:
-        metadata.update(additional_metadata)
-
-    return EnrichedChunk(text=text, embedding=None, metadata=metadata)
-```
-
----
-
-## Chunk Size Selection Guide
-
-| Document Type | Recommended Size | Overlap | Rationale |
-|--------------|------------------|---------|-----------|
-| FAQ/Q&A | 200-400 tokens | 20-50 | Keep Q&A pairs together |
-| Technical docs | 400-600 tokens | 50-100 | Balance context vs precision |
-| Legal/contracts | 600-800 tokens | 100-150 | Preserve clause context |
-| Code documentation | 300-500 tokens | 50-100 | Keep function docs together |
-| Chat transcripts | 150-300 tokens | 25-50 | Natural turn boundaries |
-| Research papers | 500-800 tokens | 100-200 | Section-level coherence |
-
----
-
-## Quick Reference
-
-| Strategy | Use Case | Code Pattern |
-|----------|----------|--------------|
-| Fixed-size | Logs, baseline | `text[i:i+chunk_size]` |
-| Recursive | General text | Split by `["\n\n", "\n", ". "]` |
-| Sentence | Q&A content | `sent_tokenize()` + merge |
-| Semantic | Technical docs | Similarity-based breaks |
-| Markdown | Documentation | Header-aware splitting |
-| Late chunking | Long-context models | Embed full, pool chunks |
-
-## Related Skills
-
-- **RAG Architect** - Integration with vector databases
-- **Python Pro** - Preprocessing pipelines
-- **NLP Engineer** - Tokenization and text processing
+# Chunking Strategies
+
+---
+
+## Strategy Comparison Matrix
+
+| Strategy | Best For | Chunk Quality | Implementation Complexity |
+|----------|----------|---------------|---------------------------|
+| **Fixed-size** | Simple documents, logs | Low-Medium | Simple |
+| **Recursive character** | General text, articles | Medium | Simple |
+| **Sentence-based** | Conversational, Q&A | Medium-High | Medium |
+| **Semantic** | Technical docs, manuals | High | Medium |
+| **Document-aware** | Structured content (MD, HTML) | High | Medium |
+| **Agentic/Contextual** | Complex documents | Very High | Complex |
+| **Late chunking** | Long-context embeddings | High | Medium |
+
+---
+
+## When to Use Each Strategy
+
+### Fixed-Size Chunking
+```
+Best For:
+- Log files and structured data
+- Quick prototyping
+- When content has no natural structure
+- Baseline comparison
+
+When to Avoid:
+- Technical documentation
+- Content with semantic units (paragraphs, sections)
+- When context preservation matters
+```
+
+### Recursive Character Splitting
+```
+Best For:
+- General articles and blog posts
+- Mixed content types
+- Default starting point for most RAG
+- LangChain/LlamaIndex default
+
+When to Avoid:
+- Highly structured documents
+- Code-heavy content
+- Tables and lists
+```
+
+### Semantic Chunking
+```
+Best For:
+- Technical documentation
+- Research papers
+- Content with natural topic boundaries
+- When retrieval precision is critical
+
+When to Avoid:
+- Real-time ingestion (slower)
+- Very short documents
+- Cost-sensitive pipelines (requires embeddings)
+```
+
+### Document-Aware Chunking
+```
+Best For:
+- Markdown documentation
+- HTML pages
+- LaTeX papers
+- Code files
+
+When to Avoid:
+- Plain text without structure
+- Inconsistent formatting
+```
+
+---
+
+## Fixed-Size Chunking
+
+```python
+def fixed_size_chunk(
+    text: str,
+    chunk_size: int = 500,
+    overlap: int = 50
+) -> list[str]:
+    """Simple fixed-size chunking with overlap."""
+    chunks = []
+    start = 0
+
+    while start < len(text):
+        end = start + chunk_size
+        chunk = text[start:end]
+
+        # Try to break at word boundary
+        if end < len(text):
+            last_space = chunk.rfind(' ')
+            if last_space > chunk_size * 0.8:  # Only if reasonably far in
+                chunk = chunk[:last_space]
+                end = start + last_space
+
+        chunks.append(chunk.strip())
+        start = end - overlap
+
+    return chunks
+
+# Usage
+chunks = fixed_size_chunk(document_text, chunk_size=500, overlap=50)
+```
+
+---
+
+## Recursive Character Splitting (LangChain Style)
+
+```python
+from typing import Callable
+
+class RecursiveCharacterSplitter:
+    """Split text recursively using multiple separators."""
+
+    def __init__(
+        self,
+        chunk_size: int = 1000,
+        chunk_overlap: int = 200,
+        separators: list[str] | None = None,
+        length_function: Callable[[str], int] = len
+    ):
+        self.chunk_size = chunk_size
+        self.chunk_overlap = chunk_overlap
+        self.separators = separators or ["\n\n", "\n", ". ", " ", ""]
+        self.length_function = length_function
+
+    def split_text(self, text: str) -> list[str]:
+        """Split text into chunks."""
+        return self._split_text(text, self.separators)
+
+    def _split_text(self, text: str, separators: list[str]) -> list[str]:
+        final_chunks = []
+        separator = separators[-1]
+
+        for i, sep in enumerate(separators):
+            if sep == "":
+                separator = sep
+                break
+            if sep in text:
+                separator = sep
+                break
+
+        splits = text.split(separator) if separator else list(text)
+
+        good_splits = []
+        for split in splits:
+            if self.length_function(split) < self.chunk_size:
+                good_splits.append(split)
+            else:
+                if good_splits:
+                    merged = self._merge_splits(good_splits, separator)
+                    final_chunks.extend(merged)
+                    good_splits = []
+                # Recursively split large chunks
+                other_chunks = self._split_text(split, separators[separators.index(separator) + 1:])
+                final_chunks.extend(other_chunks)
+
+        if good_splits:
+            merged = self._merge_splits(good_splits, separator)
+            final_chunks.extend(merged)
+
+        return final_chunks
+
+    def _merge_splits(self, splits: list[str], separator: str) -> list[str]:
+        """Merge splits into chunks respecting size limits."""
+        chunks = []
+        current_chunk = []
+        current_length = 0
+
+        for split in splits:
+            split_length = self.length_function(split)
+
+            if current_length + split_length > self.chunk_size:
+                if current_chunk:
+                    chunks.append(separator.join(current_chunk))
+                    # Keep overlap
+                    while current_length > self.chunk_overlap and current_chunk:
+                        current_length -= self.length_function(current_chunk[0])
+                        current_chunk = current_chunk[1:]
+
+            current_chunk.append(split)
+            current_length += split_length
+
+        if current_chunk:
+            chunks.append(separator.join(current_chunk))
+
+        return chunks
+
+# Usage
+splitter = RecursiveCharacterSplitter(
+    chunk_size=1000,
+    chunk_overlap=200,
+    separators=["\n\n", "\n", ". ", " "]
+)
+chunks = splitter.split_text(document_text)
+```
+
+### Token-Based Splitting
+
+```python
+import tiktoken
+
+def create_token_splitter(
+    model: str = "gpt-4",
+    chunk_size: int = 500,
+    chunk_overlap: int = 50
+):
+    """Create splitter that counts tokens instead of characters."""
+    encoding = tiktoken.encoding_for_model(model)
+
+    def token_length(text: str) -> int:
+        return len(encoding.encode(text))
+
+    return RecursiveCharacterSplitter(
+        chunk_size=chunk_size,
+        chunk_overlap=chunk_overlap,
+        length_function=token_length
+    )
+
+# Usage
+token_splitter = create_token_splitter(chunk_size=500, chunk_overlap=50)
+chunks = token_splitter.split_text(document_text)
+```
+
+---
+
+## Sentence-Based Chunking
+
+```python
+import re
+from dataclasses import dataclass
+
+@dataclass
+class SentenceChunk:
+    text: str
+    sentences: list[str]
+    start_sentence: int
+    end_sentence: int
+
+def sentence_chunk(
+    text: str,
+    sentences_per_chunk: int = 5,
+    overlap_sentences: int = 1
+) -> list[SentenceChunk]:
+    """Chunk by sentence count with overlap."""
+    # Split into sentences
+    sentence_pattern = r'(?<=[.!?])\s+'
+    sentences = re.split(sentence_pattern, text)
+    sentences = [s.strip() for s in sentences if s.strip()]
+
+    chunks = []
+    i = 0
+
+    while i < len(sentences):
+        end = min(i + sentences_per_chunk, len(sentences))
+        chunk_sentences = sentences[i:end]
+
+        chunks.append(SentenceChunk(
+            text=" ".join(chunk_sentences),
+            sentences=chunk_sentences,
+            start_sentence=i,
+            end_sentence=end - 1
+        ))
+
+        i += sentences_per_chunk - overlap_sentences
+
+    return chunks
+
+# Better sentence splitting with NLTK
+import nltk
+nltk.download('punkt')
+from nltk.tokenize import sent_tokenize
+
+def sentence_chunk_nltk(
+    text: str,
+    max_chunk_size: int = 1000,
+    overlap_sentences: int = 2
+) -> list[str]:
+    """Chunk by sentences up to max size."""
+    sentences = sent_tokenize(text)
+    chunks = []
+    current_chunk = []
+    current_size = 0
+
+    for sentence in sentences:
+        sentence_size = len(sentence)
+
+        if current_size + sentence_size > max_chunk_size and current_chunk:
+            chunks.append(" ".join(current_chunk))
+            # Keep overlap sentences
+            current_chunk = current_chunk[-overlap_sentences:] if overlap_sentences else []
+            current_size = sum(len(s) for s in current_chunk)
+
+        current_chunk.append(sentence)
+        current_size += sentence_size
+
+    if current_chunk:
+        chunks.append(" ".join(current_chunk))
+
+    return chunks
+```
+
+---
+
+## Semantic Chunking
+
+```python
+import numpy as np
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+
+class SemanticChunker:
+    """Chunk based on semantic similarity between sentences."""
+
+    def __init__(
+        self,
+        model_name: str = "all-MiniLM-L6-v2",
+        similarity_threshold: float = 0.5,
+        min_chunk_size: int = 100,
+        max_chunk_size: int = 1500
+    ):
+        self.model = SentenceTransformer(model_name)
+        self.similarity_threshold = similarity_threshold
+        self.min_chunk_size = min_chunk_size
+        self.max_chunk_size = max_chunk_size
+
+    def chunk(self, text: str) -> list[str]:
+        """Split text at semantic boundaries."""
+        # Split into sentences
+        sentences = self._split_sentences(text)
+        if len(sentences) <= 1:
+            return [text]
+
+        # Get embeddings
+        embeddings = self.model.encode(sentences)
+
+        # Find breakpoints based on similarity drops
+        breakpoints = self._find_breakpoints(embeddings)
+
+        # Create chunks
+        chunks = []
+        start = 0
+
+        for bp in breakpoints:
+            chunk_text = " ".join(sentences[start:bp])
+
+            # Handle size constraints
+            if len(chunk_text) > self.max_chunk_size:
+                # Split large chunks
+                sub_chunks = self._split_large_chunk(sentences[start:bp])
+                chunks.extend(sub_chunks)
+            elif len(chunk_text) >= self.min_chunk_size:
+                chunks.append(chunk_text)
+            elif chunks:
+                # Merge small chunk with previous
+                chunks[-1] += " " + chunk_text
+            else:
+                chunks.append(chunk_text)
+
+            start = bp
+
+        # Handle remaining sentences
+        if start < len(sentences):
+            remaining = " ".join(sentences[start:])
+            if chunks and len(remaining) < self.min_chunk_size:
+                chunks[-1] += " " + remaining
+            else:
+                chunks.append(remaining)
+
+        return chunks
+
+    def _split_sentences(self, text: str) -> list[str]:
+        """Split text into sentences."""
+        import re
+        sentences = re.split(r'(?<=[.!?])\s+', text)
+        return [s.strip() for s in sentences if s.strip()]
+
+    def _find_breakpoints(self, embeddings: np.ndarray) -> list[int]:
+        """Find semantic breakpoints using similarity drops."""
+        breakpoints = []
+
+        for i in range(1, len(embeddings)):
+            similarity = cosine_similarity(
+                embeddings[i-1:i],
+                embeddings[i:i+1]
+            )[0][0]
+
+            if similarity < self.similarity_threshold:
+                breakpoints.append(i)
+
+        return breakpoints
+
+    def _split_large_chunk(self, sentences: list[str]) -> list[str]:
+        """Split oversized chunk at midpoint."""
+        mid = len(sentences) // 2
+        return [
+            " ".join(sentences[:mid]),
+            " ".join(sentences[mid:])
+        ]
+
+# Usage
+chunker = SemanticChunker(
+    similarity_threshold=0.5,
+    min_chunk_size=200,
+    max_chunk_size=1000
+)
+semantic_chunks = chunker.chunk(document_text)
+```
+
+### Percentile-Based Breakpoints
+
+```python
+def find_breakpoints_percentile(
+    embeddings: np.ndarray,
+    percentile: int = 25
+) -> list[int]:
+    """Find breakpoints at similarity drops below percentile threshold."""
+    similarities = []
+
+    for i in range(1, len(embeddings)):
+        sim = cosine_similarity(
+            embeddings[i-1:i],
+            embeddings[i:i+1]
+        )[0][0]
+        similarities.append((i, sim))
+
+    # Dynamic threshold based on distribution
+    sim_values = [s[1] for s in similarities]
+    threshold = np.percentile(sim_values, percentile)
+
+    return [i for i, sim in similarities if sim < threshold]
+```
+
+---
+
+## Document-Aware Chunking
+
+### Markdown Chunking
+
+```python
+import re
+from dataclasses import dataclass
+
+@dataclass
+class MarkdownChunk:
+    text: str
+    heading: str | None
+    heading_level: int
+    metadata: dict
+
+def chunk_markdown(
+    text: str,
+    max_chunk_size: int = 1500,
+    include_heading_in_chunk: bool = True
+) -> list[MarkdownChunk]:
+    """Chunk markdown by headers while respecting structure."""
+    # Pattern to match headers
+    header_pattern = r'^(#{1,6})\s+(.+)$'
+
+    lines = text.split('\n')
+    chunks = []
+    current_chunk_lines = []
+    current_heading = None
+    current_level = 0
+    heading_stack = []  # For breadcrumb context
+
+    for line in lines:
+        header_match = re.match(header_pattern, line)
+
+        if header_match:
+            # Save current chunk if exists
+            if current_chunk_lines:
+                chunk_text = '\n'.join(current_chunk_lines)
+                if len(chunk_text.strip()) > 0:
+                    prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
+                    chunks.append(MarkdownChunk(
+                        text=prefix + chunk_text,
+                        heading=current_heading,
+                        heading_level=current_level,
+                        metadata={"breadcrumb": " > ".join(heading_stack)}
+                    ))
+
+            # Update heading context
+            level = len(header_match.group(1))
+            heading = header_match.group(2).strip()
+
+            # Maintain heading stack for breadcrumbs
+            while heading_stack and current_level >= level:
+                heading_stack.pop()
+                current_level -= 1
+
+            heading_stack.append(heading)
+            current_heading = heading
+            current_level = level
+            current_chunk_lines = []
+
+        else:
+            current_chunk_lines.append(line)
+
+            # Check chunk size
+            current_text = '\n'.join(current_chunk_lines)
+            if len(current_text) > max_chunk_size:
+                # Split at paragraph boundary
+                paragraphs = current_text.split('\n\n')
+                if len(paragraphs) > 1:
+                    split_point = len('\n\n'.join(paragraphs[:-1]))
+                    chunk_text = current_text[:split_point]
+                    prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
+                    chunks.append(MarkdownChunk(
+                        text=prefix + chunk_text,
+                        heading=current_heading,
+                        heading_level=current_level,
+                        metadata={"breadcrumb": " > ".join(heading_stack)}
+                    ))
+                    current_chunk_lines = [current_text[split_point:].strip()]
+
+    # Don't forget the last chunk
+    if current_chunk_lines:
+        chunk_text = '\n'.join(current_chunk_lines)
+        if len(chunk_text.strip()) > 0:
+            prefix = f"# {current_heading}\n\n" if include_heading_in_chunk and current_heading else ""
+            chunks.append(MarkdownChunk(
+                text=prefix + chunk_text,
+                heading=current_heading,
+                heading_level=current_level,
+                metadata={"breadcrumb": " > ".join(heading_stack)}
+            ))
+
+    return chunks
+```
+
+### Code-Aware Chunking
+
+```python
+import re
+from dataclasses import dataclass
+
+@dataclass
+class CodeChunk:
+    text: str
+    language: str | None
+    chunk_type: str  # "code", "text", "mixed"
+
+def chunk_with_code_blocks(
+    text: str,
+    max_chunk_size: int = 1500
+) -> list[CodeChunk]:
+    """Chunk text while keeping code blocks intact."""
+    # Pattern to match code blocks
+    code_block_pattern = r'```(\w+)?\n(.*?)```'
+
+    chunks = []
+    last_end = 0
+
+    for match in re.finditer(code_block_pattern, text, re.DOTALL):
+        # Text before code block
+        text_before = text[last_end:match.start()].strip()
+        if text_before:
+            # Chunk the text portion
+            text_chunks = recursive_chunk(text_before, max_chunk_size)
+            chunks.extend([
+                CodeChunk(text=t, language=None, chunk_type="text")
+                for t in text_chunks
+            ])
+
+        # Code block (keep intact if possible)
+        language = match.group(1)
+        code_content = match.group(2)
+        full_block = match.group(0)
+
+        if len(full_block) <= max_chunk_size:
+            chunks.append(CodeChunk(
+                text=full_block,
+                language=language,
+                chunk_type="code"
+            ))
+        else:
+            # Split large code blocks by function/class
+            code_chunks = split_code_block(code_content, language, max_chunk_size)
+            chunks.extend(code_chunks)
+
+        last_end = match.end()
+
+    # Remaining text after last code block
+    remaining = text[last_end:].strip()
+    if remaining:
+        text_chunks = recursive_chunk(remaining, max_chunk_size)
+        chunks.extend([
+            CodeChunk(text=t, language=None, chunk_type="text")
+            for t in text_chunks
+        ])
+
+    return chunks
+
+def split_code_block(code: str, language: str, max_size: int) -> list[CodeChunk]:
+    """Split code block at logical boundaries."""
+    # Simple function/class boundary splitting for Python
+    if language == "python":
+        pattern = r'\n(?=def |class |async def )'
+    elif language in ["javascript", "typescript"]:
+        pattern = r'\n(?=function |class |const |export )'
+    else:
+        pattern = r'\n\n'
+
+    parts = re.split(pattern, code)
+    chunks = []
+    current = ""
+
+    for part in parts:
+        if len(current) + len(part) > max_size and current:
+            chunks.append(CodeChunk(
+                text=f"```{language}\n{current}```",
+                language=language,
+                chunk_type="code"
+            ))
+            current = part
+        else:
+            current += part
+
+    if current:
+        chunks.append(CodeChunk(
+            text=f"```{language}\n{current}```",
+            language=language,
+            chunk_type="code"
+        ))
+
+    return chunks
+```
+
+---
+
+## Contextual/Agentic Chunking
+
+```python
+from openai import OpenAI
+
+def contextual_chunk(
+    document: str,
+    max_chunk_size: int = 1500
+) -> list[dict]:
+    """Use LLM to add context to each chunk."""
+    # First, do structural chunking
+    base_chunks = recursive_chunk(document, max_chunk_size)
+
+    client = OpenAI()
+    contextualized_chunks = []
+
+    for chunk in base_chunks:
+        # Generate contextual summary
+        response = client.chat.completions.create(
+            model="gpt-4o-mini",
+            messages=[
+                {
+                    "role": "system",
+                    "content": """Provide a brief context for this document chunk.
+                    Include: what topic it covers, how it relates to the broader document,
+                    and key concepts mentioned. Keep it under 100 words."""
+                },
+                {
+                    "role": "user",
+                    "content": f"Document excerpt:\n\n{chunk}"
+                }
+            ],
+            max_tokens=150
+        )
+
+        context = response.choices[0].message.content
+
+        contextualized_chunks.append({
+            "text": chunk,
+            "context": context,
+            "text_with_context": f"Context: {context}\n\nContent: {chunk}"
+        })
+
+    return contextualized_chunks
+```
+
+### Propositions-Based Chunking
+
+```python
+def extract_propositions(text: str) -> list[str]:
+    """Extract atomic propositions from text using LLM."""
+    client = OpenAI()
+
+    response = client.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[
+            {
+                "role": "system",
+                "content": """Extract atomic propositions from the text.
+                Each proposition should:
+                - Be a single, complete fact
+                - Be self-contained (understandable without context)
+                - Include necessary entity references
+
+                Return as a JSON array of strings."""
+            },
+            {
+                "role": "user",
+                "content": text
+            }
+        ],
+        response_format={"type": "json_object"}
+    )
+
+    import json
+    result = json.loads(response.choices[0].message.content)
+    return result.get("propositions", [])
+
+# Usage: For very fine-grained retrieval
+propositions = extract_propositions(document_text)
+# Each proposition becomes its own retrievable unit
+```
+
+---
+
+## Late Chunking (for Long-Context Embeddings)
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+class LateChunker:
+    """
+    Late chunking: embed full document, then pool token embeddings into chunks.
+    Preserves full document context while creating retrievable chunks.
+    """
+
+    def __init__(self, model_name: str = "jinaai/jina-embeddings-v2-base-en"):
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        self.model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
+        self.model.eval()
+
+    def chunk_and_embed(
+        self,
+        text: str,
+        chunk_size: int = 512,
+        overlap: int = 64
+    ) -> list[dict]:
+        """
+        Embed full document, then create chunk embeddings via mean pooling.
+        """
+        # Tokenize full document
+        inputs = self.tokenizer(
+            text,
+            return_tensors="pt",
+            truncation=True,
+            max_length=8192  # Model's max context
+        )
+
+        # Get token-level embeddings
+        with torch.no_grad():
+            outputs = self.model(**inputs)
+            token_embeddings = outputs.last_hidden_state[0]  # [seq_len, hidden_dim]
+
+        # Get token-to-text mapping
+        tokens = self.tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
+
+        # Create chunks from token embeddings
+        chunks = []
+        seq_len = token_embeddings.shape[0]
+        start = 0
+
+        while start < seq_len:
+            end = min(start + chunk_size, seq_len)
+
+            # Mean pool token embeddings for this chunk
+            chunk_embedding = token_embeddings[start:end].mean(dim=0).numpy()
+
+            # Reconstruct text for this chunk
+            chunk_token_ids = inputs["input_ids"][0][start:end]
+            chunk_text = self.tokenizer.decode(chunk_token_ids, skip_special_tokens=True)
+
+            chunks.append({
+                "text": chunk_text,
+                "embedding": chunk_embedding,
+                "start_token": start,
+                "end_token": end
+            })
+
+            start = end - overlap
+
+        return chunks
+
+# Usage
+late_chunker = LateChunker()
+chunks_with_embeddings = late_chunker.chunk_and_embed(
+    long_document,
+    chunk_size=512,
+    overlap=64
+)
+```
+
+---
+
+## Metadata Enrichment
+
+```python
+from dataclasses import dataclass
+from datetime import datetime
+import hashlib
+
+@dataclass
+class EnrichedChunk:
+    text: str
+    embedding: list[float] | None
+    metadata: dict
+
+def enrich_chunk(
+    text: str,
+    source_file: str,
+    chunk_index: int,
+    total_chunks: int,
+    additional_metadata: dict | None = None
+) -> EnrichedChunk:
+    """Add comprehensive metadata to chunk."""
+    metadata = {
+        # Source tracking
+        "source": source_file,
+        "chunk_index": chunk_index,
+        "total_chunks": total_chunks,
+
+        # Content characteristics
+        "char_count": len(text),
+        "word_count": len(text.split()),
+        "content_hash": hashlib.md5(text.encode()).hexdigest()[:12],
+
+        # Temporal
+        "indexed_at": datetime.utcnow().isoformat(),
+
+        # Position context
+        "position": "start" if chunk_index == 0 else (
+            "end" if chunk_index == total_chunks - 1 else "middle"
+        )
+    }
+
+    if additional_metadata:
+        metadata.update(additional_metadata)
+
+    return EnrichedChunk(text=text, embedding=None, metadata=metadata)
+```
+
+---
+
+## Chunk Size Selection Guide
+
+| Document Type | Recommended Size | Overlap | Rationale |
+|--------------|------------------|---------|-----------|
+| FAQ/Q&A | 200-400 tokens | 20-50 | Keep Q&A pairs together |
+| Technical docs | 400-600 tokens | 50-100 | Balance context vs precision |
+| Legal/contracts | 600-800 tokens | 100-150 | Preserve clause context |
+| Code documentation | 300-500 tokens | 50-100 | Keep function docs together |
+| Chat transcripts | 150-300 tokens | 25-50 | Natural turn boundaries |
+| Research papers | 500-800 tokens | 100-200 | Section-level coherence |
+
+---
+
+## Quick Reference
+
+| Strategy | Use Case | Code Pattern |
+|----------|----------|--------------|
+| Fixed-size | Logs, baseline | `text[i:i+chunk_size]` |
+| Recursive | General text | Split by `["\n\n", "\n", ". "]` |
+| Sentence | Q&A content | `sent_tokenize()` + merge |
+| Semantic | Technical docs | Similarity-based breaks |
+| Markdown | Documentation | Header-aware splitting |
+| Late chunking | Long-context models | Embed full, pool chunks |
+
+## Related Skills
+
+- **RAG Architect** - Integration with vector databases
+- **Python Pro** - Preprocessing pipelines
+- **NLP Engineer** - Tokenization and text processing