rag-skills 1.0.0

package/skills/chunking/choosing-a-chunking-framework.md

@@ -0,0 +1,186 @@
+---
+title: "Choosing a Chunking Framework"
+description: "Select the right chunking framework based on document type, pipeline architecture, and retrieval goals."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "chunking"
+tags: ["framework-selection", "chonkie", "langchain", "llamaindex", "haystack", "unstructured"]
+---
+
+## Overview
+Chunking quality depends as much on the framework as on the strategy itself. This guide helps a coding agent choose between chunking frameworks based on the shape of the data, the surrounding RAG stack, and whether the main need is speed, structure-awareness, retrieval quality, or integration simplicity.
+
+## Problem Statement
+Teams often pick a chunking framework for the wrong reason:
+- They default to the framework already in the app, even when it has weaker chunking primitives for the data type
+- They choose a sophisticated semantic chunker for documents that only need simple deterministic splitting
+- They use generic text splitters for PDFs, tables, or code repositories where structure matters more than raw length
+- They optimize for chunking features without considering metadata propagation, ingestion workflow, or downstream retrieval
+
+## Key Concepts
+- **Framework Fit**: How well the chunking library matches the rest of the ingestion and retrieval stack
+- **Structure Awareness**: Whether the framework understands sections, headings, tables, pages, or code blocks
+- **Semantic Awareness**: Whether the framework can chunk by meaning rather than only size or delimiters
+- **Operational Simplicity**: How easy it is to adopt the framework without introducing a second ingestion system
+- **Chunking Depth**: Whether you need only fixed-size chunks or richer patterns like hierarchical, sentence-window, or code-aware chunking
+
+## Implementation Guide
+
+### Step 1: Identify the Dominant Document Type
+Decide whether the corpus is mostly plain text, structured documents, code, or messy extracted files.
+
+**Why**: Chunking frameworks differ most sharply in how they handle document structure. The right choice for markdown docs is often wrong for scanned PDFs or source code.
+
+Use this default mapping:
+- Plain text and generic prose: favor LangChain or Chonkie
+- Metadata-rich RAG pipelines: favor LlamaIndex
+- Pipeline-centric search applications: favor Haystack
+- Parsed PDFs, office files, and layout-heavy documents: favor Unstructured
+- Code-heavy repositories: favor Chonkie first, then LangChain if you need simpler integration
+
+### Step 2: Check Whether Chunking Is a Core Requirement or Just a Utility
+Determine whether chunking itself is a strategic part of the system or just one preprocessing step.
+
+**Why**: If chunking is central to recall quality, code structure, or high-throughput ingestion, you want a framework with deeper chunking specialization. If it is incidental, use the framework already governing the RAG pipeline.
+
+Choose this way:
+- If chunking sophistication is a competitive advantage, start with Chonkie
+- If chunking is just one component in a broader orchestration framework, prefer LangChain, LlamaIndex, or Haystack based on the app stack
+
+### Step 3: Match the Framework to the Retrieval Pattern
+Choose a framework whose chunking model supports the retrieval pattern you intend to use.
+
+**Why**: Some frameworks are built around plain chunks, while others are better for hierarchical retrieval, metadata-enriched nodes, or section-preserving document flows.
+
+Recommended fit by retrieval pattern:
+- Simple vector search over text: LangChain or Chonkie
+- Hierarchical retrieval and parent-child context: LlamaIndex or Haystack
+- Section-aware retrieval for reports and long docs: Unstructured or Chonkie recursive chunking
+- Code search and AST-aware chunking: Chonkie
+- Fine-grained sentence-level retrieval with context reconstruction: LlamaIndex
+
+### Step 4: Use the Framework Decision Matrix
+Pick the framework whose strengths best match the workload.
+
+**Why**: A framework should be chosen by tradeoff, not popularity.
+
+| Framework | Best At | Choose It When | Avoid It When |
+|----------|---------|----------------|---------------|
+| **[Chonkie](https://docs.chonkie.ai/oss/chunkers/overview)** | Specialized chunking strategies, high throughput, semantic and code-aware chunking | Chunking is a first-class problem, you need semantic/code chunkers, or ingestion speed matters | You want the simplest possible default inside an existing LangChain or Haystack app |
+| **[LangChain Text Splitters](https://docs.langchain.com/oss/python/integrations/splitters/index)** | Simple, reliable general-purpose chunking tightly integrated with LangChain apps | You already use LangChain or LangGraph and need practical default chunking with minimal extra tooling | You need deeply specialized code, layout, or semantic chunking beyond standard splitter patterns |
+| **[LlamaIndex Node Parsers](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/)** | Metadata-rich nodes, sentence windows, hierarchical parsing, retrieval-aware chunking | You use LlamaIndex ingestion/query pipelines or need chunking that preserves node relationships and retrieval metadata | You only need straightforward standalone chunking without adopting LlamaIndex concepts |
+| **[Haystack Preprocessors](https://docs.haystack.deepset.ai/docs/documentsplitter)** | Deterministic pipeline-based splitting for production search systems | You are building around Haystack pipelines and want predictable document preprocessing components | You need more advanced semantic or code-aware chunking than Haystack’s built-in preprocessors provide |
+| **[Unstructured Chunking](https://docs.unstructured.io/open-source/core-functionality/chunking)** | Partition-first chunking for PDFs, Office docs, HTML, tables, and layout-heavy content | Your main challenge is document parsing and structural preservation before chunking | Your corpus is already clean text and you do not need layout-aware parsing |
+
+### Step 5: Apply Framework-Specific Guidance
+Use the framework only in the situations where it has a clear advantage.
+
+**Why**: Each framework has a distinct operating model. Mixing them without a reason usually adds complexity.
+
+#### Chonkie
+Use [Chonkie](https://www.chonkie.ai/) when chunking itself is the focus of the system.
+
+Strong fit:
+- You need a dedicated chunking library rather than a general LLM framework
+- You want to choose among token, sentence, recursive, semantic, late, fast, or code chunkers
+- You have code repositories or technical docs where chunk coherence matters
+- You have high-throughput ingestion where fast chunking is important
+
+Especially relevant docs:
+- [Chunkers overview](https://docs.chonkie.ai/oss/chunkers/overview)
+- [RecursiveChunker](https://docs.chonkie.ai/oss/chunkers/recursive-chunker)
+- [SemanticChunker](https://docs.chonkie.ai/oss/chunkers/semantic-chunker)
+- [CodeChunker](https://docs.chonkie.ai/oss/chunkers/code-chunker)
+- [FastChunker](https://docs.chonkie.ai/oss/chunkers/fast-chunker)
+
+Do not choose Chonkie just because it has more chunkers. Choose it when those chunkers materially improve your corpus handling.
+
+#### LangChain
+Use [LangChain Text Splitters](https://docs.langchain.com/oss/python/integrations/splitters/index) when you want a practical default inside a LangChain or LangGraph application.
+
+Strong fit:
+- You already use LangChain loaders, embeddings, retrievers, or agents
+- You want an opinionated default like `RecursiveCharacterTextSplitter`
+- You need markdown, JSON, HTML, or code splitting without introducing another ingestion framework
+- You value integration speed over chunking specialization
+
+Especially relevant docs:
+- [Text splitters overview](https://docs.langchain.com/oss/python/integrations/splitters/index)
+- [RecursiveCharacterTextSplitter](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter)
+- [CharacterTextSplitter](https://docs.langchain.com/oss/python/integrations/splitters/character_text_splitter)
+
+Do not choose LangChain if you need the strongest code-aware or layout-aware chunking and are willing to use a more specialized tool.
+
+#### LlamaIndex
+Use [LlamaIndex Node Parsers](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/) when chunking is tightly coupled to retrieval behavior and metadata-rich nodes.
+
+Strong fit:
+- You want sentence-window retrieval, node relationships, or hierarchical retrieval patterns
+- You care about chunk metadata because retrieval and postprocessing depend on it
+- You are already using LlamaIndex ingestion pipelines, retrievers, and node postprocessors
+- You want chunking that fits directly into a retrieval-aware framework
+
+Especially relevant docs:
+- [Node parser usage](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/)
+- [HierarchicalNodeParser](https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/hierarchical/)
+- [SentenceWindowNodeParser](https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/sentence_window/)
+
+Do not choose LlamaIndex if you only need a lightweight chunker and do not want its document and node abstractions in the pipeline.
+
+#### Haystack
+Use [Haystack preprocessors](https://docs.haystack.deepset.ai/docs/documentsplitter) when the application is built as a search or RAG pipeline and you want deterministic preprocessing components.
+
+Strong fit:
+- You use Haystack pipelines end-to-end
+- You need sentence, passage, page, line, or function splitting in a production-oriented indexing flow
+- You want hierarchical document splitting without adopting a different ingestion framework
+- You value stable preprocessing stages more than cutting-edge chunking algorithms
+
+Especially relevant docs:
+- [DocumentSplitter](https://docs.haystack.deepset.ai/docs/documentsplitter)
+- [HierarchicalDocumentSplitter](https://docs.haystack.deepset.ai/docs/hierarchicaldocumentsplitter)
+- [DocumentPreprocessor](https://docs.haystack.deepset.ai/docs/documentpreprocessor)
+
+Do not choose Haystack for advanced semantic chunking if you are not otherwise using Haystack.
+
+#### Unstructured
+Use [Unstructured chunking](https://docs.unstructured.io/open-source/core-functionality/chunking) when the hard problem is not text splitting but extracting meaningful document elements first.
+
+Strong fit:
+- Your corpus includes PDFs, PowerPoints, HTML, tables, page layouts, and other messy enterprise documents
+- You want chunking to follow structural elements created during partitioning
+- You need section-preserving chunking such as `by_title`
+- You care about table isolation, page boundaries, or composite document elements
+
+Especially relevant docs:
+- [Open-source chunking](https://docs.unstructured.io/open-source/core-functionality/chunking)
+- [Platform API chunking strategies](https://docs.unstructured.io/platform-api/partition-api/chunking)
+
+Do not choose Unstructured when the documents are already clean plain text and layout parsing adds no value.
+
+## When to Use This Skill
+- When deciding which chunking framework a coding agent should adopt for a new RAG pipeline
+- When replacing ad hoc chunking logic with a framework-backed approach
+- When the team needs a framework-level recommendation before tuning chunk size or overlap
+- When the corpus mixes prose, code, and structured enterprise documents
+
+## When NOT to Use This Skill
+- When the framework is already fixed by platform constraints
+- When the problem is chunk sizing rather than framework choice
+- When you only need one simple deterministic splitter and framework selection is not material
+
+## Related Skills
+- [Semantic Chunking](./semantic-chunking.md) - Meaning-aware chunk boundaries
+- [Hierarchical Chunking](./hierarchical-chunking.md) - Multi-level chunk structures
+- [Sliding Window Chunking](./sliding-window-chunking.md) - Overlap-based chunking
+- [RAG for Code Documentation](../data-type-handling/rag-for-code-documentation.md) - Code-heavy corpus guidance
+
+## Metrics & Success Criteria
+- **Framework Fit**: The chosen framework matches the dominant document and retrieval pattern
+- **Operational Simplicity**: The framework does not introduce unnecessary ingestion complexity
+- **Retrieval Quality**: Chunking improves recall and answer quality for the actual corpus
+- **Maintainability**: The chunking approach is understandable and sustainable for the team
+- **Migration Cost**: The framework choice does not create avoidable lock-in without clear upside

package/skills/chunking/contextual-chunk-headers.md

@@ -0,0 +1,106 @@
+---
+title: "Contextual Chunk Headers"
+description: "Add higher-level context to chunks by prepending headers before embedding for better retrieval."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "chunking"
+tags: ["contextual-headers", "metadata", "chunk-enhancement", "document-structure"]
+---
+
+## Overview
+Contextual chunk headers (CCH) enhance retrieval by prepending higher-level context (document title, section headers, summaries) to each chunk before embedding. This gives embeddings a more accurate representation of content and meaning, significantly improving retrieval quality and reducing irrelevant results.
+
+## Problem Statement
+Individual chunks often lack sufficient context:
+- **Implicit References**: Chunks use pronouns and implicit references that aren't clear in isolation
+- **Missing Context**: Chunks only make sense in the context of their section or document
+- **Misleading Isolation**: Read alone, chunks can be misleading or incomplete
+- **Poor Retrieval**: Queries that reference subjects or themes aren't matched effectively
+- **LLM Misinterpretation**: LLMs struggle to understand decontextualized chunks
+
+## Key Concepts
+- **Chunk Header**: Higher-level context prepended to chunk content
+- **Document-Level Context**: Document title or summary as header context
+- **Section-Level Context**: Hierarchy of section and subsection titles
+- **Pre-Embedding Concatenation**: Combining header and chunk before embedding
+- **Context Preservation**: Carrying headers through to retrieval and presentation
+
+## Implementation Guide
+
+### Step 1: Generate Document-Level Context
+Extract or generate document titles and summaries for use in chunk headers.
+
+**Why**: Document-level context provides the most important higher-level information for understanding any chunk within that document.
+
+Use an LLM to generate a descriptive document title if one doesn't exist, or extract titles from document structure.
+
+### Step 2: Extract Section Hierarchy
+Parse document structure to capture section and subsection titles.
+
+**Why**: Section hierarchy provides granular context that helps retrieve information about specific topics and themes within the document.
+
+Parse the document structure (markdown headings, HTML tags, etc.) to build a hierarchical representation of sections.
+
+### Step 3: Create Chunk Headers
+Combine document and section context into comprehensive headers.
+
+**Why**: Combined context provides both broad document understanding and specific section information, giving embeddings the best possible representation.
+
+For each chunk, create a header combining document title and the section path that contains the chunk.
+
+### Step 4: Prepend Headers to Chunks
+Combine headers with chunk content before embedding.
+
+**Why**: Embeddings represent the combined header+chunk content, capturing both local meaning and global context in the same vector space.
+
+When indexing each chunk, include the header as a prefix to the chunk content.
+
+### Step 5: Embed with Headers
+Use the header-enhanced chunks for embedding and indexing.
+
+**Why**: Embedding with headers ensures that retrieval considers both local content and broader context, leading to more relevant matches.
+
+Index the enhanced chunks using your preferred embedding model and vector database.
+
+### Step 6: Present Results with Headers
+Include headers when presenting retrieved chunks to users or LLMs.
+
+**Why**: Presenting headers provides necessary context for understanding retrieved chunks, reducing misinterpretation and improving answer quality.
+
+When returning retrieved content, include the original header for context.
+
+### Step 7: Build Complete CCH Pipeline
+Combine all components into a cohesive chunking and retrieval system.
+
+**Why**: A complete pipeline ensures that contextual headers are consistently applied through chunking, embedding, retrieval, and presentation.
+
+For implementation patterns, see [LangChain RecursiveCharacterTextSplitter](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter), [LlamaIndex node parser usage](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/), [Anthropic Contextual Retrieval](https://www.anthropic.com/engineering/contextual-retrieval), and [dsRAG AutoContext](https://github.com/D-Star-AI/dsRAG).
+
+## When to Use This Skill
+- Building RAG systems with structured documents (articles, reports, documentation)
+- When chunks frequently refer to their subject via pronouns or implicit references
+- For documents with clear section hierarchy or structure
+- When retrieval quality suffers from decontextualized chunks
+- For applications where LLMs need to understand chunk context
+
+## When NOT to Use This Skill
+- For unstructured documents without clear hierarchy or titles
+- When chunk headers would be redundant or too long
+- For very short documents where the entire text is already coherent
+- When storage overhead is a major concern (headers increase chunk size)
+- For domains where document structure is meaningless (code snippets, logs)
+
+## Related Skills
+- [Semantic Chunking](./semantic-chunking.md) - Context-aware chunking approach
+- [Hierarchical Chunking](./hierarchical-chunking.md) - Structure-based chunking
+- [Context Enrichment Window](../retrieval-strategies/context-enrichment-window.md) - Post-retrieval context
+
+## Metrics & Success Criteria
+- **Retrieval Precision**: Increased rate of correct retrieval for subject-based queries
+- **Context Completeness**: Retrieved chunks provide sufficient context for understanding
+- **Header Effectiveness**: Headers improve embedding representation quality
+- **Answer Quality**: Reduced hallucinations from decontextualized chunks
+- **Storage Overhead**: Acceptable increase in chunk size due to headers

package/skills/chunking/hierarchical-chunking.md

@@ -0,0 +1,77 @@
+---
+title: "Hierarchical Chunking"
+description: "Chunk nested documents into parent-child levels so retrieval can move from broad sections to fine-grained passages."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "chunking"
+tags: ["nested", "multi-level", "document-structure", "parent-child"]
+---
+
+## Overview
+Hierarchical chunking creates multi-level chunk structures that preserve document hierarchies (chapters, sections, subsections). This enables both broad-overview retrieval (high-level chunks) and detailed retrieval (low-level chunks) with parent-child relationships for context propagation.
+
+## Problem Statement
+Flat chunking strategies lose the structural relationships within documents:
+- Questions about broad topics retrieve detailed fragments instead of summaries
+- No way to retrieve "context for context" - when a detail is retrieved, its containing section is lost
+- Navigation through document hierarchies becomes impossible
+- Retrieval results can't be organized by document structure
+
+## Key Concepts
+- **Parent-Child Relationships**: Links between summary chunks and their detailed sub-chunks
+- **Multi-Level Granularity**: Chunks at different abstraction levels (document → chapter → section → paragraph)
+- **Context Propagation**: Ability to include parent context when retrieving child chunks
+- **Metadata Enrichment**: Storing hierarchy information for filtering and navigation
+- **Recursive Chunking**: Applying chunking rules recursively through document structure
+
+## Implementation Guide
+
+### Step 1: Parse Document Structure
+Extract the hierarchical structure of your documents.
+
+**Why**: Proper parsing is foundational—you can't create a hierarchy without understanding the structure.
+
+### Step 2: Create Multi-Level Chunks
+Generate chunks at multiple levels from the parsed hierarchy.
+
+**Why**: Multiple chunk levels allow different retrieval strategies—summary chunks for broad questions, detailed chunks for specifics.
+
+### Step 3: Store Hierarchy Metadata
+Persist hierarchy information for retrieval-time context propagation.
+
+**Why**: Metadata enables filtering and hierarchical queries (e.g., "retrieve from level 2 or deeper").
+
+### Step 4: Implement Hierarchical Retrieval
+Retrieve chunks with optional parent context.
+
+**Why**: Hierarchical retrieval allows users to "zoom in" from broad topics to specific details, just like browsing a table of contents.
+
+For implementation details, see the [LlamaIndex HierarchicalNodeParser](https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/hierarchical/), [LangChain RecursiveCharacterTextSplitter](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter), [Haystack HierarchicalDocumentSplitter](https://docs.haystack.deepset.ai/docs/hierarchicaldocumentsplitter), and [LlamaIndex Recursive Retriever + Node References](https://developers.llamaindex.ai/python/framework/integrations/retrievers/recursive_retriever_nodes/).
+
+## When to Use This Skill
+- Long-form documents with clear structure (books, technical documentation, research papers)
+- When you need both summary and detailed retrieval
+- For applications that support "drill-down" navigation
+- When building document browsing/search interfaces
+- For Q&A systems that benefit from hierarchical context
+
+## When NOT to Use This Skill
+- Flat content without structure (blog posts, single-page articles)
+- When storage complexity outweighs retrieval benefits
+- For small document collections where simple chunking suffices
+- When retrieval latency is critical (hierarchical retrieval adds overhead)
+
+## Related Skills
+- [Semantic Chunking](./semantic-chunking.md) - For content-aware chunking
+- [Sliding Window Chunking](./sliding-window-chunking.md) - For context overlap
+- [Multi-Pass Retrieval with Reranking](../retrieval-strategies/multi-pass-retrieval-with-reranking.md) - For refining hierarchical results
+
+## Metrics & Success Criteria
+- **Coverage**: All document content preserved across hierarchy levels
+- **Retrieval Flexibility**: Can retrieve at any abstraction level
+- **Context Relevance**: Parent context improves answer quality for detailed questions
+- **Storage Efficiency**: Metadata overhead doesn't exceed 30% of storage
+- **Query Latency**: Hierarchical retrieval within 2x of flat retrieval

package/skills/chunking/semantic-chunking.md

@@ -0,0 +1,78 @@
+---
+title: "Semantic Chunking"
+description: "Use semantic boundaries and embedding similarity to chunk text for higher-relevance retrieval."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "chunking"
+tags: ["semantic", "nlp", "sentence-boundary", "context-preservation"]
+---
+
+## Overview
+Semantic chunking divides documents into segments based on natural language boundaries and semantic meaning rather than fixed character counts. This approach preserves context integrity and improves retrieval relevance by keeping related ideas together.
+
+## Problem Statement
+Fixed-size chunking (e.g., 1000 tokens) often cuts through mid-sentence, mid-paragraph, or mid-concept, resulting in:
+- Fragments that lose semantic meaning
+- Context bleeding across unrelated chunks
+- Poor retrieval relevance for semantic search
+- Difficulty for LLMs to understand isolated fragments
+
+## Key Concepts
+- **Semantic Boundaries**: Natural break points in text (paragraphs, sections, sentences)
+- **Context Preservation**: Keeping related ideas and arguments together
+- **Sentence Boundary Detection**: Using NLP tools to identify proper sentence ends
+- **Paragraph Cohesion**: Identifying when a new thought or argument begins
+- **Overlapping Windows**: Small overlaps to preserve context between adjacent chunks
+
+## Implementation Guide
+
+### Step 1: Detect Sentence Boundaries
+Use sentence boundary detection rather than simple period splitting. This handles abbreviations, decimal numbers, and other edge cases.
+
+**Why**: Simple period/regex splitting fails on "Dr. Smith", "3.14", "U.S.A.", etc.
+
+### Step 2: Group Sentences into Semantic Units
+Group sentences into chunks based on semantic cohesion rather than fixed counts.
+
+**Why**: Similarity-based chunking adapts to document density—dense technical sections get smaller chunks, narrative sections get larger ones.
+
+### Step 3: Apply Structure-Based Chunking
+Leverage document structure (headings, markdown, HTML tags) to identify natural sections.
+
+**Why**: Document structure often encodes semantic boundaries (chapters, sections, subsections).
+
+### Step 4: Implement Adaptive Sizing
+Adjust chunk sizes based on content characteristics (code blocks, lists, tables).
+
+**Why**: Different content types need different chunking strategies for optimal retrieval.
+
+For practical implementations, compare [Sentence-BERT](https://arxiv.org/abs/1908.10084), [LlamaIndex Semantic Chunker](https://docs.llamaindex.ai/en/stable/examples/node_parsers/semantic_chunking/), [LangChain RecursiveCharacterTextSplitter](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter), and [spaCy sentence segmentation](https://spacy.io/usage/linguistic-features#sentence-segmentation).
+
+## When to Use This Skill
+- Documents with clear semantic structure (articles, reports, documentation)
+- When retrieval quality is more important than fixed-size guarantees
+- For narrative or explanatory content where context matters
+- When using semantic search (vector embeddings) for retrieval
+- For Q&A systems that need coherent context for answers
+
+## When NOT to Use This Skill
+- Code repositories where line-level precision matters
+- Log files or structured data with fixed formats
+- When you need exact token count guarantees (e.g., for API limits)
+- Streaming data where chunk boundaries can't be deferred
+- When using exact keyword matching (BM25) where chunk size matters less
+
+## Related Skills
+- [Hierarchical Chunking](./hierarchical-chunking.md) - For nested document structures
+- [Sliding Window Chunking](./sliding-window-chunking.md) - For overlapping context preservation
+- [Hybrid Search BM25 Dense](../retrieval-strategies/hybrid-search-bm25-dense.md) - Combining search methods
+
+## Metrics & Success Criteria
+- **Retrieval Relevance**: Retrieved chunks should contain complete, self-contained information
+- **Context Preservation**: Related concepts remain in same chunk
+- **Coverage**: No significant information lost in chunk boundaries
+- **Token Efficiency**: Chunks balanced between too small (lossy) and too large (noisy)
+- **Semantic Coherence**: Chunks represent coherent thoughts or sections

package/skills/chunking/sliding-window-chunking.md

@@ -0,0 +1,82 @@
+---
+title: "Sliding Window Chunking"
+description: "Use overlapping windows to preserve context across chunk boundaries while controlling retrieval size."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "chunking"
+tags: ["overlap", "context-preservation", "window", "boundary"]
+---
+
+## Overview
+Sliding window chunking creates overlapping chunks where each chunk shares content with adjacent chunks. This preserves context at chunk boundaries, ensuring that information split across chunks remains accessible through multiple retrieval paths.
+
+## Problem Statement
+Non-overlapping chunking creates hard boundaries that can break important context:
+- Critical information appears exactly at a chunk boundary
+- Concepts spanning multiple chunks become fragmented
+- Retrieval may miss relevant content because it's split across boundaries
+- LLMs lose context when only one fragment is provided
+
+## Key Concepts
+- **Overlap Percentage**: The proportion of shared content between adjacent chunks (typically 10-25%)
+- **Window Size**: The total size of each chunk including overlap
+- **Stride**: The distance between chunk start points (window size - overlap)
+- **Context Preservation**: Ensuring relevant context exists on both sides of boundaries
+- **Multiple Retrieval Paths**: Same content accessible from different chunks
+
+## Implementation Guide
+
+### Step 1: Determine Overlap Parameters
+Calculate appropriate overlap based on your use case and chunk size.
+
+**Why**: The overlap ratio determines how much context is preserved—too little loses context, too much creates redundancy.
+
+### Step 2: Implement Sliding Window Chunking
+Create chunks with configurable overlap.
+
+**Why**: Boundary adjustment prevents chunks from ending mid-sentence while maintaining the sliding window pattern.
+
+### Step 3: Add Chunk Metadata
+Track overlap information for retrieval-time decisions.
+
+**Why**: Metadata allows retrieval systems to identify and deduplicate overlapping content.
+
+### Step 4: Implement Smart Retrieval
+Handle overlapping results intelligently.
+
+**Why**: Smart retrieval prevents redundant information while preserving the benefits of overlapping chunks.
+
+### Step 5: Token-Aware Sliding Window
+Implement sliding window based on tokens rather than characters.
+
+**Why**: Token-aware chunking is more accurate for LLMs and respects their actual token limits.
+
+Useful implementations include [LangChain RecursiveCharacterTextSplitter](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter), the [OpenAI tiktoken cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb), [tiktoken](https://github.com/openai/tiktoken), and [Sentence Transformers rerankers](https://www.sbert.net/examples/cross_encoder/training/rerankers/README.html).
+
+## When to Use This Skill
+- Documents where context at boundaries is important
+- When using semantic search where retrieval might match content near boundaries
+- For narrative or sequential content where flow matters
+- When building systems that benefit from multiple retrieval paths
+- For technical documentation where definitions/examples might span chunks
+
+## When NOT to Use This Skill
+- When storage costs are critical (overlap increases storage 20-30%)
+- When exact deduplication is required (overlap makes this harder)
+- For structured data with clear boundaries (JSON, CSV)
+- When using exact keyword search where overlap doesn't help
+
+## Related Skills
+- [Semantic Chunking](./semantic-chunking.md) - For content-aware boundaries
+- [Hierarchical Chunking](./hierarchical-chunking.md) - For nested structures
+- [Hybrid Search BM25 Dense](../retrieval-strategies/hybrid-search-bm25-dense.md) - For combining search methods
+
+## Metrics & Success Criteria
+- **Boundary Coverage**: Critical terms appear in at least one chunk (not lost at boundaries)
+- **Storage Overhead**: Overlap increases storage by expected ratio (e.g., 20%)
+- **Retrieval Recall**: Same content accessible through multiple chunks
+- **Context Preservation**: Retrieved chunks contain sufficient surrounding context
+- **Deduplication Quality**: Can identify and handle overlapping content effectively

package/skills/data-type-handling/rag-for-code-documentation.md

@@ -0,0 +1,83 @@
+---
+title: "RAG for Code Documentation"
+description: "Handle code-aware retrieval by preserving symbols, file structure, and API context."
+allowed-tools:
+  - Read
+  - Grep
+  - Glob
+  - Bash
+category: "data-type-handling"
+tags: ["code", "programming", "syntax", "api", "documentation"]
+---
+
+## Overview
+RAG for code documentation requires specialized handling due to code's structured nature, syntax-specific patterns, and the importance of preserving function signatures, imports, and contextual relationships. This skill covers embedding code snippets, handling API references, and retrieving code-aware context.
+
+## Problem Statement
+Generic RAG approaches struggle with code-related queries:
+- Standard chunking breaks function/class structures mid-signature
+- Language models don't understand code syntax and semantics as well as natural language
+- Code requires preserving structural relationships (imports, dependencies, inheritance)
+- API queries need exact matches alongside semantic understanding
+- Code examples need context to be useful (imports, dependencies)
+
+## Key Concepts
+- **Code-Aware Chunking**: Preserving function/class boundaries and syntactic units
+- **Code-Specific Embeddings**: Models trained on code (CodeBERT, StarCoder) vs. general embeddings
+- **Syntax Preservation**: Maintaining formatting, indentation, and structure
+- **Context Tracking**: Following import chains and dependency relationships
+- **Hybrid Search for Code**: Combining semantic search with AST-based matching
+
+## Implementation Guide
+
+### Step 1: Code-Aware Document Parsing
+Parse code files into structured chunks that preserve syntactic units.
+
+**Why**: AST-based parsing preserves code structure that would be lost with generic text chunking.
+
+### Step 2: Code-Specific Embedding
+Use embeddings designed for code or augment text embeddings with code-aware features.
+
+**Why**: Code-specific embeddings capture semantic meaning in code (function relationships, patterns) that general embeddings miss.
+
+### Step 2.5: Store Code Chunks in Vector DB
+Store code chunks with rich metadata for effective retrieval.
+
+**Why**: Rich metadata enables filtering by language, type, file path, and other code-specific attributes.
+
+### Step 3: Implement Code-Aware Search
+Search with code-specific considerations.
+
+**Why**: Code-aware search combines semantic understanding with precise filtering for programming-specific use cases.
+
+### Step 4: Handle API Documentation
+Specialized handling for API reference queries.
+
+**Why**: API queries often require exact matching (function/class names) rather than pure semantic search.
+
+Practical references include the [CodeBERT paper](https://huggingface.co/papers/2002.08155), [GraphCodeBERT paper](https://huggingface.co/papers/2009.08366), [StarCoder paper](https://huggingface.co/papers/2305.06161), and [GitHub code search](https://github.com/features/code-search).
+
+## When to Use This Skill
+- Building code search or documentation assistants
+- When users query code syntax, APIs, or programming concepts
+- For IDE integrations and code completion systems
+- When building technical documentation with code examples
+- For troubleshooting programming queries
+
+## When NOT to Use This Skill
+- For natural language-only content (use general RAG)
+- When code structure isn't important for retrieval
+- For very simple codebases where grep suffices
+- When embedding code-specific models is not feasible
+
+## Related Skills
+- [Semantic Chunking](../chunking/semantic-chunking.md) - For document chunks
+- [Choosing Vector DB by Datatype](../vector-databases/choosing-vector-db-by-datatype.md) - Database selection
+- [Hierarchical Chunking](../chunking/hierarchical-chunking.md) - For nested code structures
+
+## Metrics & Success Criteria
+- **Code Retrieval Accuracy**: Exact matches for function/class names > 90%
+- **Semantic Understanding**: NDCG@10 for code-related queries > 0.7
+- **Context Preservation**: Relevant imports included > 80% of time
+- **Syntax Preservation**: Code formatting maintained 100%
+- **Query Latency**: < 200ms for typical code queries