llm-docs-builder 0.6.0 → 0.8.0

data/README.md

@@ -5,34 +5,21 @@
 
 **Optimize your documentation for LLMs and RAG systems. Reduce token consumption by 67-95%.**
 
-llm-docs-builder normalizes markdown documentation to be AI-friendly and generates llms.txt files. Transform relative links to absolute URLs, measure token savings when serving markdown vs HTML, and create standardized documentation indexes that help LLMs navigate your project.
+llm-docs-builder transforms markdown documentation to be AI-friendly and generates llms.txt files. It normalizes links, removes unnecessary content, optimizes documents for LLM context windows, and enhances documents for RAG retrieval with hierarchical heading context and metadata.
 
 ## The Problem
 
 When LLMs fetch documentation, they typically get HTML pages designed for humans - complete with navigation bars, footers, JavaScript, CSS, and other overhead. This wastes 70-90% of your context window on content that doesn't help answer questions.
 
 **Real example from Karafka documentation:**
-- Human HTML version: 82.0 KB (~20,500 tokens)
-- AI markdown version: 4.1 KB (~1,025 tokens)
-- **Result: 95% reduction, 19,475 tokens saved, 20x smaller**
-
-With GPT-4's pricing at $2.50 per million input tokens, that's real money saved on every API call. More importantly, you can fit 30x more actual documentation into the same context window.
-
-## What This Tool Does
-
-llm-docs-builder helps you optimize markdown documentation for AI consumption:
-
-1. **Measure Savings** - Compare what your server sends to humans (HTML) vs AI bots (markdown) to quantify context window reduction
-2. **Transform Markdown** - Normalize your markdown files with absolute links and consistent URL formats for better LLM navigation
-3. **Generate llms.txt** - Create standardized documentation indexes following the [llms.txt](https://llmstxt.org/) specification
-4. **Serve Efficiently** - Configure your server to automatically serve transformed markdown to AI bots while humans get HTML
+- Human HTML version: 104.4 KB (~26,735 tokens)
+- AI markdown version: 21.5 KB (~5,496 tokens)
+- **Result: 79% reduction, 21,239 tokens saved, 5x smaller**
 
 ## Quick Start
 
 ### Measure Your Current Token Waste
 
-Before making any changes, see how much you could save:
-
 ```bash
 # Using Docker (no Ruby installation needed)
 docker pull mensfeld/llm-docs-builder:latest
@@ -42,235 +29,162 @@ docker run mensfeld/llm-docs-builder compare \
   --url https://yoursite.com/docs/getting-started.html
 ```
 
-**Example output:**
-```
-============================================================
-Context Window Comparison
-============================================================
-
-Human version:  45.2 KB (~11,300 tokens)
-  Source: https://yoursite.com/docs/page.html (User-Agent: human)
+### Transform Your Documentation
 
-AI version:     12.8 KB (~3,200 tokens)
-  Source: https://yoursite.com/docs/page.html (User-Agent: AI)
+```bash
+# Single file
+llm-docs-builder transform --docs README.md
 
-------------------------------------------------------------
-Reduction:      32.4 KB (72%)
-Token savings:  8,100 tokens (72%)
-Factor:         3.5x smaller
-============================================================
+# Bulk transform with config
+llm-docs-builder bulk-transform --config llm-docs-builder.yml
 ```
 
-This single command shows you the potential ROI before you invest any time in optimization.
-
-### Real-World Results
-
-**[Karafka Framework Documentation](https://karafka.io/docs)** (10 pages analyzed):
-
-| Page | Human HTML | AI Markdown | Reduction | Tokens Saved | Factor |
-|------|-----------|-------------|-----------|--------------|---------|
-| Getting Started | 82.0 KB | 4.1 KB | 95% | ~19,475 | 20.1x |
-| Configuration | 86.3 KB | 7.1 KB | 92% | ~19,800 | 12.1x |
-| Routing | 93.6 KB | 14.7 KB | 84% | ~19,725 | 6.4x |
-| Deployment | 122.1 KB | 33.3 KB | 73% | ~22,200 | 3.7x |
-| Producing Messages | 87.7 KB | 8.3 KB | 91% | ~19,850 | 10.6x |
-| Consuming Messages | 105.3 KB | 21.3 KB | 80% | ~21,000 | 4.9x |
-| Web UI Getting Started | 109.3 KB | 21.5 KB | 80% | ~21,950 | 5.1x |
-| Active Job | 88.7 KB | 8.8 KB | 90% | ~19,975 | 10.1x |
-| Monitoring and Logging | 120.7 KB | 32.5 KB | 73% | ~22,050 | 3.7x |
-| Error Handling | 93.8 KB | 13.1 KB | 86% | ~20,175 | 7.2x |
-
-**Average: 83% reduction, ~20,620 tokens saved per page, 8.4x smaller files**
-
-For a typical RAG system making 1,000 documentation queries per day:
-- **Before**: ~990 KB per day (~247,500 tokens) × 1,000 queries = ~247.5M tokens/day
-- **After**: ~165 KB per day (~41,250 tokens) × 1,000 queries = ~41.25M tokens/day
-- **Savings**: 83% reduction = ~206.25M tokens saved per day
-
-At GPT-4 pricing ($2.50/M input tokens), that's approximately **$500/day or $183,000/year saved** on a documentation site with moderate traffic.
-
 ## Installation
 
-### Option 1: Docker (Recommended)
-
-No Ruby installation required. Perfect for CI/CD and quick usage:
+### Docker (Recommended)
 
 ```bash
-# Pull the image
 docker pull mensfeld/llm-docs-builder:latest
-
-# Create an alias for convenience
 alias llm-docs-builder='docker run -v $(pwd):/workspace mensfeld/llm-docs-builder'
-
-# Use like a native command
-llm-docs-builder compare --url https://yoursite.com/docs
 ```
 
-Multi-architecture support (amd64/arm64), ~50MB image size.
-
-### Option 2: RubyGems
-
-For Ruby developers or when you need the Ruby API:
+### RubyGems
 
 ```bash
 gem install llm-docs-builder
 ```
 
-Or add to your Gemfile:
-
-```ruby
-gem 'llm-docs-builder'
-```
-
-## Core Features
+## Features
 
-### 1. Compare and Measure (The "Before You Start" Tool)
-
-Quantify exactly how much context window you're wasting:
+### Measure and Compare
 
 ```bash
-# Compare what your server sends to humans vs AI bots
+# Compare what your server sends to humans vs AI
 llm-docs-builder compare --url https://yoursite.com/docs/page.html
 
-# Compare remote HTML with your local markdown
+# Compare remote HTML with local markdown
 llm-docs-builder compare \
   --url https://yoursite.com/docs/api.html \
   --file docs/api.md
-
-# Verbose mode for debugging
-llm-docs-builder compare --url https://example.com/docs --verbose
 ```
 
-**Why this matters:**
-- Validates that optimizations actually work
-- Quantifies ROI before you invest time
-- Monitors ongoing effectiveness
-- Provides concrete metrics for stakeholders
-
-### 2. Transform Markdown (The Normalizer)
-
-Normalize your markdown documentation to be LLM-friendly:
+### Generate llms.txt
 
-**Single file transformation:**
 ```bash
-# Expand relative links to absolute URLs
-llm-docs-builder transform \
-  --docs README.md \
-  --config llm-docs-builder.yml
+# Create standardized documentation index
+llm-docs-builder generate --config llm-docs-builder.yml
 ```
 
-**Bulk transformation - two modes:**
+## Configuration
 
-**a) Separate files (default)** - Creates `.llm.md` versions alongside originals:
 ```yaml
 # llm-docs-builder.yml
 docs: ./docs
 base_url: https://myproject.io
-suffix: .llm                 # Creates README.llm.md alongside README.md
-convert_urls: true           # .html → .md
-remove_comments: true        # Remove HTML comments
-remove_badges: true          # Remove badge/shield images
-remove_frontmatter: true     # Remove YAML/TOML frontmatter
-normalize_whitespace: true   # Clean up excessive blank lines
-```
-
-```bash
-llm-docs-builder bulk-transform --config llm-docs-builder.yml
-```
+title: My Project
+description: Brief description
+output: llms.txt
+suffix: .llm
+verbose: false
 
-Result:
-```
-docs/
-├── README.md          ← Original (for humans)
-├── README.llm.md      ← Optimized (for AI)
-├── api.md
-└── api.llm.md
-```
+# Basic options
+convert_urls: true
+remove_comments: true
+remove_badges: true
+remove_frontmatter: true
+normalize_whitespace: true
 
-**b) In-place transformation** - Overwrites originals (for build pipelines):
-```yaml
-# llm-docs-builder.yml
-docs: ./docs
-base_url: https://myproject.io
-suffix: ""                   # Transforms in-place
-convert_urls: true           # Convert .html to .md
-remove_comments: true        # Remove HTML comments
-remove_badges: true          # Remove badge/shield images
-remove_frontmatter: true     # Remove YAML/TOML frontmatter
-normalize_whitespace: true   # Clean up excessive blank lines
+# Additional compression options
+remove_code_examples: false
+remove_images: true
+remove_blockquotes: true
+remove_duplicates: true
+remove_stopwords: false
+simplify_links: true
+generate_toc: true
+custom_instruction: "This documentation is optimized for AI consumption"
+
+# RAG enhancement options
+normalize_headings: true          # Add hierarchical context to headings
+heading_separator: " / "          # Separator for heading hierarchy
+include_metadata: true            # Enable enhanced llms.txt metadata
+include_tokens: true              # Include token counts in llms.txt
+include_timestamps: true          # Include update timestamps in llms.txt
+include_priority: true            # Include priority labels in llms.txt
+calculate_compression: false      # Calculate compression ratios (slower)
+
+# Exclusions
 excludes:
   - "**/private/**"
+  - "**/drafts/**"
 ```
 
-```bash
-llm-docs-builder bulk-transform --config llm-docs-builder.yml
-```
-
-Perfect for CI/CD where you transform docs before deployment.
-
-**What gets normalized:**
-- **Links**: Relative → Absolute URLs (`./api.md` → `https://yoursite.com/api.md`)
-- **URLs**: HTML → Markdown format (`.html` → `.md`)
-- **Comments**: HTML comments removed (`<!-- ... -->`)
-- **Badges**: Shield/badge images removed (CI badges, version badges, etc.)
-- **Frontmatter**: YAML/TOML metadata removed (Jekyll, Hugo, etc.)
-- **Whitespace**: Excessive blank lines reduced (3+ → 2 max)
-- Clean markdown structure preserved
-- No content modification, just intelligent cleanup
-
-### 3. Generate llms.txt (The Standard)
-
-Create a standardized documentation index following the [llms.txt](https://llmstxt.org/) specification:
+**Configuration precedence:**
+1. CLI flags (highest)
+2. Config file
+3. Defaults
 
-```yaml
-# llm-docs-builder.yml
-docs: ./docs
-base_url: https://myproject.io
-title: My Project
-description: A library that does amazing things
-output: llms.txt
-```
+## CLI Commands
 
 ```bash
-llm-docs-builder generate --config llm-docs-builder.yml
+llm-docs-builder compare [options]        # Measure token savings
+llm-docs-builder transform [options]      # Transform single file
+llm-docs-builder bulk-transform [options] # Transform directory
+llm-docs-builder generate [options]       # Generate llms.txt
+llm-docs-builder parse [options]          # Parse llms.txt
+llm-docs-builder validate [options]       # Validate llms.txt
+llm-docs-builder version                  # Show version
 ```
 
-**Generated output:**
-```markdown
-# My Project
+**Common options:**
+```
+-c, --config PATH    Configuration file
+-d, --docs PATH      Documentation path
+-o, --output PATH    Output file
+-u, --url URL        URL for comparison
+-v, --verbose        Detailed output
+```
 
-> A library that does amazing things
+## Ruby API
 
-## Documentation
+```ruby
+require 'llm_docs_builder'
 
-- [README](https://myproject.io/README.md): Complete overview and installation
-- [Getting Started](https://myproject.io/getting-started.md): Quick start guide
-- [API Reference](https://myproject.io/api-reference.md): Detailed API documentation
-```
+# Transform single file with custom options
+transformed = LlmDocsBuilder.transform_markdown(
+  'README.md',
+  base_url: 'https://myproject.io',
+  remove_code_examples: true,
+  remove_images: true,
+  generate_toc: true,
+  custom_instruction: 'AI-optimized documentation'
+)
 
-**Smart prioritization:**
-1. README files (always first)
-2. Getting started guides
-3. Tutorials and guides
-4. API references
-5. Other documentation
+# Bulk transform
+files = LlmDocsBuilder.bulk_transform(
+  './docs',
+  base_url: 'https://myproject.io',
+  suffix: '.llm',
+  remove_duplicates: true,
+  generate_toc: true
+)
 
-The llms.txt file serves as an efficient entry point for AI systems to understand your project structure.
+# Generate llms.txt
+content = LlmDocsBuilder.generate_from_docs(
+  './docs',
+  base_url: 'https://myproject.io',
+  title: 'My Project'
+)
+```
 
-### 4. Serve to AI Bots (The Deployment)
+## Serving Optimized Docs to AI Bots
 
-After using `bulk-transform` with `suffix: .llm`, configure your web server to automatically serve optimized versions to AI bots:
+After using `bulk-transform` with `suffix: .llm`, configure your web server to serve optimized versions to AI bots:
 
 **Apache (.htaccess):**
 ```apache
-# Detect AI bots
-SetEnvIf User-Agent "(?i)(openai|anthropic|claude|gpt|chatgpt)" IS_LLM_BOT
-SetEnvIf User-Agent "(?i)(perplexity|gemini|copilot|bard)" IS_LLM_BOT
-
-# Serve .llm.md to AI, .md to humans
-RewriteEngine On
+SetEnvIf User-Agent "(?i)(openai|anthropic|claude|gpt)" IS_LLM_BOT
 RewriteCond %{ENV:IS_LLM_BOT} !^$
-RewriteCond %{REQUEST_URI} ^/docs/.*\.md$ [NC]
 RewriteRule ^(.*)\.md$ $1.llm.md [L]
 ```
 
@@ -279,7 +193,6 @@ RewriteRule ^(.*)\.md$ $1.llm.md [L]
 map $http_user_agent $is_llm_bot {
     default 0;
     "~*(?i)(openai|anthropic|claude|gpt)" 1;
-    "~*(?i)(perplexity|gemini|copilot)" 1;
 }
 
 location ~ ^/docs/(.*)\.md$ {
@@ -289,435 +202,222 @@ location ~ ^/docs/(.*)\.md$ {
 }
 ```
 
-**Cloudflare Workers:**
-```javascript
-const isLLMBot = /openai|anthropic|claude|gpt|perplexity/i.test(userAgent);
-if (isLLMBot && url.pathname.startsWith('/docs/')) {
-  url.pathname = url.pathname.replace(/\.md$/, '.llm.md');
-}
-```
-
-**Result**: AI systems automatically get optimized versions, humans get the original. No manual switching, no duplicate URLs.
+## Real-World Results: Karafka Framework
 
-## Configuration
-
-All commands support both config files and CLI flags. Config files are recommended for consistency:
+**Before:** 140+ lines of custom transformation code
 
+**After:** 6 lines of configuration
 ```yaml
-# llm-docs-builder.yml
-docs: ./docs
-base_url: https://myproject.io
-title: My Project
-description: Brief description
-output: llms.txt
+docs: ./online/docs
+base_url: https://karafka.io/docs
 convert_urls: true
 remove_comments: true
 remove_badges: true
 remove_frontmatter: true
 normalize_whitespace: true
-suffix: .llm
-verbose: false
-excludes:
-  - "**/private/**"
-  - "**/drafts/**"
+suffix: ""  # In-place for build pipeline
 ```
 
-**Configuration precedence:**
-1. CLI flags (highest priority)
-2. Config file values
-3. Defaults
-
-**Example of overriding:**
-```bash
-# Uses config file but overrides title
-llm-docs-builder generate --config llm-docs-builder.yml --title "Override Title"
-```
+**Results:**
+- 93% average token reduction
+- 20-36x smaller files
+- Automated via GitHub Actions
 
 ## Docker Usage
 
-All CLI commands work in Docker with the same syntax:
-
 ```bash
-# Basic pattern
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder [command] [options]
+# Pull image
+docker pull mensfeld/llm-docs-builder:latest
 
-# Examples
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder generate --docs ./docs
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder transform --docs README.md
-docker run mensfeld/llm-docs-builder compare --url https://example.com/docs
-```
+# Compare (no volume needed for remote URLs)
+docker run mensfeld/llm-docs-builder compare \
+  --url https://yoursite.com/docs
 
-**CI/CD Integration:**
+# Transform with volume mount
+docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
+  bulk-transform --config llm-docs-builder.yml
+```
 
-GitHub Actions:
+**CI/CD Example (GitHub Actions):**
 ```yaml
-- name: Generate llms.txt
+- name: Optimize documentation
   run: |
     docker run -v ${{ github.workspace }}:/workspace \
-      mensfeld/llm-docs-builder generate --config llm-docs-builder.yml
-```
-
-GitLab CI:
-```yaml
-generate-llms:
-  image: mensfeld/llm-docs-builder:latest
-  script:
-    - llm-docs-builder generate --docs ./docs
+      mensfeld/llm-docs-builder bulk-transform --config llm-docs-builder.yml
 ```
 
-See [Docker Usage](#detailed-docker-usage) section below for comprehensive examples.
-
-## Ruby API
-
-For programmatic usage:
-
-```ruby
-require 'llm_docs_builder'
-
-# Using config file
-content = LlmDocsBuilder.generate_from_docs(config_file: 'llm-docs-builder.yml')
-
-# Direct options
-content = LlmDocsBuilder.generate_from_docs('./docs',
-  base_url: 'https://myproject.io',
-  title: 'My Project'
-)
+## Compression Examples
 
-# Transform markdown
-transformed = LlmDocsBuilder.transform_markdown('README.md',
-  base_url: 'https://myproject.io',
-  convert_urls: true,
-  remove_comments: true,
-  remove_badges: true,
-  remove_frontmatter: true,
-  normalize_whitespace: true
-)
-
-# Bulk transform
-files = LlmDocsBuilder.bulk_transform('./docs',
-  base_url: 'https://myproject.io',
-  suffix: '.llm',
-  remove_comments: true,
-  remove_badges: true,
-  remove_frontmatter: true,
-  normalize_whitespace: true,
-  excludes: ['**/private/**']
-)
-
-# In-place transformation
-files = LlmDocsBuilder.bulk_transform('./docs',
-  suffix: '',  # Empty for in-place
-  base_url: 'https://myproject.io',
-  remove_comments: true,
-  remove_badges: true,
-  remove_frontmatter: true,
-  normalize_whitespace: true
-)
-```
+**Input markdown:**
+```markdown
+---
+layout: docs
+---
 
-## Real-World Case Study: Karafka Framework
+# API Documentation
 
-The [Karafka framework](https://github.com/karafka/karafka) processes millions of Kafka messages daily and maintains extensive documentation. Before llm-docs-builder:
+[![Build](badge.svg)](https://ci.com)
 
-- **140+ lines of custom Ruby code** for link expansion and URL normalization
-- Manual maintenance of transformation logic
-- No way to measure optimization effectiveness
+> Important: This is a note
 
-**After implementing llm-docs-builder:**
+[Click here to see the complete API documentation](./api.md)
 
-```yaml
-# llm-docs-builder.yml
-docs: ./online/docs
-base_url: https://karafka.io/docs
-convert_urls: true
-remove_comments: true
-remove_badges: true
-remove_frontmatter: true
-normalize_whitespace: true
-suffix: ""  # In-place transformation for build pipeline
-excludes:
-  - "**/Enterprise-License-Setup/**"
+```ruby
+api = API.new
 ```
 
-```bash
-# In their deployment script
-llm-docs-builder bulk-transform --config llm-docs-builder.yml
+![Diagram](./diagram.png)
 ```
 
-**Results:**
-- **140 lines of code → 6 lines of config**
-- **93% average token reduction** across all documentation
-- **Quantifiable savings** via the compare command
-- **Automated daily deployments** via GitHub Actions
-
-The compare command revealed that their documentation was consuming 20-36x more tokens than necessary for AI systems. After optimization, RAG queries became dramatically more efficient.
+**After transformation (with default options):**
+```markdown
+# API Documentation
 
-## CLI Reference
+[complete API documentation](./api.md)
 
-```bash
-llm-docs-builder compare [options]        # Measure token savings (start here!)
-llm-docs-builder transform [options]      # Transform single markdown file
-llm-docs-builder bulk-transform [options] # Transform entire documentation tree
-llm-docs-builder generate [options]       # Generate llms.txt index
-llm-docs-builder parse [options]          # Parse existing llms.txt
-llm-docs-builder validate [options]       # Validate llms.txt format
-llm-docs-builder version                  # Show version
-```
-
-**Common options:**
+```ruby
+api = API.new
 ```
--c, --config PATH    Configuration file (default: llm-docs-builder.yml)
--d, --docs PATH      Documentation directory or file
--o, --output PATH    Output file path
--u, --url URL        URL for comparison
--f, --file PATH      Local file for comparison
--v, --verbose        Detailed output
--h, --help           Show help
 ```
 
-For advanced options (base_url, title, suffix, excludes, convert_urls), use a config file.
-
-## Why This Matters for RAG Systems
+**Token reduction:** ~40-60% depending on configuration
 
-Retrieval-Augmented Generation (RAG) systems fetch documentation to answer questions. Every byte of overhead in those documents:
-
-1. **Costs money** - More tokens = higher API costs
-2. **Reduces capacity** - Less room for actual documentation in context window
-3. **Slows responses** - More tokens to process = longer response times
-4. **Degrades quality** - Navigation noise can confuse the model
-
-llm-docs-builder addresses all four issues by transforming markdown to be AI-friendly and enabling your server to automatically serve it to AI bots while humans get HTML.
-
-**The JavaScript Problem:**
-
-Many documentation sites rely on JavaScript for rendering. AI crawlers typically don't execute JavaScript, so they either:
-- Get incomplete content
-- Get server-side rendered HTML (bloated with framework overhead)
-- Fail entirely
-
-By detecting AI bots and serving them clean markdown instead of HTML, you sidestep this problem entirely.
-
-## Configuration Reference
-
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `docs` | String | `./docs` | Documentation directory or file |
-| `base_url` | String | - | Base URL for absolute links (e.g., `https://myproject.io`) |
-| `title` | String | Auto-detected | Project title |
-| `description` | String | Auto-detected | Project description |
-| `output` | String | `llms.txt` | Output filename for llms.txt generation |
-| `convert_urls` | Boolean | `false` | Convert `.html`/`.htm` to `.md` |
-| `remove_comments` | Boolean | `false` | Remove HTML comments (`<!-- ... -->`) |
-| `remove_badges` | Boolean | `false` | Remove badge/shield images (CI, version, etc.) |
-| `remove_frontmatter` | Boolean | `false` | Remove YAML/TOML frontmatter (Jekyll, Hugo) |
-| `normalize_whitespace` | Boolean | `false` | Normalize excessive blank lines and trailing spaces |
-| `suffix` | String | `.llm` | Suffix for transformed files (use `""` for in-place) |
-| `excludes` | Array | `[]` | Glob patterns to exclude |
-| `verbose` | Boolean | `false` | Enable detailed output |
-
-## Detailed Docker Usage
+## FAQ
 
-### Installation and Setup
+**Q: Do I need to use llms.txt?**
+No. The compare and transform commands work independently.
 
-```bash
-# Pull from Docker Hub
-docker pull mensfeld/llm-docs-builder:latest
+**Q: Will this change how humans see my docs?**
+Not with default `suffix: .llm`. Separate files are served only to AI bots.
 
-# Or from GitHub Container Registry
-docker pull ghcr.io/mensfeld/llm-docs-builder:latest
+**Q: Can I use this in my build pipeline?**
+Yes. Use `suffix: ""` for in-place transformation.
 
-# Create an alias for convenience
-alias llm-docs-builder='docker run -v $(pwd):/workspace mensfeld/llm-docs-builder'
-```
+**Q: How do I know if it's working?**
+Use `llm-docs-builder compare` to measure before and after.
 
-### Common Commands
+**Q: What about private documentation?**
+Use the `excludes` option to skip sensitive files.
 
-**Compare (no volume mount needed for remote URLs):**
-```bash
-docker run mensfeld/llm-docs-builder compare \
-  --url https://karafka.io/docs/Getting-Started/
+## RAG Enhancement Features
 
-# With local file
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder compare \
-  --url https://example.com/page.html \
-  --file docs/page.md
-```
+### Heading Normalization
 
-**Generate llms.txt:**
-```bash
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
-  generate --docs ./docs --output llms.txt
-```
+Transform headings to include hierarchical context, making each section self-contained for RAG retrieval:
 
-**Transform single file:**
-```bash
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
-  transform --docs README.md --config llm-docs-builder.yml
-```
+**Before:**
+```markdown
+# Configuration
+## Consumer Settings
+### auto_offset_reset
 
-**Bulk transform:**
-```bash
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
-  bulk-transform --config llm-docs-builder.yml
+Controls behavior when no offset exists...
 ```
 
-**Parse and validate:**
-```bash
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
-  parse --docs llms.txt --verbose
+**After (with `normalize_headings: true`):**
+```markdown
+# Configuration
+## Configuration / Consumer Settings
+### Configuration / Consumer Settings / auto_offset_reset
 
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder \
-  validate --docs llms.txt
+Controls behavior when no offset exists...
 ```
 
-### CI/CD Examples
+**Why this matters for RAG:** When documents are chunked and retrieved independently, each section retains full context. An LLM seeing just the `auto_offset_reset` section knows it's about "Configuration / Consumer Settings / auto_offset_reset" not just generic "auto_offset_reset".
 
-**GitHub Actions:**
 ```yaml
-jobs:
-  optimize-docs:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Transform documentation
-        run: |
-          docker run -v ${{ github.workspace }}:/workspace \
-            mensfeld/llm-docs-builder bulk-transform --config llm-docs-builder.yml
-      - name: Measure savings
-        run: |
-          docker run mensfeld/llm-docs-builder \
-            compare --url https://yoursite.com/docs/main.html
+# Enable in config
+normalize_headings: true
+heading_separator: " / "  # Customize separator (default: " / ")
 ```
 
-**GitLab CI:**
-```yaml
-optimize-docs:
-  image: mensfeld/llm-docs-builder:latest
-  script:
-    - llm-docs-builder bulk-transform --docs ./docs
-    - llm-docs-builder compare --url https://yoursite.com/docs
-```
-
-**Jenkins:**
-```groovy
-stage('Optimize Documentation') {
-    steps {
-        sh '''
-            docker run -v ${WORKSPACE}:/workspace \
-              mensfeld/llm-docs-builder bulk-transform --config llm-docs-builder.yml
-        '''
-    }
-}
-```
-
-### Version Pinning
-
-```bash
-# Use specific version
-docker run mensfeld/llm-docs-builder:0.3.0 version
+### Enhanced llms.txt Metadata
 
-# Use major version (gets latest patch)
-docker run mensfeld/llm-docs-builder:0 version
+Generate enriched llms.txt files with token counts, timestamps, and priority labels to help AI agents make better decisions:
 
-# Always latest
-docker run mensfeld/llm-docs-builder:latest version
+**Standard llms.txt:**
+```markdown
+- [Getting Started](https://myproject.io/docs/Getting-Started.md)
+- [Configuration](https://myproject.io/docs/Configuration.md)
 ```
 
-### Platform-Specific Usage
-
-**Windows PowerShell:**
-```powershell
-docker run -v ${PWD}:/workspace mensfeld/llm-docs-builder generate --docs ./docs
+**Enhanced llms.txt (with metadata enabled):**
+```markdown
+- [Getting Started](https://myproject.io/docs/Getting-Started.md) tokens:450 updated:2025-10-13 priority:high
+- [Configuration](https://myproject.io/docs/Configuration.md) tokens:2800 updated:2025-10-12 priority:high
+- [Advanced Topics](https://myproject.io/docs/Advanced.md) tokens:5200 updated:2025-09-15 priority:medium
 ```
 
-**Windows Command Prompt:**
-```cmd
-docker run -v %cd%:/workspace mensfeld/llm-docs-builder generate --docs ./docs
-```
+**Benefits:**
+- AI agents can see token counts → load multiple small docs vs one large doc
+- Timestamps help prefer recent documentation
+- Priority signals guide which docs to fetch first
+- Compression ratios show optimization effectiveness
 
-**macOS/Linux:**
-```bash
-docker run -v $(pwd):/workspace mensfeld/llm-docs-builder generate --docs ./docs
+```yaml
+# Enable in config
+include_metadata: true      # Master switch
+include_tokens: true        # Show token counts
+include_timestamps: true    # Show last modified dates
+include_priority: true      # Show priority labels (high/medium/low)
+calculate_compression: true # Show compression ratios (slower, requires transformation)
 ```
 
-## About llms.txt Standard
-
-The [llms.txt specification](https://llmstxt.org/) is a proposed standard for providing LLM-friendly content. It defines a structured format that helps AI systems:
-
-- Quickly understand project structure
-- Find relevant documentation efficiently
-- Navigate complex documentation hierarchies
-- Access clean, markdown-formatted content
+## Advanced Compression Options
 
-llm-docs-builder generates llms.txt files automatically by:
-1. Scanning your documentation directory
-2. Extracting titles and descriptions from markdown files
-3. Prioritizing content by importance (README first, then guides, APIs, etc.)
-4. Formatting everything according to the specification
+All compression features can be used individually for fine-grained control:
 
-The llms.txt file serves as an efficient entry point, but the real token savings come from serving optimized markdown for each individual documentation page.
+### Content Removal Options
 
-## How It Works
+- `remove_frontmatter: true` - Remove YAML/TOML metadata blocks
+- `remove_comments: true` - Remove HTML comments (`<!-- ... -->`)
+- `remove_badges: true` - Remove badge/shield images (CI badges, version badges, etc.)
+- `remove_images: true` - Remove all image syntax
+- `remove_code_examples: true` - Remove fenced code blocks, indented code, and inline code
+- `remove_blockquotes: true` - Remove blockquote formatting (preserves content)
+- `remove_duplicates: true` - Remove duplicate paragraphs using fuzzy matching
+- `remove_stopwords: true` - Remove common stopwords from prose (preserves code blocks)
 
-**Generation Process:**
-1. Scan directory for `.md` files
-2. Extract title (first H1) and description (first paragraph)
-3. Prioritize by importance (README → Getting Started → Guides → API → Other)
-4. Build formatted llms.txt with links and descriptions
+### Content Enhancement Options
 
-**Transformation Process:**
-1. Remove frontmatter (YAML/TOML metadata)
-2. Expand relative links to absolute URLs
-3. Convert `.html` URLs to `.md`
-4. Remove HTML comments
-5. Remove badge/shield images
-6. Normalize excessive whitespace
-7. Write to new file or overwrite in-place
+- `generate_toc: true` - Generate table of contents from headings with anchor links
+- `custom_instruction: "text"` - Inject AI context message at document top
+- `simplify_links: true` - Simplify verbose link text (e.g., "Click here to see the docs" → "docs")
+- `convert_urls: true` - Convert `.html`/`.htm` URLs to `.md` format
+- `normalize_whitespace: true` - Reduce excessive blank lines and remove trailing whitespace
 
-**Comparison Process:**
-1. Fetch URL with human User-Agent (or read local file)
-2. Fetch same URL with AI bot User-Agent
-3. Calculate size difference and reduction percentage
-4. Estimate token counts using character-based heuristic
-5. Display human-readable comparison results with byte and token savings
+### Example Usage
 
-**Token Estimation:**
-The tool uses a simple but effective heuristic for estimating token counts: **~4 characters per token**. This approximation works well for English documentation and provides reasonable estimates without requiring external tokenizer dependencies. While not as precise as OpenAI's tiktoken, it's accurate enough (±10-15%) for understanding context window savings and making optimization decisions.
-
-## FAQ
-
-**Q: Do I need to use llms.txt to benefit from this tool?**
-
-No. The compare and transform commands provide value independently. Many users start with `compare` to measure savings, then use `bulk-transform` to normalize their markdown files, and may never generate an llms.txt file.
-
-**Q: Will this change how humans see my documentation?**
-
-Not if you use the default `suffix: .llm` mode. This creates separate `.llm.md` files served only to AI bots. Your original files remain unchanged for human visitors.
-
-**Q: Can I use this in my build pipeline?**
-
-Yes. Use `suffix: ""` for in-place transformation. The Karafka framework does this - they transform their markdown as part of their deployment process.
-
-**Q: How do I know if it's working?**
-
-Use the `compare` command to measure before and after. It shows exact byte counts, reduction percentages, and compression factors.
-
-**Q: Does this work with static site generators?**
-
-Yes. You can transform markdown files before your static site generator processes them, or serve separate `.llm.md` versions alongside your generated HTML.
+```ruby
+# Fine-grained control
+LlmDocsBuilder.transform_markdown(
+  'README.md',
+  remove_frontmatter: true,
+  remove_badges: true,
+  remove_images: true,
+  simplify_links: true,
+  generate_toc: true,
+  normalize_whitespace: true
+)
+```
 
-**Q: What about private/internal documentation?**
+Or configure via YAML:
 
-Use the `excludes` option to skip sensitive files:
 ```yaml
-excludes:
-  - "**/private/**"
-  - "**/internal/**"
-```
-
-**Q: Can I customize the AI bot detection?**
+# llm-docs-builder.yml
+docs: ./docs
+base_url: https://myproject.io
+suffix: .llm
 
-Yes. The web server examples show the User-Agent patterns. You can add or remove patterns based on which AI systems you want to support.
+# Pick exactly what you need
+remove_frontmatter: true
+remove_comments: true
+remove_badges: true
+remove_images: true
+simplify_links: true
+generate_toc: true
+normalize_whitespace: true
+```
 
 ## Contributing