llm-docs-builder 0.8.2 → 0.9.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 574ebb294e2ebf55146cad9a17120f32aefac4ad22f01c72c35eb6dbb2b8cdf5
-  data.tar.gz: d1980a9b4186249f03016a7924376624de8b5cad632aa9a96b77cb089048e229
+  metadata.gz: 6ddb04b5a0e30d913d2043a79a3d7e14d35bd0166cf7104a300457887b1019cf
+  data.tar.gz: 5301d904225d0d139a2c2dd8184695eb66e2d858b8f3b5a86026b16a1ccb8c44
 SHA512:
-  metadata.gz: 39c8e6ebf9fccfd73be6e7ccff8345a16d869cb27051aceef1efb1f12da6c8fc1906ee5dd5fd5e485511dbf5cf7c28a707c24623451448bfde171213eff5a7ed
-  data.tar.gz: cb5f2301ad7d535e917565a9b6f0794596dd0dcf443ae2e43d973a592a7e4cf4c04b0c61c523ba05735f29cb6ce67d1e02a20c876274ba5b313485242df8683d
+  metadata.gz: 3daacfdde22c93023677e7e0e7487158b3e1f30a9c4e7ec2bf015bc220ff32296334411c74523aaa411d1ef21b1aa47a2e8cb2c7cada797922d89d5690f7ab0a
+  data.tar.gz: 50a8f8d29f9e79e6f5ab2774111385f6098378491b2732bb84a9d3eee0b2f5be4b6beae196ef76739a6ac1c81562da9ec6f3e66da562e657828874c55ec06ce1

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 0.9.1 (2025-10-17)
+- [Fix] Fixed HeadingTransformer incorrectly treating hash symbols in code blocks as headings.
+  - Now properly tracks code block boundaries (fenced with ``` or ~~~)
+  - Skips heading processing for lines inside code blocks
+  - Prevents Ruby/Python/Shell comments from being interpreted as markdown headings
+  - Added comprehensive test coverage for code block handling
+
+## 0.9.0 (2025-10-17)
+- [Feature] **No AI Version Detection** - The `compare` command now detects when websites don't serve AI-optimized versions.
+  - Triggers when reduction is <5% (nearly identical content for human and AI User-Agents)
+  - Displays prominent warning: "WARNING: NO DEDICATED AI VERSION DETECTED"
+  - Shows potential savings estimates based on typical 83% reduction rate
+  - Provides page-specific calculations (estimated token savings, potential size)
+  - Includes implementation guide with actionable steps
+  - Helps identify opportunities to optimize documentation
+- [Enhancement] Updated `OutputFormatter#display_comparison_results` to include marketing message for unoptimized sites.
+- [Enhancement] Added utility script `probe_karafka_simple.rb` for batch comparison testing.
+
 ## 0.8.2 (2025-10-17)
 - [Fix] Fixed Docker workflow test to properly invoke help command (use `generate --help` instead of `--help`).
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    llm-docs-builder (0.8.2)
+    llm-docs-builder (0.9.1)
       zeitwerk (~> 2.6)
 
 GEM

data/README.md

@@ -1,3 +1,7 @@
+<p align="center">
+  <img src="misc/logo_wide.png" alt="llm-docs-builder logo">
+</p>
+
 # llm-docs-builder
 
 [![CI](https://github.com/mensfeld/llm-docs-builder/actions/workflows/ci.yml/badge.svg)](
@@ -11,10 +15,13 @@ llm-docs-builder transforms markdown documentation to be AI-friendly and generat
 
 When LLMs fetch documentation, they typically get HTML pages designed for humans - complete with navigation bars, footers, JavaScript, CSS, and other overhead. This wastes 70-90% of your context window on content that doesn't help answer questions.
 
-**Real example from Karafka documentation:**
-- Human HTML version: 104.4 KB (~26,735 tokens)
-- AI markdown version: 21.5 KB (~5,496 tokens)
-- **Result: 79% reduction, 21,239 tokens saved, 5x smaller**
+**Real-world results from [Karafka documentation](https://karafka.io/docs/) (10 pages analyzed):**
+
+<p align="center">
+  <img src="misc/diff.png" alt="Karafka documentation optimization results">
+</p>
+
+**Average reduction: 83% fewer tokens**
 
 ## Quick Start
 
@@ -138,11 +145,6 @@ excludes:
   - "**/drafts/**"
 ```
 
-**Configuration precedence:**
-1. CLI flags (highest)
-2. Config file
-3. Defaults
-
 ## CLI Commands
 
 ```bash
@@ -164,38 +166,6 @@ llm-docs-builder version                  # Show version
 -v, --verbose        Detailed output
 ```
 
-## Ruby API
-
-```ruby
-require 'llm_docs_builder'
-
-# Transform single file with custom options
-transformed = LlmDocsBuilder.transform_markdown(
-  'README.md',
-  base_url: 'https://myproject.io',
-  remove_code_examples: true,
-  remove_images: true,
-  generate_toc: true,
-  custom_instruction: 'AI-optimized documentation'
-)
-
-# Bulk transform
-files = LlmDocsBuilder.bulk_transform(
-  './docs',
-  base_url: 'https://myproject.io',
-  suffix: '.llm',
-  remove_duplicates: true,
-  generate_toc: true
-)
-
-# Generate llms.txt
-content = LlmDocsBuilder.generate_from_docs(
-  './docs',
-  base_url: 'https://myproject.io',
-  title: 'My Project'
-)
-```
-
 ## Serving Optimized Docs to AI Bots
 
 After using `bulk-transform` with `suffix: .llm`, configure your web server to serve optimized versions to AI bots:
@@ -221,27 +191,6 @@ location ~ ^/docs/(.*)\.md$ {
 }
 ```
 
-## Real-World Results: Karafka Framework
-
-**Before:** 140+ lines of custom transformation code
-
-**After:** 6 lines of configuration
-```yaml
-docs: ./online/docs
-base_url: https://karafka.io/docs
-convert_urls: true
-remove_comments: true
-remove_badges: true
-remove_frontmatter: true
-normalize_whitespace: true
-suffix: ""  # In-place for build pipeline
-```
-
-**Results:**
-- 93% average token reduction
-- 20-36x smaller files
-- Automated via GitHub Actions
-
 ## Docker Usage
 
 ```bash
@@ -281,43 +230,21 @@ layout: docs
 
 [Click here to see the complete API documentation](./api.md)
 
-```ruby
 api = API.new
 ```
 
-![Diagram](./diagram.png)
-```
-
 **After transformation (with default options):**
+
 ```markdown
 # API Documentation
 
 [complete API documentation](./api.md)
 
-```ruby
 api = API.new
 ```
-```
 
 **Token reduction:** ~40-60% depending on configuration
 
-## FAQ
-
-**Q: Do I need to use llms.txt?**
-No. The compare and transform commands work independently.
-
-**Q: Will this change how humans see my docs?**
-Not with default `suffix: .llm`. Separate files are served only to AI bots.
-
-**Q: Can I use this in my build pipeline?**
-Yes. Use `suffix: ""` for in-place transformation.
-
-**Q: How do I know if it's working?**
-Use `llm-docs-builder compare` to measure before and after.
-
-**Q: What about private documentation?**
-Use the `excludes` option to skip sensitive files.
-
 ## RAG Enhancement Features
 
 ### Heading Normalization
@@ -405,43 +332,6 @@ All compression features can be used individually for fine-grained control:
 - `convert_urls: true` - Convert `.html`/`.htm` URLs to `.md` format
 - `normalize_whitespace: true` - Reduce excessive blank lines and remove trailing whitespace
 
-### Example Usage
-
-```ruby
-# Fine-grained control
-LlmDocsBuilder.transform_markdown(
-  'README.md',
-  remove_frontmatter: true,
-  remove_badges: true,
-  remove_images: true,
-  simplify_links: true,
-  generate_toc: true,
-  normalize_whitespace: true
-)
-```
-
-Or configure via YAML:
-
-```yaml
-# llm-docs-builder.yml
-docs: ./docs
-base_url: https://myproject.io
-suffix: .llm
-
-# Pick exactly what you need
-remove_frontmatter: true
-remove_comments: true
-remove_badges: true
-remove_images: true
-simplify_links: true
-generate_toc: true
-normalize_whitespace: true
-```
-
-## Contributing
-
-Bug reports and pull requests welcome at [github.com/mensfeld/llm-docs-builder](https://github.com/mensfeld/llm-docs-builder).
-
 ## License
 
 Available as open source under the [MIT License](https://opensource.org/licenses/MIT).

data/lib/llm_docs_builder/output_formatter.rb

@@ -8,6 +8,8 @@ module LlmDocsBuilder
   #
   # @api private
   class OutputFormatter
+    # Threshold percentage below which we consider there's no AI-optimized version
+    NO_AI_VERSION_THRESHOLD = 5
     # Format bytes into human-readable string
     #
     # @param bytes [Integer] number of bytes
@@ -56,10 +58,16 @@ module LlmDocsBuilder
 
       if result[:reduction_bytes].positive?
         display_reduction(result)
+
+        # Detect if there's no dedicated AI version
+        if result[:reduction_percent] < NO_AI_VERSION_THRESHOLD
+          display_no_ai_version_message(result)
+        end
       elsif result[:reduction_bytes].negative?
         display_increase(result)
       else
         puts 'Same size'
+        display_no_ai_version_message(result)
       end
 
       puts '=' * 60
@@ -89,5 +97,43 @@ module LlmDocsBuilder
       puts "Token increase: #{format_number(token_increase)} tokens (#{token_increase_percent}%)"
       puts "Factor:         #{result[:factor]}x larger"
     end
+
+    # Display message when no dedicated AI version is detected
+    #
+    # @param result [Hash] comparison results
+    # @api private
+    def self.display_no_ai_version_message(result)
+      puts ''
+      puts 'WARNING: NO DEDICATED AI VERSION DETECTED'
+      puts ''
+      puts 'The server is returning nearly identical content to both human and AI'
+      puts 'User-Agents, indicating no AI-optimized version is currently served.'
+      puts ''
+      puts 'POTENTIAL SAVINGS WITH AI OPTIMIZATION:'
+      puts ''
+      puts 'Based on typical documentation optimization results, you could expect:'
+      puts '  • 67-95% token reduction (average 83%)'
+      puts '  • 3-20x smaller file sizes'
+      puts '  • Faster LLM processing times'
+      puts '  • Reduced API costs for AI queries'
+      puts '  • Improved response accuracy'
+      puts ''
+      puts "For this page specifically (~#{format_number(result[:human_tokens])} tokens):"
+      puts "  • Estimated savings: ~#{format_number((result[:human_tokens] * 0.83).round)} tokens (83% reduction)"
+      puts "  • Could reduce to: ~#{format_number((result[:human_tokens] * 0.17).round)} tokens"
+      puts "  • Potential size: ~#{format_bytes((result[:human_size] * 0.17).round)}"
+      puts ''
+      puts 'HOW TO IMPLEMENT AI-OPTIMIZED DOCUMENTATION:'
+      puts ''
+      puts '1. Transform your docs with llm-docs-builder:'
+      puts '   llm-docs-builder bulk-transform --docs ./docs --config llm-docs-builder.yml'
+      puts ''
+      puts '2. Configure your web server to serve .md files to AI bots:'
+      puts '   See: https://github.com/mensfeld/llm-docs-builder#serving-optimized-docs'
+      puts ''
+      puts '3. Measure your actual savings:'
+      puts '   llm-docs-builder compare --url <your-url> --file <local-md>'
+      puts ''
+    end
   end
 end

data/lib/llm_docs_builder/transformers/heading_transformer.rb

@@ -38,8 +38,18 @@ module LlmDocsBuilder
         separator = options[:heading_separator] || ' / '
         heading_stack = []
         lines = content.lines
+        in_code_block = false
 
         transformed_lines = lines.map do |line|
+          # Track code block boundaries (fenced code blocks with ``` or ~~~)
+          if line.match?(/^```|^~~~/)
+            in_code_block = !in_code_block
+            next line
+          end
+
+          # Skip heading processing if inside code block
+          next line if in_code_block
+
           # Match markdown headings (1-6 hash symbols followed by space and text)
           heading_match = line.match(/^(#+)\s+(.+)$/)
 

data/lib/llm_docs_builder/version.rb

@@ -2,5 +2,5 @@
 
 module LlmDocsBuilder
   # Current version of the LlmDocsBuilder gem
-  VERSION = '0.8.2'
+  VERSION = '0.9.1'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm-docs-builder
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.9.1
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -147,6 +147,9 @@ files:
 - lib/llm_docs_builder/version.rb
 - llm-docs-builder.gemspec
 - llm-docs-builder.yml.example
+- misc/diff.png
+- misc/logo.png
+- misc/logo_wide.png
 - renovate.json
 homepage: https://github.com/mensfeld/llm-docs-builder
 licenses: