llm-docs-builder 0.9.0 → 0.9.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7f0653d0b7cbe76ef97429cd96ebf225783d2d454c765346c1c13279b73b7527
-  data.tar.gz: 206472b3a91c477827c3adeb73679758ebe1b3d73b47ef9c60a04afcce183c79
+  metadata.gz: e2b8530a9f890cb3deed824d3520f0a267a10b1e60a38e6bc8948b6447f50200
+  data.tar.gz: 7fdca5e5403050d9f4845bf7deb29fd64fc7df8758004a8f97d5e5ff4c519d27
 SHA512:
-  metadata.gz: eca112eb185027b8fd905d062d1abf54ed147025d16608176aae6ee6caf836af658d66bb315f5e11137d109e10fb6fce85aedd6d730afbafd9379c49061e1bbd
-  data.tar.gz: 2b6b7c2227e1b3926378238f743825b36f350cbfa97abaffad2a26e1061063e6185eafb18762152c1174cbcd3dd1c7132cce6affc1b2cbf1ad19be8eda83dfc0
+  metadata.gz: 0bd796067af5a3246ff99989d0190c33882218a5f5ea0a6f8dfc9a6291ab5fba122269fc1b522acebd2782d07f1ff99a38557efcf616e5e88bfb5ad1c34ec75e
+  data.tar.gz: 9b2e456821ccc73790fb7feb8f1046c89d4111c47be4f6755ba2059ea6ab2b5433229e5053a883b0b976b73875889af085f266089d57468084550aa6deb248d3

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.9.2 (2025-10-17)
+- [Fix] Tackle one more block boundaries tracking edge-case.
+
+## 0.9.1 (2025-10-17)
+- [Fix] Fixed HeadingTransformer incorrectly treating hash symbols in code blocks as headings.
+  - Now properly tracks code block boundaries (fenced with ``` or ~~~)
+  - Fixed regex pattern from `/^```|^~~~/` to `/^(```|~~~)/` for correct operator precedence
+  - Skips heading processing for lines inside code blocks
+  - Prevents Ruby/Python/Shell comments from being interpreted as markdown headings
+  - Added 5 comprehensive test cases covering multiple scenarios to prevent regression
+  - Skips heading processing for lines inside code blocks
+  - Prevents Ruby/Python/Shell comments from being interpreted as markdown headings
+  - Added comprehensive test coverage for code block handling
+
 ## 0.9.0 (2025-10-17)
 - [Feature] **No AI Version Detection** - The `compare` command now detects when websites don't serve AI-optimized versions.
   - Triggers when reduction is <5% (nearly identical content for human and AI User-Agents)

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    llm-docs-builder (0.9.0)
+    llm-docs-builder (0.9.2)
       zeitwerk (~> 2.6)
 
 GEM

data/README.md

@@ -15,10 +15,13 @@ llm-docs-builder transforms markdown documentation to be AI-friendly and generat
 
 When LLMs fetch documentation, they typically get HTML pages designed for humans - complete with navigation bars, footers, JavaScript, CSS, and other overhead. This wastes 70-90% of your context window on content that doesn't help answer questions.
 
-**Real example from Karafka documentation:**
-- Human HTML version: 104.4 KB (~26,735 tokens)
-- AI markdown version: 21.5 KB (~5,496 tokens)
-- **Result: 79% reduction, 21,239 tokens saved, 5x smaller**
+**Real-world results from [Karafka documentation](https://karafka.io/docs/) (10 pages analyzed):**
+
+<p align="center">
+  <img src="misc/diff.png" alt="Karafka documentation optimization results">
+</p>
+
+**Average reduction: 83% fewer tokens**
 
 ## Quick Start
 
@@ -142,11 +145,6 @@ excludes:
   - "**/drafts/**"
 ```
 
-**Configuration precedence:**
-1. CLI flags (highest)
-2. Config file
-3. Defaults
-
 ## CLI Commands
 
 ```bash
@@ -168,38 +166,6 @@ llm-docs-builder version                  # Show version
 -v, --verbose        Detailed output
 ```
 
-## Ruby API
-
-```ruby
-require 'llm_docs_builder'
-
-# Transform single file with custom options
-transformed = LlmDocsBuilder.transform_markdown(
-  'README.md',
-  base_url: 'https://myproject.io',
-  remove_code_examples: true,
-  remove_images: true,
-  generate_toc: true,
-  custom_instruction: 'AI-optimized documentation'
-)
-
-# Bulk transform
-files = LlmDocsBuilder.bulk_transform(
-  './docs',
-  base_url: 'https://myproject.io',
-  suffix: '.llm',
-  remove_duplicates: true,
-  generate_toc: true
-)
-
-# Generate llms.txt
-content = LlmDocsBuilder.generate_from_docs(
-  './docs',
-  base_url: 'https://myproject.io',
-  title: 'My Project'
-)
-```
-
 ## Serving Optimized Docs to AI Bots
 
 After using `bulk-transform` with `suffix: .llm`, configure your web server to serve optimized versions to AI bots:
@@ -225,27 +191,6 @@ location ~ ^/docs/(.*)\.md$ {
 }
 ```
 
-## Real-World Results: Karafka Framework
-
-**Before:** 140+ lines of custom transformation code
-
-**After:** 6 lines of configuration
-```yaml
-docs: ./online/docs
-base_url: https://karafka.io/docs
-convert_urls: true
-remove_comments: true
-remove_badges: true
-remove_frontmatter: true
-normalize_whitespace: true
-suffix: ""  # In-place for build pipeline
-```
-
-**Results:**
-- 93% average token reduction
-- 20-36x smaller files
-- Automated via GitHub Actions
-
 ## Docker Usage
 
 ```bash
@@ -285,43 +230,21 @@ layout: docs
 
 [Click here to see the complete API documentation](./api.md)
 
-```ruby
 api = API.new
 ```
 
-![Diagram](./diagram.png)
-```
-
 **After transformation (with default options):**
+
 ```markdown
 # API Documentation
 
 [complete API documentation](./api.md)
 
-```ruby
 api = API.new
 ```
-```
 
 **Token reduction:** ~40-60% depending on configuration
 
-## FAQ
-
-**Q: Do I need to use llms.txt?**
-No. The compare and transform commands work independently.
-
-**Q: Will this change how humans see my docs?**
-Not with default `suffix: .llm`. Separate files are served only to AI bots.
-
-**Q: Can I use this in my build pipeline?**
-Yes. Use `suffix: ""` for in-place transformation.
-
-**Q: How do I know if it's working?**
-Use `llm-docs-builder compare` to measure before and after.
-
-**Q: What about private documentation?**
-Use the `excludes` option to skip sensitive files.
-
 ## RAG Enhancement Features
 
 ### Heading Normalization
@@ -409,39 +332,6 @@ All compression features can be used individually for fine-grained control:
 - `convert_urls: true` - Convert `.html`/`.htm` URLs to `.md` format
 - `normalize_whitespace: true` - Reduce excessive blank lines and remove trailing whitespace
 
-### Example Usage
-
-```ruby
-# Fine-grained control
-LlmDocsBuilder.transform_markdown(
-  'README.md',
-  remove_frontmatter: true,
-  remove_badges: true,
-  remove_images: true,
-  simplify_links: true,
-  generate_toc: true,
-  normalize_whitespace: true
-)
-```
-
-Or configure via YAML:
-
-```yaml
-# llm-docs-builder.yml
-docs: ./docs
-base_url: https://myproject.io
-suffix: .llm
-
-# Pick exactly what you need
-remove_frontmatter: true
-remove_comments: true
-remove_badges: true
-remove_images: true
-simplify_links: true
-generate_toc: true
-normalize_whitespace: true
-```
-
 ## License
 
 Available as open source under the [MIT License](https://opensource.org/licenses/MIT).

data/lib/llm_docs_builder/transformers/heading_transformer.rb

@@ -38,8 +38,18 @@ module LlmDocsBuilder
         separator = options[:heading_separator] || ' / '
         heading_stack = []
         lines = content.lines
+        in_code_block = false
 
         transformed_lines = lines.map do |line|
+          # Track code block boundaries (fenced code blocks with ``` or ~~~)
+          if line.match?(/^(```|~~~)/)
+            in_code_block = !in_code_block
+            next line
+          end
+
+          # Skip heading processing if inside code block
+          next line if in_code_block
+
           # Match markdown headings (1-6 hash symbols followed by space and text)
           heading_match = line.match(/^(#+)\s+(.+)$/)
 

data/lib/llm_docs_builder/version.rb

@@ -2,5 +2,5 @@
 
 module LlmDocsBuilder
   # Current version of the LlmDocsBuilder gem
-  VERSION = '0.9.0'
+  VERSION = '0.9.2'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm-docs-builder
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.9.2
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -147,6 +147,7 @@ files:
 - lib/llm_docs_builder/version.rb
 - llm-docs-builder.gemspec
 - llm-docs-builder.yml.example
+- misc/diff.png
 - misc/logo.png
 - misc/logo_wide.png
 - renovate.json