llm-docs-builder 0.9.4 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 66983a07a7271c966999350d03fbde1b1080ef0ac05a7209452cbb8720074e1b
-  data.tar.gz: 5d7cb81a700db6a43c17145af56ffcd2f6cea4a9a5182d4a2b14fd7772c8ee07
+  metadata.gz: e6407836216f436b728247009f614ea4ea5c2b4de0edf855717129460df4b309
+  data.tar.gz: 8a556fc0b6307529f5c615c05b082521bd021e9e9f34ca30c8bb21d22b21deb2
 SHA512:
-  metadata.gz: 416b6f94c7e7dbac3bf3e6ae8793adcf9893d685aec29cc83665c2f9a6ab312bd422542e0074bbeec961f23740698aac6e517ebb7c87f3cb0fef3c8c6067c662
-  data.tar.gz: 2fba65092d82dbbeea60ce05317a781ed82f20367313d4eab9a69425e6e51f70ec106ca5f56218bbe255d1485752732de2a876359fde7018e706b948d51053fb
+  metadata.gz: 3a0b657545415c35187fa1595f3fcc5bd5c27a0c1a292bf00635d3c6f14221ac0a254f008f83acc1f7351ef76c6649e6b0540ac585cd64a670b05ef725b0308e
+  data.tar.gz: 68e95142e374ebae3c292db724a9163c396fd423eaa04983bbdaefd6f6cbb0bc72822dfbc5af9fd8f357f1545320db70bf5a6c8a2413fc5f796e73195adcdd13

data/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.10.0 (2025-10-27)
+- [Feature] **llms.txt Specification Compliance** - Updated output format to fully comply with the llms.txt specification from llmstxt.org.
+  - **Metadata Format**: Metadata now appears within the description field using parentheses and comma separators: `- [title](url): description (tokens:450, updated:2025-10-13, priority:high)`
+  - **Optional Descriptions**: Parser now correctly handles links without descriptions: `- [title](url)` per spec
+  - **Multi-Section Support**: Documents automatically organized into `Documentation`, `Examples`, and `Optional` sections based on priority
+  - **Body Content Support**: Added optional `body` config parameter for custom content between description and sections
+  - Priority-based categorization: 1-3 → Documentation, 4-5 → Examples, 6-7 → Optional
+  - Empty sections are automatically omitted from output
+  - Updated parser regex from `/^[-*]\s*\[([^\]]+)\]\(([^)]+)\):\s*(.*)$/m` to `/^[-*]\s*\[([^\]]+)\]\(([^)]+)\)(?::\s*([^\n]*))?$/` to make descriptions optional
+  - Fixed multiline regex greedy matching issue that was capturing only one link per section
+- [Test] Added comprehensive test suite for spec compliance (8 new parser tests, 7 new generator tests)
+- [Docs] Updated README with multi-section organization examples and body content usage
+- **Breaking Change**: Metadata format has changed from `tokens:450 updated:2025-10-13` to `(tokens:450, updated:2025-10-13)` for spec compliance
+
 ## 0.9.4 (2025-10-27)
 - [Feature] **Auto-Exclude Hidden Directories** - Hidden directories (starting with `.`) are now automatically excluded by default to prevent noise from `.git`, `.lint`, `.github`, etc.
   - Adds `include_hidden: false` as default behavior

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    llm-docs-builder (0.9.4)
+    llm-docs-builder (0.10.0)
       zeitwerk (~> 2.6)
 
 GEM

data/README.md

@@ -109,6 +109,7 @@ docs: ./docs
 base_url: https://myproject.io
 title: My Project
 description: Brief description
+body: Optional body content between description and sections
 output: llms.txt
 suffix: .llm
 verbose: false
@@ -289,9 +290,9 @@ Generate enriched llms.txt files with token counts, timestamps, and priority lab
 
 **Enhanced llms.txt (with metadata enabled):**
 ```markdown
-- [Getting Started](https://myproject.io/docs/Getting-Started.md) tokens:450 updated:2025-10-13 priority:high
-- [Configuration](https://myproject.io/docs/Configuration.md) tokens:2800 updated:2025-10-12 priority:high
-- [Advanced Topics](https://myproject.io/docs/Advanced.md) tokens:5200 updated:2025-09-15 priority:medium
+- [Getting Started](https://myproject.io/docs/Getting-Started.md): Quick start guide (tokens:450, updated:2025-10-13, priority:high)
+- [Configuration](https://myproject.io/docs/Configuration.md): Configuration options (tokens:2800, updated:2025-10-12, priority:high)
+- [Advanced Topics](https://myproject.io/docs/Advanced.md): Deep dive topics (tokens:5200, updated:2025-09-15, priority:medium)
 ```
 
 **Benefits:**
@@ -309,6 +310,68 @@ include_priority: true      # Show priority labels (high/medium/low)
 calculate_compression: true # Show compression ratios (slower, requires transformation)
 ```
 
+**Note:** Metadata is formatted according to the llms.txt specification, appearing within the description field using parentheses and comma separators for spec compliance.
+
+### Multi-Section Organization
+
+Documents are automatically organized into multiple sections based on priority, following the llms.txt specification:
+
+**Priority-based categorization:**
+- **Documentation** (priority 1-3): Essential docs like README, getting started guides, user guides
+- **Examples** (priority 4-5): Tutorials and example files
+- **Optional** (priority 6-7): Advanced topics and reference documentation
+
+**Example output:**
+```markdown
+# My Project
+
+> Project description
+
+## Documentation
+
+- [README](README.md): Main documentation
+- [Getting Started](getting-started.md): Quick start guide
+
+## Examples
+
+- [Basic Tutorial](tutorial.md): Step-by-step tutorial
+- [Code Examples](examples.md): Example code
+
+## Optional
+
+- [Advanced Topics](advanced.md): Deep dive into advanced features
+- [API Reference](reference.md): Complete API reference
+```
+
+Empty sections are automatically omitted. The "Optional" section aligns with the llms.txt spec for marking secondary content that can be skipped when context windows are limited.
+
+### Body Content
+
+Add custom body content between the description and documentation sections:
+
+```yaml
+# llm-docs-builder.yml
+title: My Project
+description: Brief description
+body: |
+  This framework is built on Ruby and focuses on performance.
+  Key concepts: streaming, batching, and parallel processing.
+docs: ./docs
+```
+
+This produces:
+```markdown
+# My Project
+
+> Brief description
+
+This framework is built on Ruby and focuses on performance.
+Key concepts: streaming, batching, and parallel processing.
+
+## Documentation
+...
+```
+
 ## Advanced Compression Options
 
 All compression features can be used individually for fine-grained control:

data/lib/llm_docs_builder/config.rb

@@ -61,6 +61,7 @@ module LlmDocsBuilder
         base_url: options[:base_url] || self['base_url'],
         title: options[:title] || self['title'],
         description: options[:description] || self['description'],
+        body: options[:body] || self['body'],
         output: options[:output] || self['output'] || 'llms.txt',
         convert_urls: if options.key?(:convert_urls)
                         options[:convert_urls]

data/lib/llm_docs_builder/generator.rb

@@ -210,7 +210,11 @@ module LlmDocsBuilder
 
     # Constructs llms.txt content from analyzed documentation files
     #
-    # Combines title, description, and documentation links into formatted output
+    # Combines title, description, body content, and documentation links into formatted output.
+    # Organizes documents into sections based on priority:
+    # - Priority 1-3: Documentation (essential docs like README, getting started)
+    # - Priority 4-5: Examples (tutorials, example files)
+    # - Priority 6-7: Optional (advanced topics, reference docs)
     #
     # @param docs [Array<Hash>] analyzed file metadata
     # @return [String] formatted llms.txt content
@@ -224,31 +228,60 @@ module LlmDocsBuilder
       content << "> #{description}" if description
       content << ''
 
-      if docs.any?
-        content << '## Documentation'
+      # Add optional body content
+      if options[:body] && !options[:body].empty?
+        content << options[:body]
         content << ''
+      end
 
-        docs.each do |doc|
-          url = build_url(doc[:path])
-          line = if doc[:description] && !doc[:description].empty?
-                   "- [#{doc[:title]}](#{url}): #{doc[:description]}"
-                 else
-                   "- [#{doc[:title]}](#{url})"
-                 end
-
-          # Append metadata if enabled
-          if options[:include_metadata]
-            metadata_parts = []
-            metadata_parts << "tokens:#{doc[:tokens]}" if doc[:tokens]
-            metadata_parts << "compression:#{doc[:compression]}" if doc[:compression]
-            metadata_parts << "updated:#{doc[:updated]}" if doc[:updated]
-            metadata_parts << priority_label(doc[:priority]) if options[:include_priority]
-
-            line += " #{metadata_parts.join(' ')}" unless metadata_parts.empty?
+      if docs.any?
+        # Categorize docs by priority into sections
+        sections = {
+          'Documentation' => docs.select { |d| d[:priority] <= 3 },
+          'Examples' => docs.select { |d| d[:priority] >= 4 && d[:priority] <= 5 },
+          'Optional' => docs.select { |d| d[:priority] >= 6 }
+        }
+
+        # Build each section (skip empty ones)
+        sections.each do |section_name, section_docs|
+          next if section_docs.empty?
+
+          content << "## #{section_name}"
+          content << ''
+
+          section_docs.each do |doc|
+            url = build_url(doc[:path])
+
+            # Build metadata string if enabled
+            metadata_str = nil
+            if options[:include_metadata]
+              metadata_parts = []
+              metadata_parts << "tokens:#{doc[:tokens]}" if doc[:tokens]
+              metadata_parts << "compression:#{doc[:compression]}" if doc[:compression]
+              metadata_parts << "updated:#{doc[:updated]}" if doc[:updated]
+              metadata_parts << priority_label(doc[:priority]) if options[:include_priority]
+
+              metadata_str = "(#{metadata_parts.join(', ')})" unless metadata_parts.empty?
+            end
+
+            # Build line according to spec: - [title](url): description (metadata)
+            line = if doc[:description] && !doc[:description].empty?
+                     base = "- [#{doc[:title]}](#{url}): #{doc[:description]}"
+                     metadata_str ? "#{base} #{metadata_str}" : base
+                   else
+                     # No description: - [title](url) (metadata)
+                     base = "- [#{doc[:title]}](#{url})"
+                     metadata_str ? "#{base}: #{metadata_str}" : base
+                   end
+
+            content << line
           end
 
-          content << line
+          content << ''
         end
+
+        # Remove trailing empty line
+        content.pop if content.last == ''
       end
 
       "#{content.join("\n")}\n"

data/lib/llm_docs_builder/parser.rb

@@ -85,7 +85,7 @@ module LlmDocsBuilder
 
     # Extracts markdown links from section content into structured format
     #
-    # Scans for markdown list items with links and descriptions. Returns raw content
+    # Scans for markdown list items with links and optional descriptions. Returns raw content
     # if no links are found in the expected format.
     #
     # @param content [String] raw section content
@@ -93,11 +93,13 @@ module LlmDocsBuilder
     def parse_section_content(content)
       links = []
 
-      content.scan(/^[-*]\s*\[([^\]]+)\]\(([^)]+)\):\s*(.*)$/m) do |title, url, description|
+      # Updated regex: description is optional (non-capturing group with ?)
+      # Use [^\n]* instead of .* to avoid matching across lines
+      content.scan(/^[-*]\s*\[([^\]]+)\]\(([^)]+)\)(?::\s*([^\n]*))?$/) do |title, url, description|
         links << {
           title: title,
           url: url,
-          description: description.strip
+          description: description&.strip || ''
         }
       end
 

data/lib/llm_docs_builder/version.rb

@@ -2,5 +2,5 @@
 
 module LlmDocsBuilder
   # Current version of the LlmDocsBuilder gem
-  VERSION = '0.9.4'
+  VERSION = '0.10.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm-docs-builder
 version: !ruby/object:Gem::Version
-  version: 0.9.4
+  version: 0.10.0
 platform: ruby
 authors:
 - Maciej Mensfeld