jekyll-llms-output 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 77bd6b72574803bc37691dc39fed7ae2630293858a717ffcdff715df1d950a7a
+  data.tar.gz: cbf85998d24ee83072793940c227902068ae4cec56a767ec54eece3cde62ca1e
+SHA512:
+  metadata.gz: d8ad8e05490ecbeb45d9b3ea064473fdb40e6d4989b9349719c45340c9ed378e21bee730ffac30477d28e0f6c3765131aa7af9ea7dcc11b527bacd66767afa2a
+  data.tar.gz: 51f71c2b80c5ec81f48c173ea9e7f9746ed3b633ccc27c1a90cd9c39ee9a8e0768a4c4357fada0d95ea6966cf357253af7d9cf4e11ab7ca6d13b8d41e6823d1b

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## 0.1.0 - 2026-05-05
+
+- Initial release.
+- Generates `/llms.txt` from `_data/llms.yml` (curated) or auto from collections.
+- Generates `/llms-full.txt` by concatenating source bodies (Liquid rendered).
+- Per-document opt-out for `llms-full.txt` via `llms_output: false` in frontmatter.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abhinav Saxena
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,126 @@
+# jekyll-llms-output
+
+A Jekyll plugin that generates two files agents and LLM crawlers expect:
+
+- **`/llms.txt`** - an index of your content following the [llmstxt.org](https://llmstxt.org) format. Hand-curate it via `_data/llms.yml`, or let the plugin auto-generate from your collections.
+- **`/llms-full.txt`** - a single concatenated dump of all your post bodies (Liquid rendered, source markdown). Useful for agents that want to ingest your whole site in one fetch.
+
+Pairs nicely with [`jekyll-markdown-output`](https://github.com/abhinavs/jekyll-markdown-output), which writes per-page `.md` siblings.
+
+## Install
+
+```ruby
+group :jekyll_plugins do
+  gem "jekyll-llms-output"
+end
+```
+
+```yaml
+plugins:
+  - jekyll-llms-output
+```
+
+## Configure
+
+Defaults are sensible for a typical blog. Override via `_config.yml`:
+
+```yaml
+llms_output:
+  enabled: true                  # global on/off
+
+  index:
+    enabled: true                # write /llms.txt
+    output: /llms.txt
+    data: llms                   # reads _data/llms.yml if it exists
+    title: ~                     # default: site.title
+    description: ~               # default: site.description
+    collections: [posts]         # used in auto mode (no data file)
+
+  full:
+    enabled: true                # write /llms-full.txt
+    output: /llms-full.txt
+    collections: [posts]
+    pages: false
+    page_extensions: [.md, .markdown]
+    separator: "\n\n---\n\n"
+    include_url: true
+    include_date: true
+    respect_markdown_output: false  # if true, also honor jekyll-markdown-output's opt-out flag
+```
+
+## `/llms.txt` modes
+
+### Curated (recommended)
+
+Drop a `_data/llms.yml` in your site:
+
+```yaml
+title: Abhinav Saxena
+description: |
+  Personal website of Abhinav Saxena, a generalist with interests in
+  entrepreneurship, software engineering, and engineering leadership.
+
+sections:
+  - heading: Agent Resources
+    links:
+      - title: agents.md
+        url: https://www.abhinav.co/.well-known/agents.md
+        description: Context and instructions for coding agents
+      - title: skills.md
+        url: https://www.abhinav.co/.well-known/skills.md
+        description: Decision table mapping needs to specific posts
+
+  - heading: Featured Writing
+    links:
+      - title: Terminal is having a second life
+        url: https://www.abhinav.co/terminal-second-life
+        description: Why agentic tooling has pulled the terminal back to the centre of the dev workflow
+```
+
+The plugin renders this exact structure to `/llms.txt`. Curation is preserved, you can section by topic, and you control which links appear.
+
+### Auto
+
+If `_data/llms.yml` is absent, the plugin generates from `index.collections` - one `## Section` per collection, one bullet per document, with the `summary` (or excerpt) as the description.
+
+## `/llms-full.txt`
+
+Concatenates the source body of every document in `full.collections` (and pages if `full.pages: true`), with a `# Title` header per item and a `---` separator between them.
+
+```text
+# Hello World
+
+URL: https://example.com/2024/01/01/hello
+Date: 2024-01-01T09:00:00+00:00
+
+This is the body of the post.
+
+---
+
+# Another Post
+...
+```
+
+### Per-document opt-out
+
+Skip a single document from `llms-full.txt`:
+
+```yaml
+---
+title: Draft thinking
+llms_output: false
+---
+```
+
+## Compatibility
+
+- Jekyll 3.7+ and 4.x
+- Ruby 2.7+
+
+### GitHub Pages
+
+GitHub Pages restricts plugins to a whitelist; this gem isn't on it. Build your site in CI (Actions, Netlify, Cloudflare Pages, Vercel) and deploy `_site/` to GH Pages.
+
+## License
+
+MIT. See `LICENSE`.

data/lib/jekyll-llms-output/full_builder.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module Jekyll
+  module LlmsOutput
+    # Builds the body of /llms-full.txt: a concatenation of every document's
+    # source markdown, separated by a configurable delimiter, each with a
+    # small header (title + url) for context.
+    class FullBuilder
+      attr_reader :site, :options
+
+      def initialize(site, options)
+        @site = site
+        @options = options
+      end
+
+      def build
+        chunks = []
+
+        Array(options["collections"]).each do |coll_name|
+          collection = site.collections[coll_name.to_s]
+          next unless collection
+          collection.docs.each { |doc| chunks << render_doc(doc) }
+        end
+
+        if options["pages"]
+          exts = Array(options["page_extensions"]).map(&:downcase)
+          site.pages.each do |page|
+            next unless exts.include?(File.extname(page.path).downcase)
+            chunks << render_doc(page)
+          end
+        end
+
+        chunks.compact.join(options["separator"] || "\n\n---\n\n") + "\n"
+      end
+
+      private
+
+      def render_doc(doc)
+        return nil if doc.data["llms_output"] == false
+        return nil if doc.data["markdown_output"] == false && options["respect_markdown_output"]
+
+        url = absolute_url(doc.url)
+        title = doc.data["title"] || url
+        body = source_body(doc).strip
+        body = render_liquid(doc, body) if doc.data["render_with_liquid"] != false
+
+        header = +"# #{title}\n"
+        header << "\nURL: #{url}\n" if options["include_url"]
+        header << "Date: #{doc.data["date"].iso8601}\n" if options["include_date"] && doc.data["date"].respond_to?(:iso8601)
+        header << "\n"
+
+        header + body
+      end
+
+      def source_path(doc)
+        File.expand_path(doc.path.to_s, site.source)
+      end
+
+      def source_body(doc)
+        raw = File.read(source_path(doc), encoding: "UTF-8")
+        parts = raw.split(/^---\s*$\n/, 3)
+        parts.length >= 3 ? parts[2] : raw
+      end
+
+      def render_liquid(doc, body)
+        info = {
+          filters:   [Jekyll::Filters],
+          registers: { site: site, page: doc.to_liquid },
+        }
+        template = site.liquid_renderer.file(source_path(doc)).parse(body)
+        template.render!(site.site_payload.merge("page" => doc.to_liquid), info)
+      rescue StandardError => e
+        rel = doc.respond_to?(:relative_path) ? doc.relative_path : doc.path
+        Jekyll.logger.warn("LlmsOutput:", "render failed for #{rel}: #{e.message}")
+        body
+      end
+
+      def absolute_url(url)
+        site_url = site.config["url"]
+        return url if url.to_s.start_with?("http")
+        site_url ? "#{site_url}#{url}" : url
+      end
+    end
+  end
+end

data/lib/jekyll-llms-output/generator.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require "fileutils"
+
+module Jekyll
+  module LlmsOutput
+    DEFAULTS = {
+      "enabled" => true,
+      "index" => {
+        "enabled"     => true,
+        "output"      => "/llms.txt",
+        "data"        => "llms",
+        "title"       => nil,
+        "description" => nil,
+        "collections" => ["posts"],
+      },
+      "full" => {
+        "enabled"          => true,
+        "output"           => "/llms-full.txt",
+        "collections"      => ["posts"],
+        "pages"            => false,
+        "page_extensions"  => [".md", ".markdown"],
+        "separator"        => "\n\n---\n\n",
+        "include_url"      => true,
+        "include_date"     => true,
+        "respect_markdown_output" => false,
+      },
+    }.freeze
+
+    def self.config_for(site)
+      user = site.config["llms_output"] || {}
+      merged = deep_merge(DEFAULTS, user)
+      merged
+    end
+
+    def self.deep_merge(a, b)
+      a.merge(b) do |_, av, bv|
+        av.is_a?(Hash) && bv.is_a?(Hash) ? deep_merge(av, bv) : bv
+      end
+    end
+
+    def self.write_all(site)
+      config = config_for(site)
+      return unless config["enabled"]
+
+      if config["index"]["enabled"]
+        body = IndexBuilder.new(site, config["index"]).build
+        write_file(site, config["index"]["output"], body)
+        Jekyll.logger.info("LlmsOutput:", "wrote #{config["index"]["output"]}")
+      end
+
+      if config["full"]["enabled"]
+        body = FullBuilder.new(site, config["full"]).build
+        write_file(site, config["full"]["output"], body)
+        Jekyll.logger.info("LlmsOutput:", "wrote #{config["full"]["output"]}")
+      end
+    end
+
+    def self.write_file(site, output_path, body)
+      path = File.join(site.dest, output_path)
+      FileUtils.mkdir_p(File.dirname(path))
+      File.write(path, body)
+    end
+  end
+end
+
+Jekyll::Hooks.register :site, :post_write do |site|
+  Jekyll::LlmsOutput.write_all(site)
+end

data/lib/jekyll-llms-output/index_builder.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module Jekyll
+  module LlmsOutput
+    # Builds the body of /llms.txt.
+    #
+    # Two modes:
+    #   1. Data-driven: site.data[<key>] holds a hash with title / description /
+    #      sections. The plugin renders that structure to llmstxt.org format.
+    #   2. Auto: no data hash present. The plugin generates one section per
+    #      configured collection with a bullet for each document.
+    #
+    # llmstxt.org spec we follow:
+    #   # Title
+    #   > Optional one-line description
+    #   ## Section heading
+    #   - [Title](url): optional description
+    class IndexBuilder
+      attr_reader :site, :options
+
+      def initialize(site, options)
+        @site = site
+        @options = options
+      end
+
+      def build
+        data_key = options["data"]
+        data_hash = data_key && site.data[data_key.to_s]
+
+        if data_hash.is_a?(Hash) && data_hash["sections"]
+          render_from_data(data_hash)
+        else
+          render_auto
+        end
+      end
+
+      private
+
+      def render_from_data(data)
+        out = +""
+        title = data["title"] || site.config["title"]
+        description = data["description"] || site.config["description"]
+
+        out << "# #{title}\n\n" if title && !title.to_s.empty?
+        if description && !description.to_s.empty?
+          out << "> #{description.to_s.strip}\n\n"
+        end
+
+        Array(data["sections"]).each do |section|
+          heading = section["heading"] || section["title"]
+          next unless heading
+          out << "## #{heading}\n\n"
+          Array(section["links"]).each do |link|
+            out << format_link(link["title"], link["url"], link["description"])
+          end
+          out << "\n"
+        end
+
+        out.strip + "\n"
+      end
+
+      def render_auto
+        out = +""
+        title = options["title"] || site.config["title"]
+        description = options["description"] || site.config["description"]
+
+        out << "# #{title}\n\n" if title && !title.to_s.empty?
+        if description && !description.to_s.empty?
+          out << "> #{description.to_s.strip}\n\n"
+        end
+
+        Array(options["collections"]).each do |coll_name|
+          collection = site.collections[coll_name.to_s]
+          next unless collection && !collection.docs.empty?
+          out << "## #{section_heading_for(coll_name)}\n\n"
+          docs_in_render_order(collection.docs).each do |doc|
+            url = absolute_url(doc.url)
+            out << format_link(doc.data["title"] || url, url, summary_for(doc))
+          end
+          out << "\n"
+        end
+
+        out.strip + "\n"
+      end
+
+      def docs_in_render_order(docs)
+        # Newest first when docs have a date.
+        docs.sort_by { |d| d.respond_to?(:date) && d.date ? d.date : Time.at(0) }.reverse
+      end
+
+      def section_heading_for(name)
+        # "posts" -> "Posts", "design_notes" -> "Design notes"
+        s = name.to_s.tr("_-", " ")
+        s.empty? ? "Items" : s[0].upcase + s[1..]
+      end
+
+      def format_link(title, url, description)
+        line = "- [#{title}](#{url})"
+        line += ": #{description.to_s.strip}" if description && !description.to_s.empty?
+        line + "\n"
+      end
+
+      def summary_for(doc)
+        s = doc.data["summary"]
+        return s.strip if s.is_a?(String) && !s.strip.empty?
+        excerpt = doc.data["excerpt"]
+        return nil if excerpt.nil?
+        text = excerpt.respond_to?(:content) ? excerpt.content : excerpt
+        text = text.to_s.gsub(/<[^>]+>/, "").strip
+        text.empty? ? nil : text.split(/\n\n/).first.to_s.strip
+      end
+
+      def absolute_url(url)
+        site_url = site.config["url"]
+        return url if url.to_s.start_with?("http")
+        site_url ? "#{site_url}#{url}" : url
+      end
+    end
+  end
+end

data/lib/jekyll-llms-output/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Jekyll
+  module LlmsOutput
+    VERSION = "0.1.0"
+  end
+end

data/lib/jekyll-llms-output.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "jekyll"
+
+require_relative "jekyll-llms-output/version"
+require_relative "jekyll-llms-output/index_builder"
+require_relative "jekyll-llms-output/full_builder"
+require_relative "jekyll-llms-output/generator"

metadata

@@ -0,0 +1,122 @@
+--- !ruby/object:Gem::Specification
+name: jekyll-llms-output
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Abhinav Saxena
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-05-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: jekyll
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.7'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.7'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: kramdown-parser-gfm
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+description: |
+  A Jekyll plugin that writes /llms.txt (an index following llmstxt.org)
+  and /llms-full.txt (a concatenated full-text dump of your content).
+  Supports a hand-curated _data/llms.yml structure or auto-generation
+  from configured collections.
+email:
+- abhinav061@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/jekyll-llms-output.rb
+- lib/jekyll-llms-output/full_builder.rb
+- lib/jekyll-llms-output/generator.rb
+- lib/jekyll-llms-output/index_builder.rb
+- lib/jekyll-llms-output/version.rb
+homepage: https://github.com/abhinavs/jekyll-llms-output
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/abhinavs/jekyll-llms-output
+  source_code_uri: https://github.com/abhinavs/jekyll-llms-output/tree/main
+  bug_tracker_uri: https://github.com/abhinavs/jekyll-llms-output/issues
+  changelog_uri: https://github.com/abhinavs/jekyll-llms-output/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Generate /llms.txt and /llms-full.txt for a Jekyll site.
+test_files: []