jekyll-markdown-output 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: eef5e7fe6ffaa6a867b61c55c4e7da552745caac57be7f45c79402ddfe562283
+  data.tar.gz: 1c12840c040cf8f0ead3234d542dd702e3496fba1bb2bbfaeb067b4c464508c8
+SHA512:
+  metadata.gz: 64f3a07d2967ee838f3ce05dd3d58721a6028c7e905f29a00b97059b00982a3635fb9c0e36262f2bbe7d64f33b2ea3a0aafedb9a12714e6159e82438c8d02817
+  data.tar.gz: 4de984d5b6570bb87f89b018f7095521b2efc05bdb5fdd04809931e82489c88a206b105220f0d8c1c841d2d1cdf0d70efd3a39739f3ddada219ded183944193f

data/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## 0.1.1 - 2026-05-05
+
+- Fix: when a post had no `summary` set, the fallback to `doc.data["excerpt"]`
+  serialized the entire `Jekyll::Excerpt` Ruby object graph into the
+  frontmatter. The fallback now coerces the excerpt to plain text and strips
+  HTML tags.
+- Add an RSpec test suite (unit + integration against a fixture Jekyll site).
+
+## 0.1.0
+
+- Initial release.
+- Generates a `.md` sibling for every document in the configured collections.
+- Mirrors Markdown-sourced `site.pages` (e.g. `index.md`, `about.md`) too.
+  Disable with `pages: false`.
+- Minimal YAML frontmatter (title, date, url, summary, tags, category, author).
+- Per-document opt-out via `markdown_output: false` in frontmatter.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abhinav Saxena
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,148 @@
+# jekyll-markdown-output
+
+A Jekyll plugin that emits a `.md` sibling for every post (or any document in a configured collection), so AI agents, LLM crawlers, and other machine consumers can fetch clean Markdown instead of parsing HTML.
+
+For a post rendered at `/foo`, this plugin also writes `/foo.md` containing:
+
+- a small YAML frontmatter block (title, date, url, summary, tags, category, author)
+- the post's source Markdown with Liquid rendered
+
+No HTML conversion. No layout chrome. No nav, footer, theme toggles, or analytics scripts.
+
+### Before / after
+
+```
+_site/
+  foo.html              <- as before
+  foo.md                <- new: clean Markdown, same URL
+  posts/
+    hello.html
+    hello.md
+```
+
+Agents fetching `/foo.md` get the source content with a small frontmatter block. Browsers fetching `/foo` get the rendered HTML, untouched.
+
+## Why
+
+Agents that read your site spend tokens parsing HTML and stripping boilerplate. Serving a `.md` twin is the smallest change that gives them the actual content. It is the same pattern used by Anthropic's docs, Stripe, and a growing set of agent-friendly sites.
+
+## Install
+
+Add to your `Gemfile`:
+
+```ruby
+group :jekyll_plugins do
+  gem "jekyll-markdown-output"
+end
+```
+
+Then in `_config.yml`:
+
+```yaml
+plugins:
+  - jekyll-markdown-output
+```
+
+## Configure
+
+Defaults are sensible for a typical blog. Override via `_config.yml`:
+
+```yaml
+markdown_output:
+  enabled: true                      # set false to disable globally
+  collections: [posts]               # which collections to mirror
+  pages: true                        # also mirror site.pages
+  page_extensions: [.md, .markdown]  # which page sources count as Markdown
+  extension: .md                     # output extension
+  include_title_heading: true        # prepend "# Title" to body
+  frontmatter_keys:                  # which fields to include
+    - title
+    - date
+    - url
+    - summary
+    - tags
+    - category
+    - author
+```
+
+`pages: true` (the default) emits `.md` for top-level Markdown files such
+as `index.md`, `about.md`, `now.md`. HTML-sourced pages are skipped: if
+you want a `.md` twin for a page, write it in Markdown.
+
+### Per-document opt-out
+
+Add to a single post's frontmatter to skip it:
+
+```yaml
+---
+title: Draft thinking
+markdown_output: false
+---
+```
+
+## URL mapping
+
+| Source URL | Generated file |
+|---|---|
+| `/foo`     | `/foo.md`           |
+| `/a/foo`   | `/a/foo.md`         |
+| `/foo/`    | `/foo/index.md`     |
+| `/`        | `/index.md`         |
+
+## Output shape
+
+```markdown
+---
+title: Terminal is having a second life
+date: '2025-09-12T00:00:00+05:30'
+url: https://www.abhinav.co/terminal-second-life
+summary: How agentic coding tools have pulled the terminal back to the centre of the developer workflow.
+tags:
+- Terminal
+- Tools
+category: technology
+author: Abhinav Saxena
+---
+
+# Terminal is having a second life
+
+For years the terminal was the place you only opened to run a build...
+```
+
+## How it works
+
+The plugin registers a `:site, :post_write` hook that runs after Jekyll has finished its main build. For each document in the configured collections (and each Markdown-sourced page if `pages: true`), it reads the original source from disk, strips the frontmatter, optionally renders Liquid against the document context, and writes a `.md` file directly into `_site/`.
+
+Because output goes through `File.write` rather than Jekyll's renderer, the file never passes through layouts, the Markdown-to-HTML converter, or any other plugin's hooks. The body stays as Markdown; Liquid (`{{ site.url }}`, `{% include %}`) resolves against the live site context.
+
+## Compatibility
+
+- Jekyll 3.7+ and 4.x
+- Ruby 2.7+
+
+### GitHub Pages
+
+GitHub Pages restricts Jekyll plugins to a [whitelist](https://pages.github.com/versions/), and `jekyll-markdown-output` is not on it. If you host on GH Pages, you have two options:
+
+1. Build the site yourself in CI (GitHub Actions, Netlify, Cloudflare Pages, Vercel) and deploy the built `_site/` to GH Pages, instead of relying on GH Pages' own Jekyll build.
+2. Skip this plugin and serve `.html` only.
+
+Cloudflare Pages, Netlify, Vercel, and self-hosted builds run the plugin without restriction.
+
+## FAQ
+
+**How is this different from `llms.txt`?**
+
+`llms.txt` is one root file listing your content. This plugin emits a per-page `.md` next to each `.html`, so an agent that lands on `/foo` can fetch `/foo.md` directly without consulting an index. The two compose: ship both if you want.
+
+**Why not just convert the rendered HTML back to Markdown?**
+
+The HTML has already gone through layouts, includes, theme chrome, syntax highlighting wrappers, and possibly a markdown converter that drops information (smart quotes, ID anchors). Round-tripping is lossy. Reading the source is faithful.
+
+**Will it slow my build down?**
+
+No measurable cost on a site with hundreds of posts. The hook runs once after `:site, :post_write` and writes files in a tight loop.
+
+## License
+
+MIT. See `LICENSE`.

data/lib/jekyll-markdown-output/generator.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "fileutils"
+
+module Jekyll
+  module MarkdownOutput
+    DEFAULTS = {
+      "enabled"               => true,
+      "collections"           => ["posts"],
+      "pages"                 => true,
+      "page_extensions"       => [".md", ".markdown"],
+      "extension"             => ".md",
+      "include_title_heading" => true,
+      "frontmatter_keys"      => MarkdownPage::DEFAULT_FRONTMATTER_KEYS,
+    }.freeze
+
+    def self.config_for(site)
+      DEFAULTS.merge(site.config["markdown_output"] || {})
+    end
+
+    def self.write_all(site)
+      config = config_for(site)
+      return unless config["enabled"]
+
+      written = 0
+
+      Array(config["collections"]).each do |coll_name|
+        collection = site.collections[coll_name.to_s]
+        unless collection
+          Jekyll.logger.warn("MarkdownOutput:", "collection '#{coll_name}' not found, skipping")
+          next
+        end
+        collection.docs.each { |doc| written += 1 if write_one(site, doc, config) }
+      end
+
+      if config["pages"]
+        exts = Array(config["page_extensions"]).map(&:downcase)
+        site.pages.each do |page|
+          next unless exts.include?(File.extname(page.path).downcase)
+          written += 1 if write_one(site, page, config)
+        end
+      end
+
+      Jekyll.logger.info("MarkdownOutput:", "wrote #{written} markdown file(s)")
+    end
+
+    def self.write_one(site, source, config)
+      return false if source.data["markdown_output"] == false
+
+      page = MarkdownPage.new(site, source, config)
+      path = page.destination
+      FileUtils.mkdir_p(File.dirname(path))
+      File.write(path, page.to_s)
+      true
+    end
+  end
+end
+
+Jekyll::Hooks.register :site, :post_write do |site|
+  Jekyll::MarkdownOutput.write_all(site)
+end

data/lib/jekyll-markdown-output/markdown_page.rb

@@ -0,0 +1,136 @@
+# frozen_string_literal: true
+
+module Jekyll
+  module MarkdownOutput
+    # Builds the Markdown bytes for a single source document.
+    # Not a Jekyll::Page on purpose: we write directly to _site/
+    # in a post_write hook so the converter and layout pipeline
+    # cannot touch the output.
+    class MarkdownPage
+      DEFAULT_FRONTMATTER_KEYS = %w[title date url summary tags category author].freeze
+
+      attr_reader :doc, :site, :options
+
+      def initialize(site, doc, options = {})
+        @site = site
+        @doc = doc
+        @options = options
+      end
+
+      # Absolute path on disk where this file should be written.
+      def destination
+        File.join(@site.dest, relative_destination)
+      end
+
+      # Path relative to the site root, e.g. "/foo.md".
+      def relative_destination
+        ext = @options.fetch("extension", ".md")
+        url = @doc.url
+        if url.end_with?("/")
+          File.join(url, "index#{ext}")
+        else
+          dir = File.dirname(url)
+          base = File.basename(url, ".*")
+          File.join(dir, "#{base}#{ext}")
+        end
+      end
+
+      def to_s
+        parts = []
+        fm = build_frontmatter
+        parts << "---\n#{fm}---" unless fm.empty?
+        if @options.fetch("include_title_heading", true) && @doc.data["title"]
+          parts << "# #{@doc.data["title"]}"
+        end
+        parts << rendered_source.to_s.strip
+        "#{parts.join("\n\n")}\n"
+      end
+
+      private
+
+      def build_frontmatter
+        keys = @options["frontmatter_keys"] || DEFAULT_FRONTMATTER_KEYS
+        url = @doc.url
+        site_url = @site.config["url"]
+        url = "#{site_url}#{url}" if site_url && !url.start_with?("http")
+
+        candidates = {
+          "title"    => @doc.data["title"],
+          "date"     => format_date(@doc.data["date"]),
+          "url"      => url,
+          "summary"  => extract_summary,
+          "tags"     => Array(@doc.data["tags"]).reject { |t| t.to_s.empty? },
+          "category" => @doc.data["category"],
+          "author"   => @doc.data["author"] || @site.config["author"],
+        }
+
+        picked = keys.each_with_object({}) do |key, h|
+          v = candidates[key]
+          next if v.nil?
+          next if v.respond_to?(:empty?) && v.empty?
+          h[key] = v
+        end
+
+        return "" if picked.empty?
+
+        YAML.dump(picked).sub(/\A---\s*\n/, "")
+      end
+
+      def format_date(date)
+        return nil if date.nil?
+        date.respond_to?(:iso8601) ? date.iso8601 : date.to_s
+      end
+
+      # `summary` falls back to the document's excerpt when not set in
+      # frontmatter. Excerpt may be a Jekyll::Excerpt object (whose YAML dump
+      # is a huge object graph) or already a string. Coerce to plain text and
+      # strip tags so it's safe to embed in YAML frontmatter.
+      def extract_summary
+        s = @doc.data["summary"]
+        return s.strip if s.is_a?(String) && !s.strip.empty?
+
+        excerpt = @doc.data["excerpt"]
+        return nil if excerpt.nil?
+        text = excerpt.respond_to?(:content) ? excerpt.content : excerpt
+        text = text.to_s.gsub(/<[^>]+>/, "").strip
+        text.empty? ? nil : text
+      end
+
+      # Resolve to an absolute path on disk. Document#path is already
+      # absolute; Page#path is relative to the site source.
+      def source_path
+        File.expand_path(@doc.path.to_s, @site.source)
+      end
+
+      # Re-read the source body from disk. By the time post_write fires,
+      # Jekyll has overwritten doc.content with the converted HTML.
+      # Force UTF-8 because some build environments (e.g. Cloudflare Pages)
+      # default to US-ASCII, which breaks regex splits on non-ASCII bytes.
+      def source_body
+        raw = File.read(source_path, encoding: "UTF-8")
+        parts = raw.split(/^---\s*$\n/, 3)
+        parts.length >= 3 ? parts[2] : raw
+      end
+
+      def rendered_source
+        body = source_body
+        # Skip Liquid only when the document explicitly opted out. We do not
+        # rely on Document#render_with_liquid? here because Jekyll mutates
+        # that flag to false after the main render pass, and our post_write
+        # hook runs strictly after that.
+        return body if @doc.data["render_with_liquid"] == false
+
+        info = {
+          filters:   [Jekyll::Filters],
+          registers: { site: @site, page: @doc.to_liquid },
+        }
+        template = @site.liquid_renderer.file(source_path).parse(body)
+        template.render!(@site.site_payload.merge("page" => @doc.to_liquid), info)
+      rescue StandardError => e
+        rel = @doc.respond_to?(:relative_path) ? @doc.relative_path : @doc.path
+        Jekyll.logger.warn("MarkdownOutput:", "render failed for #{rel}: #{e.message}")
+        source_body
+      end
+    end
+  end
+end

data/lib/jekyll-markdown-output/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Jekyll
+  module MarkdownOutput
+    VERSION = "0.1.1"
+  end
+end

data/lib/jekyll-markdown-output.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "jekyll"
+require "yaml"
+
+require_relative "jekyll-markdown-output/version"
+require_relative "jekyll-markdown-output/markdown_page"
+require_relative "jekyll-markdown-output/generator"

metadata

@@ -0,0 +1,121 @@
+--- !ruby/object:Gem::Specification
+name: jekyll-markdown-output
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Abhinav Saxena
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-05-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: jekyll
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.7'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.7'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: kramdown-parser-gfm
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.12'
+description: |
+  A Jekyll generator that writes a .md file alongside each rendered HTML page,
+  so AI agents and crawlers can fetch clean Markdown (with a small machine-
+  friendly frontmatter block) instead of parsing HTML. Configurable per
+  collection.
+email:
+- abhinav061@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/jekyll-markdown-output.rb
+- lib/jekyll-markdown-output/generator.rb
+- lib/jekyll-markdown-output/markdown_page.rb
+- lib/jekyll-markdown-output/version.rb
+homepage: https://github.com/abhinavs/jekyll-markdown-output
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/abhinavs/jekyll-markdown-output
+  source_code_uri: https://github.com/abhinavs/jekyll-markdown-output/tree/main
+  bug_tracker_uri: https://github.com/abhinavs/jekyll-markdown-output/issues
+  changelog_uri: https://github.com/abhinavs/jekyll-markdown-output/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.7.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Emit a Markdown sibling for every Jekyll post or note, for agents and LLMs.
+test_files: []