bridgetown-html-to-markdown 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 52d1a9575cba29d63575e50ed5295e36b5dc1d25f1e06bd24b2d322d1ae143e5
+  data.tar.gz: 258b703bc73609e771810e4abfb3bf74e1a4c39528226e4aba5107b1921bd92f
+SHA512:
+  metadata.gz: 2c7c8169b978df01be6805a3e2cfa076e20307c35d7fcbed07e1fd1c943558822c438a23618ed6bc52be50d1554b230752d6ef7f909b9266f83bbaffa83f2cb7
+  data.tar.gz: e58159f11a5a382a4c01cd04340222c4b3b2cf7b3d0c2a0122d8ce687aadc858ef155db06ef14b2f77e751bf37ff055f60f458c5a97f0375131b31add4f5b872

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-09-07
+
+### Added
+- Automated generation of `.md` Markdown files for every rendered `.html` file upon site build.
+- Dual file generation for nested pages (e.g. `categories/index.md` and `categories.md`).
+- HTML cleanup pipeline before conversion:
+  - Custom regex and string pattern replacements.
+  - Dialog stripping with configurable selector preservation (`exclude_dialogs`, `preserve_dialog_selectors`).
+  - SVG label extraction (`aria-label`, `alt`, `<title>`) and inline SVG stripping (`extract_svg_labels`).
+  - Spacing cleanup between inline tags and adjacent links (`clean_spacing`).
+- File inclusion and exclusion filters (`include_files`, `exclude_files`).
+- Explicit UTF-8 encoding for file operations and registration of `text/markdown; charset=utf-8` MIME type for Rack servers.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Marc Heiligers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,81 @@
+# BridgetownHtmlToMarkdown
+
+A [Bridgetown](https://www.bridgetownrb.com/) plugin that automatically generates a `.md` Markdown file for every rendered `.html` file during the site build process, powered by the fast Rust-backed [`html-to-markdown`](https://github.com/thewhodidthis/html-to-markdown) gem.
+
+## Features
+
+- Converts rendered HTML pages to Markdown on `:site, :post_write`.
+- Generates dual paths for nested pages (e.g., `categories/index.html` creates both `categories/index.md` and `categories.md`).
+- Strips interactive dialogs/lightboxes while allowing selective preservation.
+- Extracts SVG accessibility labels (`aria-label`, `alt`, `<title>`) and strips noisy inline SVG vector data.
+- Cleans spacing around inline formatting and adjacent navigation links.
+- Supports custom regex/string replacements prior to Markdown conversion.
+- Registers `text/markdown; charset=utf-8` MIME type for Rack-based Bridgetown servers to prevent character encoding issues in browsers.
+
+## Installation
+
+Add the gem to your Bridgetown application:
+
+```bash
+bundle add bridgetown-html-to-markdown
+```
+
+Or add it directly to your `Gemfile`:
+
+```ruby
+gem "bridgetown-html-to-markdown", "~> 0.1.0"
+```
+
+## Usage
+
+In your Bridgetown site's `config/initializers.rb`:
+
+```ruby
+init :"bridgetown-html-to-markdown"
+```
+
+### Configuration Options
+
+You can pass options directly to `init`:
+
+```ruby
+init :"bridgetown-html-to-markdown",
+  # Custom string or regex replacements before markdown conversion
+  replacements: {
+    /<section class="interactive-tool".*?<\/section>/m => "<p>Please see our catalog for details.</p>"
+  },
+
+  # Glob patterns of files to exclude (default includes 404.html, 500.html, search verification files)
+  exclude_files: ["google*.html", "404.html", "500.html", "BingSiteAuth.xml"],
+
+  # Glob patterns of files to include even if they match exclude_files
+  include_files: ["404.html"],
+
+  # Strip <dialog> elements (default: true)
+  exclude_dialogs: true,
+
+  # Preserve specific dialogs by CSS selector (id or class)
+  preserve_dialog_selectors: ["#contact-dialog"],
+
+  # Extract aria-label, alt, or <title> from SVGs and strip svg markup (default: true)
+  extract_svg_labels: true,
+
+  # Clean spacing between adjacent inline tags and links (default: true)
+  clean_spacing: true
+```
+
+Alternatively, options can be set in `config/bridgetown.config.yml`:
+
+```yaml
+html_to_markdown:
+  exclude_dialogs: true
+  clean_spacing: true
+```
+
+## Development
+
+After checking out the repository, run `bin/setup` to install dependencies. Then run `bundle exec rake` to run tests and code style checks.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/bridgetown-html-to-markdown/builder.rb

@@ -0,0 +1,191 @@
+# frozen_string_literal: true
+
+require "html_to_markdown"
+
+module BridgetownHtmlToMarkdown
+  class Builder < Bridgetown::Builder
+    DEFAULT_EXCLUDED_FILES = [
+      "google*.html",
+      "404.html",
+      "500.html",
+      "yandex_*.html",
+      "BingSiteAuth.xml"
+    ].freeze
+
+    def initialize(name = nil, current_site = nil)
+      current_site ||= Bridgetown::Current.site || Struct.new(:config).new(Bridgetown::Configuration.new)
+      super(name, current_site)
+    end
+
+    def build
+      hook :site, :post_write do |site|
+        convert_site_html(site)
+      end
+    end
+
+    def convert_site_html(site, converter: HtmlToMarkdown)
+      options = html_to_markdown_options(site)
+      html_files = Dir.glob(File.join(site.dest, "**", "*.html"))
+
+      html_files.each do |html_path|
+        next if excluded_file?(html_path, dest_dir: site.dest, options: options)
+
+        convert_html_file(html_path, dest_dir: site.dest, converter: converter, options: options)
+      end
+    end
+
+    def convert_html_file(html_path, dest_dir: nil, converter: HtmlToMarkdown, options: {})
+      md_paths = destination_md_paths(html_path, dest_dir: dest_dir)
+      html_content = File.read(html_path, encoding: "UTF-8")
+      markdown = convert_string(html_content, converter: converter, options: options)
+
+      md_paths.each do |md_path|
+        File.write(md_path, markdown, encoding: "UTF-8")
+
+        if Bridgetown.respond_to?(:logger)
+          rel_path = dest_dir ? md_path.delete_prefix(dest_dir.to_s) : md_path
+          Bridgetown.logger.info("HtmlToMarkdown:", "Generated #{rel_path}")
+        end
+      end
+
+      md_paths
+    rescue StandardError => e
+      Bridgetown.logger.error("HtmlToMarkdown:", "Failed to convert #{html_path}: #{e.message}") if Bridgetown.respond_to?(:logger)
+      nil
+    end
+
+    def convert_string(html, converter: HtmlToMarkdown, options: {})
+      cleaned_html = clean_html(html, options: options)
+
+      result = if defined?(HtmlToMarkdown::ConversionOptions) && converter == HtmlToMarkdown
+                 conv_opts = HtmlToMarkdown::ConversionOptions.new(strip_tags: ["svg"])
+                 converter.convert(cleaned_html, conv_opts)
+               else
+                 converter.convert(cleaned_html)
+               end
+
+      result.respond_to?(:content) ? result.content : result.to_s
+    end
+
+    def clean_html(html, options: {})
+      cleaned = html.to_s
+
+      # Custom replacements first
+      replacements = options[:replacements] || {}
+      replacements.each do |target, replacement|
+        pattern = if target.is_a?(Regexp)
+                    target
+                  elsif target.is_a?(String) && target =~ /\A\(\?([a-z-]*):(.*)\)\z/m
+                    flags = 0
+                    active_flags = Regexp.last_match(1).split("-").first || ""
+                    flags |= Regexp::MULTILINE if active_flags.include?("m")
+                    flags |= Regexp::IGNORECASE if active_flags.include?("i")
+                    Regexp.new(Regexp.last_match(2), flags)
+                  else
+                    target
+                  end
+
+        cleaned = cleaned.gsub(pattern, replacement)
+      end
+
+      # Clean dialogs if exclude_dialogs is enabled (default true)
+      if options.fetch(:exclude_dialogs, true)
+        preserve_selectors = Array(options[:preserve_dialog_selectors])
+        cleaned = clean_dialogs(cleaned, preserve_selectors: preserve_selectors)
+      end
+
+      # Extract or strip SVGs if enabled (default true)
+      cleaned = extract_or_strip_svgs(cleaned) if options.fetch(:extract_svg_labels, true)
+
+      # Clean spacing between adjacent tags if enabled (default true)
+      cleaned = clean_spacing(cleaned) if options.fetch(:clean_spacing, true)
+
+      cleaned
+    end
+
+    def clean_dialogs(html, preserve_selectors: [])
+      html.to_s.gsub(%r{<dialog\b([^>]*?)>(.*?)</dialog>}mi) do |match|
+        attrs = Regexp.last_match(1) || ""
+        preserved = preserve_selectors.any? do |sel|
+          if sel.start_with?("#")
+            attrs =~ /\bid\s*=\s*["']#{Regexp.escape(sel[1..])}["']/i
+          elsif sel.start_with?(".")
+            attrs =~ /\bclass\s*=\s*["'][^"']*\b#{Regexp.escape(sel[1..])}\b[^"']*["']/i
+          else
+            attrs.include?(sel)
+          end
+        end
+
+        preserved ? match : ""
+      end
+    end
+
+    def clean_spacing(html)
+      # Ensure whitespace between adjacent inline formatting tags
+      cleaned = html.to_s.gsub(%r{(</(?:strong|b|em|i|small)>)\s*(<(?:strong|b|em|i|small)>)}i, '\1 \2')
+      # Ensure line break between adjacent anchor tags in navigation/headers
+      cleaned.gsub(%r{(</a>)\s*(<a\b)}i, "\\1<br>\\2")
+    end
+
+    def excluded_file?(html_path, dest_dir: nil, options: {})
+      file_name = File.basename(html_path)
+      rel_path = dest_dir ? html_path.delete_prefix(dest_dir.to_s).delete_prefix("/") : file_name
+
+      include_patterns = Array(options[:include_files])
+      return false if include_patterns.any? { |pat| File.fnmatch?(pat, file_name) || File.fnmatch?(pat, rel_path) }
+
+      exclude_patterns = options.key?(:exclude_files) ? Array(options[:exclude_files]) : DEFAULT_EXCLUDED_FILES
+      exclude_patterns.any? { |pat| File.fnmatch?(pat, file_name) || File.fnmatch?(pat, rel_path) }
+    end
+
+    def html_to_markdown_options(target_site = nil)
+      target_site ||= site
+      opts = {}
+      if target_site.respond_to?(:config) && target_site.config
+        init_params = target_site.config.dig(:init_params, :"bridgetown-html-to-markdown") ||
+                      target_site.config.dig(:init_params, "bridgetown-html-to-markdown")
+        opts = target_site.config[:html_to_markdown] || init_params || {}
+      end
+      opts.transform_keys(&:to_sym)
+    end
+
+    def extract_or_strip_svgs(html)
+      html.to_s.gsub(%r{<svg\b([^>]*?)(?:/>|>(.*?)</svg>)}mi) do
+        attrs = Regexp.last_match(1) || ""
+        inner = Regexp.last_match(2) || ""
+
+        if attrs =~ /\baria-hidden\s*=\s*["']true["']/i
+          ""
+        elsif attrs =~ /\b(?:aria-label|alt)\s*=\s*["']([^"']+)["']/i
+          " #{::Regexp.last_match(1).strip} "
+        elsif inner =~ %r{<title\b[^>]*>(.*?)</title>}mi
+          " #{::Regexp.last_match(1).strip} "
+        else
+          ""
+        end
+      end
+    end
+
+    def destination_md_paths(html_path, dest_dir: nil)
+      primary_path = destination_md_path(html_path)
+      paths = [primary_path]
+
+      if File.basename(html_path) == "index.html"
+        dir = File.dirname(html_path)
+        is_root = if dest_dir
+                    File.expand_path(dir) == File.expand_path(dest_dir)
+                  else
+                    [".", "/", ""].include?(dir)
+                  end
+
+        paths << "#{dir}.md" unless is_root
+      end
+
+      paths
+    end
+
+    def destination_md_path(html_path)
+      html_path.sub(/\.html\z/, ".md")
+    end
+  end
+end

data/lib/bridgetown-html-to-markdown/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module BridgetownHtmlToMarkdown
+  VERSION = "0.1.0"
+end

data/lib/bridgetown-html-to-markdown.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require "bridgetown"
+require_relative "bridgetown-html-to-markdown/version"
+require_relative "bridgetown-html-to-markdown/builder"
+
+begin
+  require "rack/mime"
+  Rack::Mime::MIME_TYPES[".md"] = "text/markdown; charset=utf-8"
+  Rack::Mime::MIME_TYPES[".markdown"] = "text/markdown; charset=utf-8"
+rescue LoadError
+  # Rack not present
+end
+
+module BridgetownHtmlToMarkdown
+end
+
+Bridgetown.initializer :"bridgetown-html-to-markdown" do |config, **options|
+  if defined?(Rack::Mime)
+    Rack::Mime::MIME_TYPES[".md"] = "text/markdown; charset=utf-8"
+    Rack::Mime::MIME_TYPES[".markdown"] = "text/markdown; charset=utf-8"
+  end
+
+  if config.respond_to?(:set)
+    config.set :html_to_markdown, options
+  elsif config.respond_to?(:[]=)
+    config[:html_to_markdown] = options
+  end
+  config.builder(BridgetownHtmlToMarkdown::Builder) if config.respond_to?(:builder)
+end

metadata

@@ -0,0 +1,79 @@
+--- !ruby/object:Gem::Specification
+name: bridgetown-html-to-markdown
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Marc Heiligers
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bridgetown
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+- !ruby/object:Gem::Dependency
+  name: html-to-markdown
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: A Bridgetown plugin that converts rendered HTML files into Markdown files
+  alongside each HTML file using the html-to-markdown converter.
+email:
+- marc@eternal.co.za
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/bridgetown-html-to-markdown.rb
+- lib/bridgetown-html-to-markdown/builder.rb
+- lib/bridgetown-html-to-markdown/version.rb
+homepage: https://github.com/FASCINATION-works/bridgetown-html-to-markdown
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/FASCINATION-works/bridgetown-html-to-markdown
+  changelog_uri: https://github.com/FASCINATION-works/bridgetown-html-to-markdown/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/FASCINATION-works/bridgetown-html-to-markdown/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.6
+specification_version: 4
+summary: Bridgetown plugin to generate a .md file for every html file when building
+test_files: []