markdown-merge 1.0.0

data/RUBOCOP.md

@@ -0,0 +1,71 @@
+# RuboCop Usage Guide
+
+## Overview
+
+A tale of two RuboCop plugin gems.
+
+### RuboCop Gradual
+
+This project uses `rubocop_gradual` instead of vanilla RuboCop for code style checking. The `rubocop_gradual` tool allows for gradual adoption of RuboCop rules by tracking violations in a lock file.
+
+### RuboCop LTS
+
+This project uses `rubocop-lts` to ensure, on a best-effort basis, compatibility with Ruby >= 1.9.2.
+RuboCop rules are meticulously configured by the `rubocop-lts` family of gems to ensure that a project is compatible with a specific version of Ruby. See: https://rubocop-lts.gitlab.io for more.
+
+## Checking RuboCop Violations
+
+To check for RuboCop violations in this project, always use:
+
+```bash
+bundle exec rake rubocop_gradual:check
+```
+
+**Do not use** the standard RuboCop commands like:
+- `bundle exec rubocop`
+- `rubocop`
+
+## Understanding the Lock File
+
+The `.rubocop_gradual.lock` file tracks all current RuboCop violations in the project. This allows the team to:
+
+1. Prevent new violations while gradually fixing existing ones
+2. Track progress on code style improvements
+3. Ensure CI builds don't fail due to pre-existing violations
+
+## Common Commands
+
+- **Check violations**
+    - `bundle exec rake rubocop_gradual`
+    - `bundle exec rake rubocop_gradual:check`
+- **(Safe) Autocorrect violations, and update lockfile if no new violations**
+  - `bundle exec rake rubocop_gradual:autocorrect`
+- **Force update the lock file (w/o autocorrect) to match violations present in code**
+  - `bundle exec rake rubocop_gradual:force_update`
+
+## Workflow
+
+1. Before submitting a PR, run `bundle exec rake rubocop_gradual:autocorrect`
+   a. or just the default `bundle exec rake`, as autocorrection is a pre-requisite of the default task.
+2. If there are new violations, either:
+   - Fix them in your code
+   - Run `bundle exec rake rubocop_gradual:force_update` to update the lock file (only for violations you can't fix immediately)
+3. Commit the updated `.rubocop_gradual.lock` file along with your changes
+
+## Never add inline RuboCop disables
+
+Do not add inline `rubocop:disable` / `rubocop:enable` comments anywhere in the codebase (including specs, except when following the few existing `rubocop:disable` patterns for a rule already being disabled elsewhere in the code). We handle exceptions in two supported ways:
+
+- Permanent/structural exceptions: prefer adjusting the RuboCop configuration (e.g., in `.rubocop.yml`) to exclude a rule for a path or file pattern when it makes sense project-wide.
+- Temporary exceptions while improving code: record the current violations in `.rubocop_gradual.lock` via the gradual workflow:
+  - `bundle exec rake rubocop_gradual:autocorrect` (preferred; will autocorrect what it can and update the lock only if no new violations were introduced)
+  - If needed, `bundle exec rake rubocop_gradual:force_update` (as a last resort when you cannot fix the newly reported violations immediately)
+
+In general, treat the rules as guidance to follow; fix violations rather than ignore them. For example, RSpec conventions in this project expect `described_class` to be used in specs that target a specific class under test.
+
+## Benefits of rubocop_gradual
+
+- Allows incremental adoption of code style rules
+- Prevents CI failures due to pre-existing violations
+- Provides a clear record of code style debt
+- Enables focused efforts on improving code quality over time

data/SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version  | Supported |
+|----------|-----------|
+| 1.latest | ✅         |
+
+## Security contact information
+
+To report a security vulnerability, please use the
+[Tidelift security contact](https://tidelift.com/security).
+Tidelift will coordinate the fix and disclosure.
+
+## Additional Support
+
+If you are interested in support for versions older than the latest release,
+please consider sponsoring the project / maintainer @ https://liberapay.com/pboling/donate,
+or find other sponsorship links in the [README].
+
+[README]: README.md

data/lib/markdown/merge/cleanse/block_spacing.rb

@@ -0,0 +1,253 @@
+# frozen_string_literal: true
+
+require "parslet"
+
+module Markdown
+  module Merge
+    module Cleanse
+      # Fixes missing blank lines between block elements in Markdown.
+      #
+      # Markdown best practices require blank lines between:
+      # - List items and headings
+      # - Thematic breaks (---) and following content
+      # - HTML blocks (like </details>) and following markdown
+      # - Nested list items and headings
+      #
+      # This class detects and fixes these issues without using a full
+      # markdown parser, making it safe to use on documents that might
+      # have syntax issues.
+      #
+      # @example Basic usage
+      #   fixer = Markdown::Merge::Cleanse::BlockSpacing.new(content)
+      #   if fixer.malformed?
+      #     fixed_content = fixer.fix
+      #   end
+      #
+      # @example Check specific issues
+      #   fixer = Markdown::Merge::Cleanse::BlockSpacing.new(content)
+      #   fixer.issues.each do |issue|
+      #     puts "Line #{issue[:line]}: #{issue[:type]}"
+      #   end
+      #
+      class BlockSpacing
+        # Patterns for block elements that should have blank lines after them
+        THEMATIC_BREAK = /\A\s*(?:---+|\*\*\*+|___+)\s*\z/
+        HEADING = /\A\s*\#{1,6}\s+/
+        LIST_ITEM = /\A\s*(?:[-*+]|\d+\.)\s+/
+        HTML_CLOSE_TAG = /\A\s*<\/[a-zA-Z][a-zA-Z0-9]*>\s*\z/
+        HTML_OPEN_TAG = /\A\s*<[a-zA-Z][a-zA-Z0-9]*(?:\s|>)/
+        HTML_ANY_TAG = /\A\s*<\/?[a-zA-Z]/
+        LINK_REF_DEF = /\A\s*\[[^\]]+\]:\s*/
+
+        # Block-level HTML elements that can span multiple lines
+        # These create a context where we shouldn't insert blank lines
+        HTML_BLOCK_ELEMENTS = %w[
+          ul
+          ol
+          li
+          dl
+          dt
+          dd
+          div
+          table
+          thead
+          tbody
+          tfoot
+          tr
+          th
+          td
+          blockquote
+          pre
+          figure
+          figcaption
+          details
+          summary
+          section
+          article
+          aside
+          nav
+          header
+          footer
+          main
+          address
+          form
+          fieldset
+        ].freeze
+
+        # Pattern to match opening block-level HTML tags
+        HTML_BLOCK_OPEN = /\A\s*<(#{HTML_BLOCK_ELEMENTS.join("|")})(?:\s|>)/i
+
+        # Pattern to match closing block-level HTML tags
+        HTML_BLOCK_CLOSE = /\A\s*<\/(#{HTML_BLOCK_ELEMENTS.join("|")})>/i
+
+        # HTML elements that contain markdown content (not HTML content)
+        # These should have blank lines before their closing tags
+        MARKDOWN_CONTAINER_ELEMENTS = %w[details].freeze
+
+        # Pattern to match closing tags for markdown containers
+        MARKDOWN_CONTAINER_CLOSE = /\A\s*<\/(#{MARKDOWN_CONTAINER_ELEMENTS.join("|")})>/i
+
+        # Markdown content: anything that's not blank, not HTML, and not a link ref def
+        MARKDOWN_CONTENT = ->(line) {
+          stripped = line.strip
+          return false if stripped.empty?
+          return false if stripped.start_with?("<")
+          return false if line.match?(LINK_REF_DEF)
+          true
+        }
+
+        # @return [String] The original content
+        attr_reader :source
+
+        # @return [Array<Hash>] Issues found
+        attr_reader :issues
+
+        # Initialize a new BlockSpacing fixer.
+        #
+        # @param source [String] The markdown content to analyze
+        def initialize(source)
+          @source = source
+          @issues = []
+          analyze
+        end
+
+        # Check if the content has block spacing issues.
+        #
+        # @return [Boolean] true if issues were found
+        def malformed?
+          @issues.any?
+        end
+
+        # Get the count of issues found.
+        #
+        # @return [Integer] number of issues
+        def issue_count
+          @issues.size
+        end
+
+        # Fix the block spacing issues.
+        #
+        # @return [String] Content with blank lines added where needed
+        def fix
+          return source unless malformed?
+
+          lines = source.lines
+          result = []
+          insertions = @issues.map { |i| i[:line] }.to_set
+
+          lines.each_with_index do |line, idx|
+            result << line
+            # If this line needs a blank line after it, add one
+            if insertions.include?(idx + 1) # issues use 1-based line numbers
+              result << "\n" unless line.strip.empty?
+            end
+          end
+
+          result.join
+        end
+
+        private
+
+        def analyze
+          lines = source.lines
+          return if lines.empty?
+
+          # Track depth of block-level HTML elements
+          # When depth > 0, we're inside an HTML block and shouldn't add blank lines
+          html_block_depth = 0
+
+          lines.each_with_index do |line, idx|
+            next_line = lines[idx + 1]
+            prev_line = (idx > 0) ? lines[idx - 1] : nil
+
+            # Special case: closing tags for markdown containers like </details>
+            # These contain markdown content, so we need blank lines before them
+            # even when inside an HTML block
+            is_markdown_container_close = line.match?(MARKDOWN_CONTAINER_CLOSE)
+
+            # Check for issues BEFORE updating depth
+            if html_block_depth <= 0
+              # Check for issues that need blank line AFTER current line
+              if next_line && !next_line.strip.empty?
+                check_thematic_break(line, next_line, idx)
+                check_list_before_heading(line, next_line, idx)
+                check_html_close_before_markdown(line, next_line, idx)
+              end
+
+              # Check for issues that need blank line BEFORE current line
+              if prev_line && !prev_line.strip.empty?
+                check_markdown_before_html(prev_line, line, idx)
+              end
+            end
+
+            # Special case: always check for blank line before </details> etc.
+            # because they contain markdown content
+            if is_markdown_container_close && prev_line && !prev_line.strip.empty?
+              check_markdown_before_html(prev_line, line, idx)
+            end
+
+            # Update HTML block depth AFTER checking for issues
+            # Count opening block-level tags
+            if line.match?(HTML_BLOCK_OPEN)
+              html_block_depth += 1
+            end
+
+            # Check for closing block-level tags
+            line.scan(HTML_BLOCK_CLOSE) do
+              html_block_depth -= 1 if html_block_depth > 0
+            end
+          end
+        end
+
+        def check_thematic_break(line, next_line, idx)
+          return unless line.match?(THEMATIC_BREAK)
+          return if next_line.strip.empty?
+
+          @issues << {
+            type: :thematic_break_needs_blank,
+            line: idx + 1,
+            description: "Thematic break should be followed by blank line",
+          }
+        end
+
+        def check_list_before_heading(line, next_line, idx)
+          return unless line.match?(LIST_ITEM)
+          return unless next_line.match?(HEADING)
+
+          @issues << {
+            type: :list_before_heading,
+            line: idx + 1,
+            description: "List item should be followed by blank line before heading",
+          }
+        end
+
+        def check_html_close_before_markdown(line, next_line, idx)
+          return unless line.match?(HTML_CLOSE_TAG)
+          # Next line is markdown (heading, list, paragraph start, etc.)
+          # but not HTML or blank
+          return if next_line.match?(/\A\s*</)
+          return if next_line.match?(LINK_REF_DEF)
+
+          @issues << {
+            type: :html_before_markdown,
+            line: idx + 1,
+            description: "HTML close tag should be followed by blank line before markdown",
+          }
+        end
+
+        def check_markdown_before_html(prev_line, line, idx)
+          # Current line is HTML (open or close tag)
+          return unless line.match?(HTML_ANY_TAG)
+          # Previous line is markdown content (not HTML, not blank, not link ref)
+          return unless MARKDOWN_CONTENT.call(prev_line)
+
+          @issues << {
+            type: :markdown_before_html,
+            line: idx, # Insert blank line BEFORE this line (so after prev_line)
+            description: "Markdown content should be followed by blank line before HTML",
+          }
+        end
+      end
+    end
+  end
+end

data/lib/markdown/merge/cleanse/code_fence_spacing.rb

@@ -0,0 +1,294 @@
+# frozen_string_literal: true
+
+require "parslet"
+
+module Markdown
+  module Merge
+    module Cleanse
+      # Parslet-based parser for fixing malformed fenced code blocks in Markdown.
+      #
+      # == The Problem
+      #
+      # This class fixes **improperly formatted fenced code blocks** where there is
+      # unwanted whitespace between the fence markers (``` or ~~~) and the language
+      # identifier.
+      #
+      # A bug in ast-merge (or its dependencies) caused fenced code blocks to be
+      # rendered with a space between the fence markers and the language identifier.
+      #
+      # == Bug Pattern
+      #
+      # CommonMark and most Markdown parsers expect NO space between fence and language:
+      # - **Correct:** ` ```ruby` or ` ~~~python`
+      # - **Incorrect:** ` ``` ruby` or ` ~~~ python` (extra space)
+      #
+      # The extra space can cause:
+      # - Syntax highlighting to fail
+      # - The language identifier to be ignored
+      # - Rendering issues in various Markdown processors
+      #
+      # @example Malformed (buggy) input
+      #   "``` console\nsome code\n```"
+      #
+      # @example Fixed output
+      #   "```console\nsome code\n```"
+      #
+      # == Scope
+      #
+      # This fixer handles:
+      # - **Any indentation level** (0+ spaces before fence)
+      #   - Top-level: ` ```ruby`
+      #   - In lists: `    ```python` (4 spaces)
+      # - **Both fence types:** backticks (```) and tildes (~~~)
+      # - **Any fence length:** 3+ markers (````, ~~~~~, etc.)
+      #
+      # == How It Works
+      #
+      # The parser uses a **PEG grammar** (via Parslet) to:
+      # - Detect fence opening lines with optional indentation
+      # - Identify spacing between fence and language identifier
+      # - Track opening/closing fence pairs to avoid false positives
+      # - Reconstruct fences with proper formatting (no space)
+      #
+      # **Why PEG?** The previous regex-based implementation used patterns like
+      # `([ \t]*)` which can cause polynomial backtracking (ReDoS vulnerability)
+      # when processing malicious input with many tabs/spaces. PEG parsers are
+      # linear-time and immune to ReDoS attacks.
+      #
+      # @example Basic usage
+      #   parser = Markdown::Merge::Cleanse::CodeFenceSpacing.new(content)
+      #   fixed_content = parser.fix
+      #
+      # @example Check if content has malformed fences
+      #   parser = Markdown::Merge::Cleanse::CodeFenceSpacing.new(content)
+      #   parser.malformed? # => true/false
+      #
+      # @example Process a file
+      #   content = File.read("README.md")
+      #   parser = Markdown::Merge::Cleanse::CodeFenceSpacing.new(content)
+      #   if parser.malformed?
+      #     File.write("README.md", parser.fix)
+      #   end
+      #
+      # @example Get details about code blocks
+      #   parser = Markdown::Merge::Cleanse::CodeFenceSpacing.new(content)
+      #   parser.code_blocks.each do |block|
+      #     puts "#{block[:fence]}#{block[:language]}: malformed=#{block[:malformed]}"
+      #   end
+      #
+      # @api public
+      class CodeFenceSpacing
+        # Grammar for parsing fenced code blocks with PEG parser.
+        #
+        # Recognizes:
+        # - Any amount of indentation (handles nested lists)
+        # - Backtick fences (```) and tilde fences (~~~)
+        # - Optional info string (language identifier)
+        # - Properly handles spacing issues
+        #
+        # This PEG grammar is linear-time and cannot have polynomial backtracking,
+        # eliminating ReDoS vulnerabilities.
+        #
+        # @api private
+        class CodeFenceGrammar < Parslet::Parser
+          # Any amount of indentation (handles code blocks in lists)
+          # Captured as string, not array
+          rule(:indent) { match("[ ]").repeat }
+
+          # Fence markers - 3+ backticks or tildes
+          rule(:backtick) { str("`") }
+          rule(:tilde) { str("~") }
+          rule(:backtick_fence) { backtick.repeat(3, nil) }
+          rule(:tilde_fence) { tilde.repeat(3, nil) }
+          rule(:fence) { backtick_fence | tilde_fence }
+
+          # Whitespace after fence (the bug we're fixing)
+          rule(:space) { match('[ \t]') }
+          rule(:spaces) { space.repeat(1) }
+          rule(:spaces?) { space.repeat }
+
+          # Info string (language identifier + optional attributes)
+          # Cannot contain backticks or tildes per CommonMark
+          rule(:info_char) { match('[^\r\n`~]') }
+          rule(:info_string) { info_char.repeat(1) }
+
+          # Line ending
+          rule(:line_end) { str("\r").maybe >> str("\n").maybe >> any.absent? }
+
+          # Fence line with optional indentation, optional spacing, optional info
+          # Capture: indent (raw), fence (as :fence), spacing (as :spacing), info (as :info)
+          rule(:fence_line) {
+            indent.as(:indent) >> fence.as(:fence) >> spaces?.as(:spacing) >> info_string.maybe.as(:info) >> line_end
+          }
+
+          root(:fence_line)
+        end
+
+        # @return [String] the input text to parse
+        attr_reader :source
+
+        # Create a new parser for the given text.
+        #
+        # @param source [String] the text that may contain malformed code fences
+        def initialize(source)
+          @source = source.to_s
+          @grammar = CodeFenceGrammar.new
+          @code_blocks = nil
+        end
+
+        # Check if the source contains malformed fenced code blocks.
+        #
+        # Detects the pattern where there's whitespace between the fence
+        # markers and the language identifier.
+        #
+        # @return [Boolean] true if malformed fences are detected
+        def malformed?
+          code_blocks.any? { |block| block[:malformed] }
+        end
+
+        # Parse and return information about all fenced code blocks.
+        #
+        # Only returns opening fences (not closing fences).
+        #
+        # @return [Array<Hash>] Array of code block info
+        #   - :indent [String] The indentation before the fence
+        #   - :fence [String] The fence markers (e.g., "```" or "~~~")
+        #   - :language [String, nil] The language identifier
+        #   - :spacing [String] Any spacing between fence and language
+        #   - :malformed [Boolean] Whether this block has improper spacing
+        #   - :line_number [Integer] Line number where block starts (1-based)
+        #   - :original [String] The original opening fence line
+        def code_blocks
+          return @code_blocks if @code_blocks
+
+          @code_blocks = []
+          line_number = 0
+          in_code_block = false
+          current_fence_char = nil
+
+          source.each_line do |line|
+            line_number += 1
+
+            # Try to parse as fence line using PEG grammar
+            parsed = parse_fence_line(line)
+            next unless parsed
+
+            fence = parsed[:fence]
+            fence_char = fence[0]
+            spacing = parsed[:spacing] || ""
+            info = parsed[:info] || ""
+            indent = parsed[:indent] || ""
+
+            # Closing fence: matches current fence type and has no info
+            if in_code_block && fence_char == current_fence_char && info.empty?
+              in_code_block = false
+              current_fence_char = nil
+              next
+            end
+
+            # Opening fence
+            in_code_block = true
+            current_fence_char = fence_char
+
+            # Extract just the language (first word of info string)
+            language = info.strip.split(/\s+/).first
+            language = nil if language&.empty?
+
+            @code_blocks << {
+              indent: indent,
+              fence: fence,
+              language: language,
+              info_string: info.strip,
+              spacing: spacing,
+              malformed: !spacing.empty? && !language.nil?,
+              line_number: line_number,
+              original: line.chomp,
+            }
+          end
+
+          @code_blocks
+        end
+
+        # Fix malformed fenced code blocks by removing improper spacing.
+        #
+        # @return [String] the source with code fences fixed
+        def fix
+          return source unless malformed?
+
+          result = source.dup
+
+          # Process line by line, fixing malformed fences
+          lines = result.lines
+          fixed_lines = lines.map do |line|
+            fix_fence_line(line)
+          end
+
+          fixed_lines.join
+        end
+
+        # Count the number of malformed code blocks.
+        #
+        # @return [Integer] number of malformed fences found
+        def malformed_count
+          code_blocks.count { |block| block[:malformed] }
+        end
+
+        # Count the total number of code blocks.
+        #
+        # @return [Integer] total number of fenced code blocks
+        def count
+          code_blocks.size
+        end
+
+        private
+
+        # Parse a single line as a fence using PEG grammar.
+        #
+        # @param line [String] the line to parse
+        # @return [Hash, nil] parsed fence data or nil if not a fence
+        def parse_fence_line(line)
+          tree = @grammar.parse(line)
+
+          # Convert Parslet tree to simple hash
+          # Note: Parslet returns [] for empty repeats, we convert to empty string
+          indent_val = tree[:indent]
+          indent_str = indent_val.is_a?(Array) ? indent_val.join : indent_val.to_s
+
+          spacing_val = tree[:spacing]
+          spacing_str = spacing_val.is_a?(Array) ? spacing_val.join : spacing_val.to_s
+
+          info_val = tree[:info]
+          info_str = if info_val.is_a?(Array)
+            info_val.join
+          else
+            (info_val ? info_val.to_s : "")
+          end
+
+          {
+            indent: indent_str,
+            fence: tree[:fence].to_s,
+            spacing: spacing_str,
+            info: info_str,
+          }
+        rescue Parslet::ParseFailed
+          nil
+        end
+
+        # Fix a single line if it's a malformed fence.
+        #
+        # @param line [String] the line to potentially fix
+        # @return [String] the fixed line (or original if not malformed)
+        def fix_fence_line(line)
+          parsed = parse_fence_line(line)
+          return line unless parsed
+
+          # Only fix if there's spacing AND info string
+          return line if parsed[:spacing].empty? || parsed[:info].empty?
+
+          # Reconstruct: indent + fence + info (no spacing)
+          "#{parsed[:indent]}#{parsed[:fence]}#{parsed[:info]}\n"
+        end
+      end
+    end
+  end
+end