yaml-converter 0.1.0

data/RUBOCOP.md

@@ -0,0 +1,71 @@
+# RuboCop Usage Guide
+
+## Overview
+
+A tale of two RuboCop plugin gems.
+
+### RuboCop Gradual
+
+This project uses `rubocop_gradual` instead of vanilla RuboCop for code style checking. The `rubocop_gradual` tool allows for gradual adoption of RuboCop rules by tracking violations in a lock file.
+
+### RuboCop LTS
+
+This project uses `rubocop-lts` to ensure, on a best-effort basis, compatibility with Ruby >= 1.9.2.
+RuboCop rules are meticulously configured by the `rubocop-lts` family of gems to ensure that a project is compatible with a specific version of Ruby. See: https://rubocop-lts.gitlab.io for more.
+
+## Checking RuboCop Violations
+
+To check for RuboCop violations in this project, always use:
+
+```bash
+bundle exec rake rubocop_gradual:check
+```
+
+**Do not use** the standard RuboCop commands like:
+- `bundle exec rubocop`
+- `rubocop`
+
+## Understanding the Lock File
+
+The `.rubocop_gradual.lock` file tracks all current RuboCop violations in the project. This allows the team to:
+
+1. Prevent new violations while gradually fixing existing ones
+2. Track progress on code style improvements
+3. Ensure CI builds don't fail due to pre-existing violations
+
+## Common Commands
+
+- **Check violations**
+    - `bundle exec rake rubocop_gradual`
+    - `bundle exec rake rubocop_gradual:check`
+- **(Safe) Autocorrect violations, and update lockfile if no new violations**
+  - `bundle exec rake rubocop_gradual:autocorrect`
+- **Force update the lock file (w/o autocorrect) to match violations present in code**
+  - `bundle exec rake rubocop_gradual:force_update`
+
+## Workflow
+
+1. Before submitting a PR, run `bundle exec rake rubocop_gradual:autocorrect`
+   a. or just the default `bundle exec rake`, as autocorrection is a pre-requisite of the default task.
+2. If there are new violations, either:
+   - Fix them in your code
+   - Run `bundle exec rake rubocop_gradual:force_update` to update the lock file (only for violations you can't fix immediately)
+3. Commit the updated `.rubocop_gradual.lock` file along with your changes
+
+## Never add inline RuboCop disables
+
+Do not add inline `rubocop:disable` / `rubocop:enable` comments anywhere in the codebase (including specs, except when following the few existing `rubocop:disable` patterns for a rule already being disabled elsewhere in the code). We handle exceptions in two supported ways:
+
+- Permanent/structural exceptions: prefer adjusting the RuboCop configuration (e.g., in `.rubocop.yml`) to exclude a rule for a path or file pattern when it makes sense project-wide.
+- Temporary exceptions while improving code: record the current violations in `.rubocop_gradual.lock` via the gradual workflow:
+  - `bundle exec rake rubocop_gradual:autocorrect` (preferred; will autocorrect what it can and update the lock only if no new violations were introduced)
+  - If needed, `bundle exec rake rubocop_gradual:force_update` (as a last resort when you cannot fix the newly reported violations immediately)
+
+In general, treat the rules as guidance to follow; fix violations rather than ignore them. For example, RSpec conventions in this project expect `described_class` to be used in specs that target a specific class under test.
+
+## Benefits of rubocop_gradual
+
+- Allows incremental adoption of code style rules
+- Prevents CI failures due to pre-existing violations
+- Provides a clear record of code style debt
+- Enables focused efforts on improving code quality over time

data/SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version  | Supported |
+|----------|-----------|
+| 1.latest | ✅         |
+
+## Security contact information
+
+To report a security vulnerability, please use the
+[Tidelift security contact](https://tidelift.com/security).
+Tidelift will coordinate the fix and disclosure.
+
+## Additional Support
+
+If you are interested in support for versions older than the latest release,
+please consider sponsoring the project / maintainer @ https://liberapay.com/pboling/donate,
+or find other sponsorship links in the [README].
+
+[README]: README.md

data/exe/yaml-convert

@@ -0,0 +1,122 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+require "fileutils"
+require "yaml/converter"
+
+options = {
+  max_line_length: nil,
+  truncate: nil,
+  margin_notes: nil,
+  validate: nil,
+  use_pandoc: nil,
+  pandoc_args: nil,
+  glob: nil,
+  out_ext: nil,
+  out_dir: nil,
+  concurrency: nil,
+  streaming: nil,
+  streaming_threshold_bytes: nil,
+}
+
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: yaml-convert INPUT.yaml OUTPUT.(md|html|pdf|docx) [options]\n       yaml-convert --glob 'dir/**/*.yaml' --out-ext (md|html|pdf|docx) [--out-dir DIR] [options]"
+
+  opts.on("--max-line-length N", Integer, "Set maximum line length (default 70)") { |v| options[:max_line_length] = v }
+  opts.on("--no-truncate", "Disable line truncation") { options[:truncate] = false }
+  opts.on("--no-validate", "Disable YAML validation") { options[:validate] = false }
+  opts.on("--margin-notes MODE", "Set margin notes mode (auto|inline|ignore)") { |v| options[:margin_notes] = v.to_sym }
+  opts.on("--use-pandoc", "Use pandoc for non-md/html output formats") { options[:use_pandoc] = true }
+  opts.on("--pandoc-args STRING", "Extra args for pandoc (split on spaces)") { |v| options[:pandoc_args] = v.split(/\s+/) }
+  opts.on("--streaming", "Force streaming mode for markdown output") { options[:streaming] = true }
+  opts.on("--streaming-threshold BYTES", Integer, "Auto-enable streaming when input file >= BYTES") { |v| options[:streaming_threshold_bytes] = v }
+
+  # Batch mode
+  opts.on("--glob PATTERN", "Batch convert all YAML files matching the glob pattern") { |v| options[:glob] = v }
+  opts.on("--out-ext EXT", "Output extension for batch mode (md|html|pdf|docx)") { |v| options[:out_ext] = v }
+  opts.on("--out-dir DIR", "Output directory for batch mode (defaults next to input)") { |v| options[:out_dir] = v }
+  opts.on("--concurrency N", Integer, "Batch concurrency (currently not used; reserved)") { |v| options[:concurrency] = v }
+
+  opts.on("--version", "Show version") {
+    puts Yaml::Converter::VERSION
+    exit(0)
+  }
+  opts.on("--help", "Show help") {
+    puts opts
+    exit(0)
+  }
+end
+
+begin
+  parser.parse!(ARGV)
+rescue OptionParser::InvalidOption => e
+  warn(e.message)
+  puts parser
+  exit(2)
+end
+
+# Batch mode if --glob is provided
+if options[:glob]
+  pattern = options[:glob]
+  out_ext = options[:out_ext]
+  unless out_ext&.match?(/^\w+$/)
+    warn "Error: --out-ext is required for --glob and must be like: md|html|pdf|docx"
+    puts parser
+    exit 2
+  end
+  files = Dir.glob(pattern).select { |p| File.file?(p) }
+  if files.empty?
+    warn "No files matched glob: #{pattern}"
+    exit 2
+  end
+  out_dir = options[:out_dir]
+  FileUtils.mkdir_p(out_dir) if out_dir
+
+  failures = 0
+  files.each do |input|
+    base = File.basename(input, File.extname(input))
+    dest_dir = out_dir || File.dirname(input)
+    FileUtils.mkdir_p(dest_dir)
+    output = File.join(dest_dir, base + "." + out_ext)
+    begin
+      result = Yaml::Converter.convert(input_path: input, output_path: output, options: options.compact)
+      puts "Converted: #{result[:output_path]}" unless ENV["KETTLE_TEST_SILENT"] == "true"
+    rescue Yaml::Converter::InvalidArgumentsError, Yaml::Converter::PandocNotFoundError, Yaml::Converter::RendererUnavailableError => e
+      failures += 1
+      warn("Failed: #{input}: #{e.message}")
+    rescue StandardError => e
+      failures += 1
+      warn("Unexpected error converting #{input}: #{e.class}: #{e.message}")
+    end
+  end
+  puts "Batch complete: #{files.size - failures} succeeded, #{failures} failed" unless ENV["KETTLE_TEST_SILENT"] == "true"
+  exit(failures.zero? ? 0 : 5)
+end
+
+# Single-file mode
+if ARGV.length != 2
+  warn "Error: require INPUT.yaml and OUTPUT.<ext>"
+  puts parser
+  exit 2
+end
+
+input, output = ARGV
+
+begin
+  result = Yaml::Converter.convert(input_path: input, output_path: output, options: options.compact)
+  puts "Converted: #{result[:output_path]}" unless ENV["KETTLE_TEST_SILENT"] == "true"
+  exit(0)
+rescue Yaml::Converter::InvalidArgumentsError => e
+  warn(e.message)
+  exit(2)
+rescue Yaml::Converter::PandocNotFoundError => e
+  warn(e.message)
+  exit(5)
+rescue Yaml::Converter::RendererUnavailableError => e
+  warn(e.message)
+  exit(5)
+rescue StandardError => e
+  warn("Unexpected error: #{e.class}: #{e.message}")
+  exit(9)
+end

data/lib/yaml/converter/config.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+require "date"
+
+module Yaml
+  module Converter
+    # Central configuration handling: merges explicit options with ENV and defaults.
+    # Use {Yaml::Converter::Config.resolve} to obtain the finalized options hash.
+    module Config
+      DEFAULTS = {
+        max_line_length: 70,
+        truncate: true,
+        margin_notes: :auto, # :auto | :inline | :ignore
+        validate: true,
+        use_pandoc: false,
+        pandoc_args: ["-N", "--toc"],
+        pandoc_path: nil,
+        html_theme: :basic,
+        pdf_page_size: "LETTER",
+        pdf_margin: [36, 36, 36, 36], # top,right,bottom,left points (0.5")
+        pdf_title_font_size: 14,
+        pdf_body_font_size: 11,
+        pdf_yaml_font_size: 9,
+        pdf_two_column_notes: false,
+        current_date: Date.today, # allows injection for deterministic tests
+        emit_footer: true,
+        # Streaming options (Phase 3 feature now implemented):
+        streaming: false, # force streaming mode even for small files
+        streaming_threshold_bytes: 5_000_000, # auto-enable streaming for large files when not forced
+      }.freeze
+
+      ENV_MAP = {
+        max_line_length: "YAML_CONVERTER_MAX_LINE_LEN",
+        truncate: "YAML_CONVERTER_TRUNCATE",
+        margin_notes: "YAML_CONVERTER_MARGIN_NOTES",
+        validate: "YAML_CONVERTER_VALIDATE",
+        use_pandoc: "YAML_CONVERTER_USE_PANDOC",
+        pdf_page_size: "YAML_CONVERTER_PDF_PAGE_SIZE",
+        pdf_title_font_size: "YAML_CONVERTER_PDF_TITLE_FONT_SIZE",
+        pdf_body_font_size: "YAML_CONVERTER_PDF_BODY_FONT_SIZE",
+        pdf_yaml_font_size: "YAML_CONVERTER_PDF_YAML_FONT_SIZE",
+        pdf_two_column_notes: "YAML_CONVERTER_PDF_TWO_COLUMN_NOTES",
+        emit_footer: "YAML_CONVERTER_EMIT_FOOTER",
+        streaming: "YAML_CONVERTER_STREAMING",
+        streaming_threshold_bytes: "YAML_CONVERTER_STREAMING_THRESHOLD_BYTES",
+      }.freeze
+
+      BOOLEAN_KEYS = %i[truncate validate use_pandoc pdf_two_column_notes emit_footer streaming].freeze
+
+      class << self
+        # Merge caller options with environment overrides and defaults.
+        # Environment bools accept values: 1, true, yes, on (case-insensitive).
+        #
+        # @param options [Hash]
+        # @return [Hash] normalized options suitable for passing to emitters/renderers
+        def resolve(options = {})
+          opts = DEFAULTS.dup
+          ENV_MAP.each do |key, env_key|
+            val = ENV[env_key]
+            next if val.nil?
+
+            opts[key] = coerce_env_value(key, val)
+          end
+          options.each do |k, v|
+            opts[k] = v unless v.nil?
+          end
+          normalize(opts)
+        end
+
+        # Normalize symbolic values loaded from ENV.
+        # @param opts [Hash]
+        # @return [Hash]
+        def normalize(opts)
+          opts[:margin_notes] = opts[:margin_notes].to_sym if opts[:margin_notes].is_a?(String)
+          opts[:html_theme] = opts[:html_theme].to_sym if opts[:html_theme].is_a?(String)
+          if opts[:streaming_threshold_bytes].is_a?(String)
+            opts[:streaming_threshold_bytes] = opts[:streaming_threshold_bytes].to_i
+          end
+          opts
+        end
+
+        # Coerce ENV string values into typed Ruby objects.
+        # @param key [Symbol]
+        # @param value [String]
+        # @return [Object]
+        def coerce_env_value(key, value)
+          if BOOLEAN_KEYS.include?(key)
+            %w[1 true yes on].include?(value.to_s.downcase)
+          elsif key == :max_line_length || key == :streaming_threshold_bytes
+            value.to_i
+          else
+            value
+          end
+        end
+      end
+    end
+  end
+end

data/lib/yaml/converter/markdown_emitter.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require "date"
+require_relative "parser"
+require_relative "state_machine"
+
+module Yaml
+  module Converter
+    # High-level emitter orchestrates parsing tokens and applying the state machine,
+    # producing a Markdown document with fenced YAML blocks and extracted notes.
+    class MarkdownEmitter
+      START_YAML = "```yaml"
+      END_YAML = "```"
+      VALIDATED_STR = "YAML validation:"
+      NOTE_STR = "note:"
+
+      # @param options [Hash] Configuration values merged from {Config.resolve}
+      def initialize(options)
+        @options = options
+        @max_len = options.fetch(:max_line_length)
+        @truncate = options.fetch(:truncate)
+        @margin_notes = options.fetch(:margin_notes)
+        @validate = options.fetch(:validate)
+        @validation_status = :ok
+        @is_latex = false
+      end
+
+      # Set the validation status used when injecting the validation status line.
+      # @param status [Symbol] :ok or :fail
+      # @return [void]
+      def set_validation_status(status)
+        @validation_status = status
+      end
+
+      # Convert input lines to markdown using parser + state machine, then append a footer.
+      # @param lines [Array<String>] Raw YAML file lines
+      # @return [Array<String>] Final markdown lines
+      def emit(lines)
+        parser = Parser.new(@options)
+        tokens = parser.tokenize(lines)
+        sm = StateMachine.new(validation_status: @validation_status, max_line_length: @max_len, truncate: @truncate, margin_notes: @margin_notes, current_date: @options[:current_date])
+        body = sm.apply(tokens)
+        if @options[:emit_footer]
+          body << "---- \n\nProduced by [yaml-converter](https://github.com/kettle-rb/yaml-converter)"
+        end
+        body
+      end
+    end
+  end
+end

data/lib/yaml/converter/parser.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+module Yaml
+  module Converter
+    # Tokenizes input YAML lines (with inline annotations) into structured tokens
+    # consumable by the {Yaml::Converter::StateMachine}.
+    #
+    # Input assumptions:
+    # - Comment titles: lines starting with `# ` become title tokens.
+    # - Validation marker: a comment line starting with `# YAML validation:` is recognized.
+    # - Separator lines (`---`) are recognized and currently ignored by the state machine.
+    # - Inline notes: fragments after `#note:` are captured as out-of-band NOTE tokens.
+    # - Other non-empty lines are treated as YAML content.
+    class Parser
+      # Lightweight token structure used by the parser/state machine pipeline.
+      #
+      # @!attribute [rw] type
+      #   @return [Symbol] One of :blank, :title, :validation, :separator, :dash_heading, :yaml_line, :note
+      # @!attribute [rw] text
+      #   @return [String] Payload string for this token
+      # @!attribute [rw] meta
+      #   @return [Hash,nil] Optional metadata bag (currently unused)
+      Token = Struct.new(:type, :text, :meta, keyword_init: true)
+
+      # Comment line prefix indicating a validation status line will be injected
+      VALIDATION_PREFIX = "# YAML validation:"
+      # Inline note marker captured from right side of a line
+      NOTE_MARK = "#note:"
+
+      # @param options [Hash] Reserved for future parsing options
+      def initialize(options = {})
+        @options = options
+      end
+
+      # Convert raw lines into token objects.
+      #
+      # @param lines [Array<String>] Input lines (including newlines)
+      # @return [Array<Token>] Sequence of tokens representing the document structure
+      def tokenize(lines)
+        tokens = []
+        lines.each do |raw|
+          line = raw.rstrip
+          if line.empty?
+            tokens << Token.new(type: :blank, text: "")
+            next
+          end
+
+          if line.start_with?("# ") || line == "#"
+            if line.start_with?(VALIDATION_PREFIX)
+              tokens << Token.new(type: :validation, text: line[2..])
+            else
+              content = (line == "#") ? "" : line[2..]
+              tokens << Token.new(type: :title, text: content)
+            end
+            next
+          end
+
+          if line == "---"
+            tokens << Token.new(type: :separator, text: line)
+            next
+          end
+
+          if line.start_with?("-")
+            tokens << Token.new(type: :dash_heading, text: line[1..].strip)
+          end
+
+          note_idx = line.index(NOTE_MARK)
+          if note_idx
+            base = line[0...note_idx].rstrip
+            note = line[(note_idx + NOTE_MARK.length)..].to_s.strip
+            tokens << Token.new(type: :yaml_line, text: base)
+            tokens << Token.new(type: :note, text: note)
+          else
+            tokens << Token.new(type: :yaml_line, text: line)
+          end
+        end
+        tokens
+      end
+    end
+  end
+end

data/lib/yaml/converter/renderer/pandoc_shell.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "open3"
+
+module Yaml
+  module Converter
+    module Renderer
+      # Shells out to pandoc to convert markdown to target format.
+      # Provides lightweight detection of pandoc and execution with arbitrary arguments.
+      #
+      # @example Convert markdown to PDF
+      #   Yaml::Converter::Renderer::PandocShell.render(
+      #     md_path: 'doc.md', out_path: 'doc.pdf', args: ['-N', '--toc']
+      #   )
+      module PandocShell
+        module_function
+
+        # Locate an executable in the current PATH.
+        # @param cmd [String] command name, e.g. 'pandoc'
+        # @return [String,nil] full path if found, otherwise nil
+        def which(cmd)
+          ENV["PATH"].split(File::PATH_SEPARATOR).each do |path|
+            exe = File.join(path, cmd)
+            return exe if File.executable?(exe)
+          end
+          nil
+        end
+
+        # Invoke pandoc to convert markdown file to another format.
+        #
+        # @param md_path [String] path to markdown file
+        # @param out_path [String] desired output file path with extension
+        # @param pandoc_path [String,nil] override path to pandoc binary (auto-detected if nil)
+        # @param args [Array<String>] extra pandoc arguments (e.g. ['-N','--toc'])
+        # @return [Boolean] true if successful, false otherwise
+        # @example Basic HTML conversion
+        #   Yaml::Converter::Renderer::PandocShell.render(
+        #     md_path: 'in.md', out_path: 'out.html'
+        #   )
+        def render(md_path:, out_path:, pandoc_path: nil, args: [])
+          bin = pandoc_path || which("pandoc")
+          return false unless bin
+          cmd = [bin] + args + ["-o", out_path, md_path]
+          _stdout, stderr, status = Open3.capture3(*cmd)
+          unless status.success?
+            warn("pandoc failed: #{stderr}")
+            return false
+          end
+          true
+        end
+      end
+    end
+  end
+end

data/lib/yaml/converter/renderer/pdf_prawn.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+module Yaml
+  module Converter
+    module Renderer
+      # Native PDF rendering using prawn.
+      # Basic layout: title lines, YAML block in monospace, notes as italic paragraphs.
+      #
+      # For PDF rendering via pandoc, see {Yaml::Converter::Renderer::PandocShell}.
+      module PdfPrawn
+        module_function
+
+        # Render a PDF document from the given markdown string.
+        #
+        # @param markdown [String] Markdown that includes fenced YAML and blockquote notes
+        # @param out_path [String] Destination PDF path
+        # @param options [Hash] PDF options (see Config defaults: page size, margins, font sizes, two-column notes)
+        # @option options [String] :pdf_page_size ("LETTER")
+        # @option options [Array<Integer>] :pdf_margin ([36,36,36,36])
+        # @option options [Integer] :pdf_title_font_size (14)
+        # @option options [Integer] :pdf_body_font_size (11)
+        # @option options [Integer] :pdf_yaml_font_size (9)
+        # @option options [Boolean] :pdf_two_column_notes (false)
+        # @return [Boolean] true if rendering succeeded
+        def render(markdown:, out_path:, options: {})
+          require "prawn"
+
+          notes = extract_notes(markdown)
+          yaml_section = fenced_yaml(markdown)
+          title_lines = header_lines(markdown)
+
+          page_size = options[:pdf_page_size] || "LETTER"
+          margin = options[:pdf_margin] || [36, 36, 36, 36]
+          title_fs = options[:pdf_title_font_size] || 14
+          body_fs = options[:pdf_body_font_size] || 11
+          yaml_fs = options[:pdf_yaml_font_size] || 9
+          two_col = !!options[:pdf_two_column_notes]
+
+          Prawn::Document.generate(out_path, page_size: page_size, margin: margin) do |pdf|
+            pdf.font_size(title_fs)
+            title_lines.each { |l| pdf.text(l, style: :bold) }
+            pdf.move_down(10) if title_lines.any?
+
+            if two_col && notes.any?
+              # Create two columns: left for YAML, right for notes
+              col_gap = 16
+              col_width = (pdf.bounds.width - col_gap) / 2.0
+              pdf.bounding_box([pdf.bounds.left, pdf.cursor], width: col_width, height: pdf.cursor) do
+                pdf.font_size(yaml_fs)
+                pdf.font("Courier") { yaml_section.each { |l| pdf.text(l) } }
+              end
+              pdf.bounding_box([pdf.bounds.left + col_width + col_gap, pdf.cursor], width: col_width, height: pdf.cursor) do
+                pdf.font_size(body_fs)
+                notes.each { |n| pdf.text("NOTE: #{n}", style: :italic) }
+              end
+            else
+              pdf.font_size(yaml_fs)
+              pdf.font("Courier") { yaml_section.each { |l| pdf.text(l) } }
+              pdf.move_down(10)
+              unless notes.empty?
+                pdf.font_size(body_fs)
+                notes.each { |n| pdf.text("NOTE: #{n}", style: :italic) }
+              end
+            end
+
+            pdf.number_pages("Page <page> of <total>", at: [pdf.bounds.right - 100, 0])
+          end
+          true
+        rescue StandardError => e
+          warn("prawn pdf failed: #{e.class}: #{e.message}")
+          false
+        end
+
+        # Extract leading `# Title lines`.
+        # @param markdown [String]
+        # @return [Array<String>]
+        def header_lines(markdown)
+          markdown.lines.take_while { |l| l.start_with?("# ") }.map { |l| l.sub(/^# /, "").strip }
+        end
+
+        # Return the lines inside the first fenced ```yaml block.
+        # @param markdown [String]
+        # @return [Array<String>]
+        def fenced_yaml(markdown)
+          inside = false
+          lines = []
+          markdown.each_line do |l|
+            if l.start_with?("```yaml")
+              inside = true
+              next
+            elsif inside && l.strip == "```"
+              inside = false
+              break
+            elsif inside
+              lines << l.rstrip
+            end
+          end
+          lines
+        end
+
+        # Extract note strings from Markdown blockquote lines.
+        # @param markdown [String]
+        # @return [Array<String>]
+        def extract_notes(markdown)
+          markdown.lines.grep(/^> NOTE:/).map { |l| l.sub(/^> NOTE:\s*/, "").strip }
+        end
+      end
+    end
+  end
+end