mmmd 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8368bf270f4bdba9a1510c3c1f3106a5a56a43bdaf63673219bd7a082de436c6
+  data.tar.gz: f7a9e9a9457be72c8ffae3edfb531d4193be174df5db1714fd215809a4c41825
+SHA512:
+  metadata.gz: 53fea17654973076e1c9fab3256d8034577c04ea684ee64aaa620bf71bd228c98bed317f3c2b7028298a0eae7ccc6cef4587b3b95b83bc52396327f366d01544
+  data.tar.gz: 469aba569708c8e2b88efa34bddf594ac6cd71fb389365e69ff3ef50fc09d7611eaf96c91bbe182ef52ba1c36505bf5df5a51ea733d0231defcaa4b72bb3ffca

data/README.md

@@ -0,0 +1,3 @@
+# rubymark
+
+Modular, compliant Markdown parser in Ruby

data/architecture.md

@@ -0,0 +1,278 @@
+Architecture of madness
+=======================
+
+Prelude
+-------
+
+It needs to be stressed that making the parser modular while keeping it
+relatively simple was a laborous undertaking. There has not been a standard
+more hostile towards the people who dare attempt to implement it than
+CommonMark. It should also be noted, that despite it being titled a
+"Standard" in this document, it is less widely adopted than the Github
+Flavored Markdown syntax. Github Flavored Markdown, however, is only but
+a mere subset of this parser's model, albeit requiring a few extensions.
+
+Current state (as of March 02, 2025)
+------------------------------------
+
+This parser processes text in what can be boiled down to three phases.
+
+- Block/Line phase
+- Overlay phase
+- Inline phase
+
+It should be noted that all phases have their own related parser
+classes, and a shared behaviour system, where each parser takes control
+at some point, and may win ambiguous cases by having higher priority
+(see `#define_child`, `#define_overlay` methods for priority parameter)
+
+### Block/Line phase ###
+
+The first phase breaks down blocks, line by line, into block structures.
+Blocks (preferably inherited from the Block class) can contain other blocks.
+(i.e. QuoteBlock, ULBlock, OLBlock). Other blocks (known as leaf blocks)
+may not contain anything else (except inline content, more on that later).
+
+Blocks are designed to be parsed independently. This means that it *should*
+be possible to tear out any standard block and make it not get parsed.
+This, however, isn't thoroughly tested for.
+
+Blocks as proper, real classes have a certain lifecycle to follow when
+being constructed:
+
+1. Open condition
+    - A block needs to find its first marker on the current line to open
+      (see `#begin?` method)
+    - Once it's open, it's immediately initialized and fed the line it just
+      read (but now as an object, not as a class) (see `#consume` method)
+2. Marker/Line consumption
+    - While it should be kept open, the block parser instance will
+      keep reading inupt through `#consume` method, returning a pair
+      of modified line (after consuming its tokens from it) and
+      a boolean value indicating permission of lazy continuation
+      (if it's a block like a QuoteBlock or ULBlock that can be lazily
+      overflowed).
+      Every line the parser needs to record needs to be pushed
+      through the `#push` method.
+3. Closure
+    - If the current line no longer belongs to the current block
+      (if the block should have been closed on the previous line),
+      it simply needs to `return` a pair of `nil`, and a boolean value for
+      permission of lazy continuation
+    - If a block should be closed on the current line, it should capture it,
+      keep track of the "closed" state, then `return` `nil` on the next call
+      of `#consume`
+    - Once a block is closed, it:
+        1. Receives its content from the parser
+        2. Parser receives the "close" method call
+        3. (optional) Parser may have a callable method `#applyprops`. If
+           it exists, it gets called with the current constructed block.
+        4. (optional) All overlays assigned to this block's class are
+           processed on the contents of this block (more on that in
+           Overlay phase)
+        5. (optional) Parser may return a different class, which
+           the current block should be cast into (Overlays may change
+           the class as well)
+        6. (optional) If a block can respond to `#parse_inner` method, it
+           will get called, allowing the block to parse its own contents.
+    - After this point, the block is no longer touched until the document
+      fully gets processed.
+4. Inline processing
+    - (Applies only to Paragraph and any child of LeafBlock)
+      When the document gets fully processed, the contents of the current
+      block are taken, assigned to an InlineRoot instance, and then parsed
+      in Inline mode
+5. Completion
+    - The resulting document is then returned.
+
+While there is a lot of functionality available in desgining blocks, it is
+not necessary for the simplest of the block kinds available. The simplest
+example of a block parser is likely the ThematicBreakParser class, which
+implements the only 2 methods needed for a block parser to function.
+
+While parsing text, a block may use additional info:
+
+- In consume method: `lazy` hasharg, if the current line is being processed
+  in lazy continuation mode (likely only ever matters for Paragraph); and
+  `parent` - the parent block containing this block.
+
+Block interpretations are tried in decreasing order of their priority
+value, as applied using the `#define_child` method.
+
+For blocks to be properly indexed, they need to be a valid child or
+a valid descendant (meaning reachable through child chain) of the 
+Document class.
+
+### Overlay phase ###
+
+Overlay phase doesn't start at some specific point in time. Rather,
+Overlay phase happens for every block individually - when that block
+closes.
+
+Overlay mechanism can be applied to any DOMObject type, so long as its
+close method is called at some point (this may not be of interest to
+people that do not implement custom syntax, as it generally translates
+to "only block level elements get their overlays processed")
+
+Overlay mechanism provides the ability to perform some action on the block
+right after it gets closed and right before it gets interpreted by the
+inline phase. Overlays may do the following:
+
+- Change the block's class
+  (by returning a class from the `#process` method)
+- Change the block's content (by directly editing it)
+- Change the block's properties (by modifying its `properties` hash)
+
+Overlay interpretations are tried in decreasing order of their priority
+value, as defined using the `#define_overlay` method.
+
+### Inline phase ###
+
+Once all blocks have been processed, and all overlays have been applied
+to their respective block types, the hook in the Document class's
+`#parser` method executes inline parsing phase of all leaf blocks
+(descendants of the `Leaf` class) and paragraphs.
+
+The outer class encompassing all inline children of a block is
+`InlineRoot`. As such, if an inline element is to ever appear within the
+text, it needs to be reachable as a child or a descendant of InlineRoot.
+
+Inline parsing works in three parts:
+
+- First, the contens are tokenized (every parser marks its own tokens)
+- Second, the forward walk procedure is called
+- Third, the reverse walk procedure is called
+
+This process is repeated for every group of parsers with equal priority.
+At one point in time, only all the parsers of equal priority may run in
+the same step. Then, the process goes to the next step, of parsers of
+higher priority value. As counter-intuitive as this is, this means that
+it goes to the parsers of _lower_ priority.
+
+At the very end of the process, the remaining strings are concatenated
+within the mixed array of inlines and strings, and turned into Text
+nodes, after which the contents of the array are appended as children to
+the root node.
+
+This process is recursively applied to all elements which may have child
+elements. This is ensured when an inline parser calls the "build"
+utility method.
+
+The inline parser is a class that implements static methods `tokenize`
+and either `forward_walk` or `reverse_walk`. Both may be implemented at
+the same time, but this isn't advisable.
+
+The tokenization process is characterized by calling every parser in the
+current group with every string in tokens array using the `tokenize`
+method. It is expected that the parser breaks the string down into an
+array of other strings and tokens. A token is an array where the first
+element is the literal text representation of the token, the second
+value is the class of the parser, and the _last_ value (_not third_) is
+the `:close` or `:open` symbol (though functionally it may hold any
+symbol value). Any additional information the parser may need in later
+stages may be stored between the last element and the second element.
+
+Example:
+
+    Input:
+    
+    "_this _is a string of_ tokens_"
+
+    Output:
+
+    [["_", ::PointBlank::Parsing::EmphInline, :open],
+     "this ",
+     ["_", ::PointBlank::Parsing::EmphInline, :open],
+     "is a string of",
+     ["_", ::PointBlank::Parsing::EmphInline, :close],
+     " tokens",
+     ["_", ::PointBlank::Parsing::EmphInline, :close]]
+
+The forward walk is characterized by calling parsers which implement the
+`#forward_walk` method. When the main class encounters an opening token
+in `forward_walk`, it will call the `#forward_walk` method of the class
+that represents this token. It is expected that the parser class will
+then attempt to build the first available occurence of the inline
+element it represents, after which it will return the array of all
+tokens and strings that it was passed where the first element will be
+the newly constructed inline element. If it is unable to close the
+block, it should simply return the original contents, unmodified.
+
+Example:
+
+    Original text:
+
+    this is outside the inline `this is inside the inline` and this
+    is right after the inline `and this is the next inline`
+
+    Input:
+
+    [["`", ::PointBlank::Parsing::CodeInline, :open],
+     "this is inside the inline"
+     ["`", ::PointBlank::Parsing::CodeInline, :close],
+     " and this is right after the inline ",
+     ["`", ::PointBlank::Parsing::CodeInline, :open],
+     "and this is the next inline"
+     ["`", ::PointBlank::Parsing::CodeInline, :close]]
+
+    Output:
+
+    [<::PointBlank::DOM::InlineCode
+      @content = "this is inside the inline">,
+     " and this is right after the inline ",
+     ["`", ::PointBlank::Parsing::CodeInline, :open],
+     "and this is the next inline"
+     ["`", ::PointBlank::Parsing::CodeInline, :close]]
+
+The reverse walk is characterized by calling parsers which implement the
+`#reverse_walk` method when the main class encounters a closing token
+for this class (the one that contains the `:close` symbol in the last
+position of the token information array). After that the main class will
+call the parser's `#reverse_walk` method with the current list of
+tokens, inlines and strings. It is expected that the parser will then
+collect all the blocks, strings and inlines that fit within the block
+closed by the last element in the list, and once it encounters the
+appropriate opening token for the closing token in the last position of
+the array, it will then replace the elements fitting within that inline
+with a class containing all the collected elements. If it is unable to
+find a matching opening token for the closing token in the last
+position, it should simply return the original contents, unmodified.
+
+Example:
+
+    Original text:
+
+    blah blah something something lots of text before the emphasis
+    _this is emphasized `and this is an inline` but it's still
+    emphasized_
+
+
+    Input:
+
+    ["blah blah something something lots of text before the emphasis",
+     ["_", ::PointBlank::Parsing::EmphInline, :open],
+     "this is emphasized",
+     <::PointBlank::DOM::InlineCode,
+      @content = "and this is an inline">,
+     " but it's still emphasized",
+     ["_", ::PointBlank::Parsing::EmphInline, :close]]
+
+    Output:
+
+    ["blah blah something something lots of text before the emphasis",
+     <::PointBlank::DOM::InlineEmphasis,
+      children = [...,
+      <::PointBlank::DOM::InlineCode ...>
+      ...]>]
+
+Both `#forward_walk` and `#reverse_walk` are not restricted to making
+just the changes discussed above, and can arbitrarily modify the token
+arrays. That, however, should be done with great care, so as to not
+accidentally break compatibility with other parsers.
+
+To ensure that the collected tokens in the `#reverse_walk` and
+`#forward_walk` are processes correctly, the colllected arrays of
+tokens, blocks and inlines should be built into an object that
+represents this parser using the `build` method (it will automatically
+attempt to find the correct class to construct using the
+`#define_parser` directive in the DOMObject subclass definition)

data/bin/mmmdpp

@@ -0,0 +1,168 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'io/console/size'
+require 'optionparser'
+require 'json'
+require 'mmmd'
+
+class ParserError < StandardError
+end
+
+class OptionNavigator
+  def initialize
+    @options = {}
+  end
+
+  # Read a definition
+  # @param define [String]
+  def read_definition(define)
+    define.split(";").each do |part|
+      locstring, _, value = part.partition(":")
+      locstring = deconstruct(locstring.strip)
+      assign(locstring, JSON.parse(value))
+    end
+  end
+
+  attr_reader :options
+
+  private
+
+  def check_unescaped(str, index)
+    return true if index.zero?
+
+    reverse_index = index - 1
+    count = 0
+    while str[reverse_index] == "\\"
+      break if reverse_index.zero?
+
+      count += 1
+      reverse_index -= 1
+    end
+    count.even?
+  end
+
+  def find_unescaped(str, pattern, index)
+    found = str.index(pattern, index)
+    return nil unless found
+
+    until check_unescaped(str, found)
+      index = found + 1
+      found = str.index(pattern, index)
+      return nil unless found
+    end
+    found
+  end
+
+  def deconstruct(locstring)
+    parts = []
+    buffer = ""
+    part = nil
+    until locstring.empty?
+      case locstring[0]
+      when '"'
+        raise ParserError, 'separator missing' unless buffer.empty?
+
+        closepart = find_unescaped(locstring, '"', 1)
+        raise ParserError, 'unclosed string' unless closepart
+
+        buffer = locstring[0..closepart]
+        part = buffer[1..-2]
+        locstring = locstring[closepart + 1..]
+      when '.'
+        parts.append(part)
+        buffer = ""
+        part = nil
+        locstring = locstring[1..]
+      when '['
+        raise ParserError, 'separator missing' unless buffer.empty?
+
+        closepart = find_unescaped(locstring, ']', 1)
+        raise ParserError, 'unclosed index' unless closepart
+
+        buffer = locstring[0..closepart]
+        part = locstring[1..-2].to_i
+        locstring = locstring.delete_prefix(buffer)
+      else
+        raise ParserError, 'separator missing' unless buffer.empty?
+
+        buffer = locstring.match(/^[\w_]+/)[0]
+        part = buffer.to_sym
+        locstring = locstring.delete_prefix(buffer)
+      end
+    end
+    parts.append(part) if part
+    parts
+  end
+
+  def assign(keys, value)
+    current = @options
+    while keys.length > 1
+      current_key = keys.shift
+      unless current[current_key]
+        next_key = keys.first
+        case next_key
+        when Integer
+          current[current_key] = []
+        when String
+          current[current_key] = {}
+        when Symbol
+          current[current_key] = {}
+        end
+      end
+      current = current[current_key]
+    end
+    current[keys.shift] = value
+  end
+end
+
+options = {
+  include: [],
+  nav: OptionNavigator.new
+}
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: mmmdpp [OPTIONS] (input|-) (output|-)"
+
+  opts.on("-r", "--renderer [STRING]", String,
+          "Specify renderer to use for this document") do |renderer|
+    options[:renderer] = renderer
+  end
+
+  opts.on("-i", "--include [STRING]", String,
+          "Script to execute before rendering.\
+          May be specified multiple times.") do |inc|
+    options[:include].append(inc)
+  end
+
+  opts.on("-o", "--option [STRING]", String,
+          "Add option string. Can be repeated. Format: <key>: <JSON value>\n"\
+          "<key>: (<\"string\">|<symbol>|<[integer]>)"\
+          "[.(<\"string\"|<symbol>|<[integer]>[...]]\n"\
+          "Example: \"style\".\"CodeBlock\".literal.[0]: 50") do |value|
+    options[:nav].read_definition(value) if value
+  end
+end
+parser.parse!
+
+unless ARGV[1]
+  warn parser.help
+  exit 1
+end
+
+Renderers = {
+  "HTML" => -> { ::MMMD::Renderers::HTML },
+  "Plainterm" => -> { ::MMMD::Renderers::Plainterm }
+}.freeze
+
+options[:include].each { |name| Kernel.load(name) }
+renderer_opts = options[:nav].options
+renderer_opts["hsize"] ||= IO.console_size[1]
+input = ARGV[0] == "-" ? $stdin.read : File.read(ARGV[0])
+output = ARGV[1] == "-" ? $stdout : File.open(ARGV[1], "w")
+doc = MMMD.parse(input)
+rclass = Renderers[options[:renderer] || "Plainterm"]
+raise StandardError, "unknown renderer: #{options[:renderer]}" unless rclass
+
+renderer = rclass.call.new(doc, renderer_opts)
+output.puts(renderer.render)
+output.close