lmt 0.1.2

data/lib/lmt/version.rb

@@ -0,0 +1,3 @@
+module Lmt
+  VERSION = "0.1.2"
+end

data/lmt.gemspec

@@ -0,0 +1,30 @@
+
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "lmt/version"
+
+Gem::Specification.new do |spec|
+  spec.name          = "lmt"
+  spec.license       = "MIT"
+  spec.version       = Lmt::VERSION
+  spec.authors       = ["Marty Gentillon"]
+  spec.email         = ["marty.gentillon+lmt-ruby@gmail.com"]
+
+  spec.summary       = %q{A literate tangler written in Ruby for use with MarkDown.}
+  spec.description   = %q{A literate tangler written in Ruby for use with MarkDown.}
+  spec.homepage      = "https://github.com/MartyGentillon/lmt-ruby"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "bin"
+  spec.executables   = ["lmt", "lmw"]
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.16"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency('rdoc')
+  spec.add_development_dependency('pry')
+  spec.add_dependency('methadone', '~> 1.9.5')
+  spec.add_development_dependency('test-unit')
+end

data/src/lmt/error_reporting.lmd

@@ -0,0 +1,13 @@
+# Error Reporting
+
+A simple method to make sure that errors get reported.
+
+``` ruby report_self_test_failure
+def self.report_self_test_failure(message)
+  if @dev
+    p message
+  else
+    throw message
+  end
+end
+```

data/src/lmt/lmt.rb.lmd

@@ -0,0 +1,652 @@
+# Lmt-Ruby
+
+``` text description
+A literate Markdown tangle tool written in Ruby.
+```
+
+Lmt is a literate Markdown tangle program for [literate programing](https://en.wikipedia.org/wiki/Literate_programming) in a slightly extended [Markdown](http://daringfireball.net/projects/markdown/syntax) syntax that is written in Ruby.
+
+In literate programming, a program is contained within a prose essay describing the thinking which goes into the program.  The source code is then extracted from the essay using a program called tangle (this application).  The essay can also be formatted into a document for human consumption using a program called "weave" and can be found [here](lmm.md).
+
+## Why?
+
+While there are other Markdown tanglers available (especially [lmt](https://github.com/driusan/lmt), which this program is designed to be superficially similar to) none quite match the combination of simplicity and extensibility which I need.
+
+## Features
+
+In order to be useful for literate programming we need a few features:
+
+1. The ability to strip code out of a Markdown file and place it into a tangled output file.
+2. The ability to embed macros so that the code can be expressed in any order desired.
+3. The ability to apply filters on the contents of a macro
+4. The ability to to identify code blocks which will be expanded when referenced
+5. The ability to append to or replace code blocks
+6. The ability to include another file.
+
+There are also a few potentially useful features that are not implemented but might be in the future:
+
+1. The ability to extend the tangler with Ruby code from a block.
+2. The ability to write out other files.
+3. Source mapping
+4. Further source verification.  For instance, all instances of the same block should be in the same language.  Also, detect and prevent double inclusion.
+
+Also, the only filter currently existing just escapes strings for ruby code.  There are many more that could be useful.
+
+### Blocks
+
+Markdown already supports code blocks expressed with code fences starting with three backticks, usually enabling syntax highlighting on the output.  This should work excellently for identifying block boundaries.
+
+There are two types of blocks: the default block and macro blocks.
+
+Ouput begins with the default block.  It is simply a markdown code block which has no macro name.  with no further information.  It looks like this.
+
+``` ruby
+#Output starts here
+```
+
+If there is no default block, then no output file will be created.
+
+In order to add the macro feature we need, we will need to add header content at the beginning of such a quote. For code blocks, we can add it after the language name.  For example to create a macro named macro_description we could use:
+
+```ruby macro_description
+# this shouldn't be in the output, it should have been replaced.
+block_replacement = false
+```
+
+Of course, these do not play well with Markdown rendering, so we will need a weaver to display the name appropriately.
+
+To replace a block put `=` before the block name like so:
+
+```ruby =macro_description
+# this is the replacement
+replaced_block = true
+```
+
+To append to a block, just open it again.  The macro expansion only happens after the entire file is read.
+
+```ruby macro_description
+# Yay appended code gets injected
+block_appendment = true
+```
+
+#### Macros
+
+We will also need a way to trigger macro insertion.  Given that unicode tends not to be in use, why don't we say that anything inside `⦅⦆` refers to a block by name and should be replaced by the contents of that block.
+
+```ruby macro_insertion_description
+block_replacement = true
+replaced_block = false
+block_appendment = false
+
+⦅macro_description⦆
+```
+
+Given the definition of `macro_description` above, all the variables will be true at the end of that block.
+
+This also works with spaces inside the `⦅⦆`
+
+``` ruby macro_insertion_description
+insertion_works_with_spaces = false
+⦅ insertion_works_with_spaces ⦆
+```
+
+Finally, if substitution isn't desired, you may escape the `⦅` and `⦆` with `\` which will prevent macro expansion.  As below; the first character of escaped string is `⦅`
+
+``` ruby macro_insertion_description
+escaped_string = '\⦅macro_description\⦆'
+```
+
+### Filters
+
+Filters can be defined as functions which take an array of lines and return the altered array.  They are applied after a macro's contents are expanded and before it is inserted.  They are triggered with the `|` symbol in expansion. for example: given
+
+``` text string_with_backslash
+this string ends in \.
+```
+
+The following will escape the `\`
+
+``` ruby filter_use_description
+string_with_backslash = "⦅string_with_backslash | ruby_escape⦆"
+```
+
+There are a few built in filters:
+
+``` ruby filter_list
+{
+  'ruby_escape' => ⦅ruby_escape⦆
+}
+```
+
+### Includes
+
+Other files may be using an include directive and a markdown link.  Include directive are lines starting with `! include` followed by a space.  No further text may follow the markdown link.  Paths are relative to the file being included from.
+
+During tangle the link line will be replaced with the lines from the included file.  This means that they may replace blocks defined in the file that includes them such as this one
+
+``` ruby included_block
+included_string = "I am in lmt.lmd"
+```
+
+! include [an include](lmt_include.lmd)
+
+### Self Test
+
+Of course, we will also need a testing procedure.  Since this is written as a literate program, our test procedure is: can we tangle ourself.  If the output of the tangler run on this file can tangle this file, then we know that the tangler works.
+
+``` ruby self_test
+def self.self_test()
+  ⦅test_description⦆
+end
+```
+
+## Interface
+
+We need to know where to get the input from and where to send the output to.  For that, we will use the following command line options
+
+``` ruby options
+on("--file FILE", "-f", "Required: input file")
+on("--output FILE", "-o", "Required: output file")
+on("--dev", "disables self test failure for development")
+```
+
+Of which, both are required
+
+``` ruby options
+required(:file, :output)
+```
+
+## Implementation and Example
+
+Now for an example in implementation.  Using Ruby we can write a template as below:  (We are replacing the default block because the version above doesn't have a #!.)
+
+```ruby =
+#!/usr/bin/env ruby
+# Encoding: utf-8
+
+⦅includes⦆
+
+module Lmt
+
+class Tangle
+  include Methadone::Main
+  include Methadone::CLILogging
+
+  @dev = false
+
+  main do
+    check_arguments()
+    begin
+      ⦅main_body⦆
+    rescue Exception => e
+      puts "Error: #{e.message} #{extract_causes(e)}At:"
+      e.backtrace.each do |trace|
+        puts "    #{trace}"
+      end
+    end
+  end
+
+  def self.extract_causes(error)
+    if (error.cause)
+      "  Caused by: #{error.cause.message}\n#{extract_causes(error.cause)}"
+    else
+      ""
+    end
+  end
+
+  ⦅self_test⦆
+
+  ⦅report_self_test_failure⦆
+  
+  ⦅filter_class⦆
+
+  ⦅tangle_class⦆
+
+  ⦅option_verification⦆
+
+  description "⦅description⦆"
+  ⦅options⦆
+
+  version Lmt::VERSION
+
+  use_log_level_option :toggle_debug_on_signal => 'USR1'
+
+  go! if __FILE__ == $0
+end
+
+end
+
+```
+
+This is a basic template using the [Ruby methadone](https://github.com/davetron5000/methadone) command line application framework and making sure that we report errors (because silent failure sucks).
+
+The main body will first test itself then, invoke the library component, which isn't in lib as traditional because it is in this file and I don't want to move it around.
+
+``` ruby main_body
+self_test()
+tangler = Tangle::Tangler.new(options[:file])
+tangler.tangle()
+tangler.write(options[:output])
+```
+
+Finally, we have the dependencies.  Optparse and methadone are used for cli argument handling and other niceties.
+
+``` ruby includes
+require 'optparse'
+require 'methadone'
+require 'lmt/version'
+```
+
+There, now we are done with the boilerplate. On to:
+
+## The Actual Tangler
+
+The tangler is defined within a class that contains the tangling implementation.  It contains the following blocks
+
+``` ruby tangle_class
+class Tangler
+  class << self
+    attr_reader :filters
+  end
+
+  @filters = ⦅filter_list⦆
+
+  ⦅initializer⦆
+  ⦅tangle⦆
+  ⦅read_file⦆
+  ⦅include_includes⦆
+  ⦅parse_blocks⦆
+  ⦅expand_macros⦆
+  ⦅apply_filters⦆
+  ⦅unescape_double_parens⦆
+  ⦅write⦆
+
+  private
+  ⦅tangle_class_privates⦆
+end
+```
+
+### Initializer
+
+The initializer takes in the input file and sets up our state.  We are keeping the unnamed top level block separate from the rest.  Then we have a hash of blocks.  Finally, we need to make sure we have tangled before we write the output.
+
+``` ruby initializer
+def initialize(input)
+  @input = input
+  @block = ""
+  @blocks = {}
+  @tangled = false
+end
+
+```
+
+### Tangle
+
+Now we have the basic tangle process wherein a file is read, includes are substituted, the blocks extracted, macros expanded recursively, and escaped double parentheses unescaped.  If there is no default block, then there is no further work to be done.
+
+``` ruby tangle
+def tangle()
+  contents = include_includes(read_file(@input))
+  @block, @blocks = parse_blocks(contents)
+  if @block
+    @block = expand_macros(@block)
+    @block = unescape_double_parens(@block)
+  end
+  @tangled = true
+end
+
+```
+
+### Reading The File
+
+This is fairly self explanatory, though note, we are storing the file in memory as an array of lines.
+
+``` ruby read_file
+def read_file(file)
+  File.open(file, 'r') do |f|
+    f.readlines
+  end
+end
+
+```
+
+### Including the Includes
+
+As our specification is a regular language (we do not support any kind of nesting), we will be using regular expressions to process it.  Those expressions are detailed in:
+
+! include [here](lmt_expressions.lmd)  
+
+Here we go through each line looking for an include statement.  When we find one, we replace it with the lines from that file.  Those lines will, of course, need to have includes processed as well.
+
+``` ruby include_includes
+def include_includes(lines, current_file = @input, depth = 0)
+  raise "too many includes" if depth > 1000
+  include_exp = ⦅include_expression⦆
+  lines.map do |line|
+    match = include_exp.match(line)
+    if match
+      file = File.dirname(current_file) + '/' + match[1]
+      include_includes(read_file(file), file, depth + 1)
+    else
+      [line]
+    end
+  end.flatten(1)
+end
+
+```
+
+### Parsing The Blocks
+
+Now we get to the meat of the algorithm.  This uses the regular expression in [lmt_expressions](lmt_expressions.lmd#The-Code-Block-Expression)
+
+First, we filter out all non block lines, keeping the block headers, slice it into separate blocks at the header, process the header and turn it into a map of lists of lines.  We then group by the headers and combine the blocks which follow the last reset for that block name.
+
+We also need to remove the last newline from the block as it causes problems when injecting a block onto a line with stuff after the end.
+
+Finally, (after making sure we aren't missing a code fence) we extract the unnamed block from the hash and return both it and the rest.
+
+``` ruby parse_blocks
+def parse_blocks(lines)
+  code_block_exp = ⦅code_block_expression⦆
+  in_block = false
+  blocks = lines.find_all do |line|
+    in_block = !in_block if line =~ code_block_exp
+    in_block
+  end.slice_before do |line|
+    code_block_exp =~ line
+  end.map do |(header, *rest)|
+    white_space, language, replacement_mark, name = code_block_exp.match(header)[1..-1]
+    [name, replacement_mark, rest]
+  end.group_by do |(name, _, _)|
+    name
+  end.transform_values do |bodies|
+    last_replacement_index = get_last_replacement_index(bodies)
+    bodies[last_replacement_index..-1].map { |(_, _, body)| body}
+      .flatten(1)
+  end.transform_values do |body_lines|
+    body_lines[-1] = body_lines[-1].chomp if body_lines[-1]
+    body_lines
+  end
+  throw "Missing code fence" if in_block
+  main = blocks[""]
+  blocks.delete("")
+  [main, blocks]
+end
+
+```
+
+We have a private helper helper method here.  So, after we turn each block chunk into an array of `[name, replacement_mark, body]` we can find the last one by scanning for a replacement mark set to `=`.  Otherwise the answer is `0` as there is no replacement index.
+
+``` ruby tangle_class_privates
+def get_last_replacement_index(bodies)
+  last_replacement = bodies.each_with_index
+      .select do |((_, replacement_mark, _), _)|
+        replacement_mark == '='
+      end[-1]
+  if last_replacement
+    last_replacement[1]
+  else
+    0
+  end
+end
+
+```
+
+### Handling the macros
+
+The other half of the meat.  Here we use two regular expressions.  One to identify and propagate whitespace and the other to actually find the replacements in a line.
+
+This is implemented by splitting the line on the replacement section, grouping into pairs, and then reducing.  Afterwords, we end up with an extra layer of lists which need to be flattened.  (Yes I am using a monad and bind.)
+
+``` ruby expand_macros
+def expand_macros(lines, depth = 0)
+  throw "too deep macro expansion {depth}" if depth > 1000
+  lines.map do |line|
+    begin
+      expand_macro_on_line(line, depth)
+    rescue Exception => e
+      raise Exception, "Failed to process line: #{line}", e.backtrace
+    end
+  end.flatten(1)
+end
+
+```
+
+Expand_macro_on_line turns a line into a list of lines.  The collected results will have to be flattened by 1.
+
+First we process the white space off the front of the expression.  This will be added to each line in the extended macros so that the output file is nicely indented.  It also means that indentation sensitive languages like python will be tangled correctly.
+
+``` ruby tangle_class_privates
+def expand_macro_on_line(line, depth)
+  white_space_exp = /^(\s*)(.*\n?)/
+  macro_substitution_exp = ⦅macro_substitution_expression⦆
+  filter_extraction_exp = / *\| *([-\w]+) */
+  white_space, text = white_space_exp.match(line)[1..2]
+```
+
+Then we chop it into pieces using the [macro substitution expression](lmt_expressions.lmd#The-Macro-Substitution-Expression) This results in text, macro_name / filter pairs.  If there is a macro name, we then split the filter names off with the filter expression which provides filter names followed by stuff between them (nothing) which we discard.
+
+``` ruby tangle_class_privates
+  section = text.split(macro_substitution_exp)
+      .each_slice(2)
+      .map do |(text_before_macro, macro_match)|
+        if (macro_match)
+          macro_name, *filters = macro_match.strip.split(filter_extraction_exp)
+          [text_before_macro, macro_name, filters.each_slice(2).map(&:first)]
+        else
+          [text_before_macro]
+        end
+```
+
+Finally, we are ready to actually process the text and macros.  We build the list of ines with just the white space, and appending the results of precessing to the end.  Each potential line is built up by appending to the end of the last line.  If there is no macro, then we can just append the text.  
+
+``` ruby tangle_class_privates
+      end.inject([white_space]) do
+        |(*new_lines, last_line), (text_before_macro, macro_name, filters)|
+        if macro_name.nil?
+          last_line = "" unless last_line
+          new_lines << last_line + text_before_macro
+        else
+```
+
+If there is a macro substitution, first we get the new lines.  The we append the first line of the macro text to the last line.  Finally, we append the white space to the front of each of the macro's lines and insert them into the middle.  of the list of lines we are building.
+
+``` ruby tangle_class_privates
+          throw "Macro '#{macro_name}' unknown" unless @blocks[macro_name]
+          macro_lines = apply_filters(
+              expand_macros(@blocks[macro_name], depth + 1), filters)
+          unless macro_lines.empty?
+            new_line = last_line + text_before_macro + macro_lines[0]
+            macro_continued = macro_lines[1..-1].map do |macro_line|
+              white_space + macro_line
+            end
+            (new_lines << new_line) + macro_continued
+          else
+            new_lines
+          end
+        end
+      end
+end
+```
+
+Finally, throughout this process, we have to be ware that a macro may have no content.  We must deal with `nil` and empty lists where they occur.
+
+### Unescaping Double Parentheses
+
+This is fairly self explanatory, gsub is global substitution.  We need three `\`s two to match the escape sequence for `\` in ruby and a third to handle the escaped `⦅` and `⦆` when this file itself is tangled.
+
+``` ruby unescape_double_parens
+def unescape_double_parens(block)
+  block.map do |l|
+    l = l.gsub("\\\⦅", "⦅")
+    l = l.gsub("\\\⦆", "⦆")
+    l
+  end
+end
+
+```
+
+### Write The Output
+
+Finally, if there is a default block, write the output.
+
+``` ruby write
+def write(output)
+  tangle() unless @tangled
+  if @block
+    fout = File.open(output, 'w')
+    @block.each {|line| fout << line}
+  end
+end
+
+```
+
+## The Filters
+
+The filters are instances of the Filter class which can be created by passing a block to the initializer of the class.  When the filter is executed, this block of code will be called on all of the lines of code being filtered.
+
+``` ruby filter_class
+class Filter
+  def initialize(&block)
+    @code = block;
+  end
+
+  def filter(lines)
+    @code.call(lines)
+  end
+end
+```
+
+Because it is fairly common to filter lines one at a time, LineFilter will pass in each line instead of the whole block.
+
+``` ruby filter_class
+class LineFilter < Filter
+  def filter(lines)
+    lines.map do |line|
+      @code.call(line)
+    end
+  end
+end
+```
+
+Filters are applied by the following method:
+
+``` ruby apply_filters
+def apply_filters(strings, filters)
+  filters.map do |filter_name|
+    Tangler.filters[filter_name]
+  end.inject(strings) do |strings, filter|
+    filter.filter(strings)
+  end
+end
+```
+
+
+### Ruby Escape
+
+Ruby escape escapes strings appropriately for Ruby.  
+
+``` ruby ruby_escape
+LineFilter.new do |line|
+  line.dump[1..-2]
+end
+```
+
+## Option Verification
+
+Option verification is described here:
+
+! include [Option verification](option_verification.lmd)
+
+## Self Test, Details
+
+So, now we need to go into details of our self test and also include regressions which have caused problems.
+
+First, we need a method to report test failures:
+
+! include [Error reporting](error_reporting.lmd)
+
+Then we need the tests we are doing.  The intentionally empty block is included both at the beginning and end to make sure that we handled all the edge cases related to empty blocks appropriately.
+
+``` ruby test_description
+⦅intentionally_empty_block⦆
+⦅test_macro_insertion_description⦆
+⦅test_filters⦆
+⦅test_inclusion⦆
+⦅intentionally_empty_block⦆
+```
+
+### Testing: Macros
+
+At [the top of the file](#Macros), we described the macros.  Lets make sure that works by ensuring the variables are as they were described above
+
+``` ruby test_macro_insertion_description
+⦅macro_insertion_description⦆
+# These require the code in the macro to work.
+report_self_test_failure("block replacement doesn't work") unless block_replacement and replaced_block
+report_self_test_failure("appending to macros doesn't work") unless block_appendment
+report_self_test_failure("insertion must support spaces") unless insertion_works_with_spaces
+report_self_test_failure("double parentheses may be escaped") unless escaped_string[0] != '\\'
+```
+
+Finally, we need to make sure two macros on the same line works.
+
+``` ruby test_macro_insertion_description
+two_macros = "⦅foo⦆ ⦅foo⦆"
+report_self_test_failure("Should be able to place two macros on the same line") unless two_macros == "foo foo"
+```
+
+For that to work we need:
+
+``` ruby insertion_works_with_spaces
+insertion_works_with_spaces = true
+```
+
+and
+
+``` ruby foo
+foo
+```
+
+### Testing: Filters
+
+At the [top of the file](Filters) we described the usage of filters.  Let's make sure that works.  The extra `.?` in the regular expression is a workaround for an editor bug in Visual Studio Code, where, apparently, `/\\/` escapes the `/` rather than the `\`.... annoying.
+
+``` ruby test_filters
+⦅filter_use_description⦆
+report_self_test_failure("ruby escape doesn't escape backslash") unless string_with_backslash =~ /\\.?/
+```
+
+### Testing: Inclusion
+
+``` ruby test_inclusion
+⦅included_block⦆
+report_self_test_failure("included replacements should replace blocks") unless included_string == "I came from lmt_include.lmd"
+```
+
+### Regressions
+
+Some regressions / edge cases that we need to watch for.  These should not break our tangle operation.
+
+#### Empty Blocks
+
+We need to be able to tangle empty blocks such as:
+
+``` ruby intentionally_empty_block
+```
+
+#### Unused blocks referencing nonexistent blocks
+
+If a block is unused, then don't break if it uses a nonexistent block.
+
+``` ruby unused_block
+⦅this_block_does_not_exist⦆
+```
+
+## Fin ┐( ˘_˘)┌
+
+And with that, we have tangled a file.
+
+At current, there are a few more features that would be nice to have. First, this does not yet support extension by commands.  Second, we cannot write to any file other than the output file.  Third, we don't have many filters.  These features can wait for now.
+
+∎