lmt 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 92a8904e1d7de30d5a09b28df29ffa7b164f4afe7e8bf78806c7c94a8381804a
-  data.tar.gz: 9cfeaf0336b6e7b4162162aa258f7082a95078ac3299153d217fccb7b110bfcb
+  metadata.gz: 20c0cdd948065ea59b76df0f83bd7abd0d7df99a55a8532a1a7368d3e6e46924
+  data.tar.gz: baff975ca361ba930e73a815283f824fb4f5d913a9f54b303a64ee0622be6b3a
 SHA512:
-  metadata.gz: db3d8d46c342cbde06a564b7db00fe674d0eda158edc1f95bf8263e7b641739fe72ff3f1568dab3d7b79e12cc0b6a3793f9aa58c5b2c2844d1b188772c27d26e
-  data.tar.gz: a3d8b02b2d7cbdd2aadb112149f65b1b7cb31d91649a16a176166014888e761c67924eb53e2ac4e4455773d29b70c562d0f3c018c72f0777118d6d7dd1f04076
+  metadata.gz: 2c415acb8d48be392624f7b64648b9873490cb6282e4f90969a63290e8aa13a4ee6c7cb6a03a05cc177339a3b917a66be894724621d65d1bd80018ddc2e49f6c
+  data.tar.gz: dec0d39c012cfa8c3f57955744a0cf9909c2f755c36ed0e893abdf684b20ebdd224bdbbd6c988129b9727c31947cabf5cf4fe468b64a7683efe1539e6349eb1e

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    lmt (0.1.2)
+    lmt (0.1.3)
       methadone (~> 1.9.5)
 
 GEM

data/README.md

@@ -29,13 +29,13 @@ Or install it yourself as:
 The tangle program takes input files and produces tangled output files.  It is used as follows:
 
 ``` bash
-bin/lmt --file {input file} --output {tangled destination}
+lmt --file {input file} --output {tangled destination}
 ```
 
 The weave program is similar but produces weaved output files.  It does not recurse down include statements, and so will need to be run independently for each included file.  An example usage:
 
 ``` bash
-bin/lmw --file {input file} --output [weaved destination]
+lmw --file {input file} --output [weaved destination]
 ```
 
 ## Development
@@ -56,6 +56,12 @@ To test the weave you can use the following command which will weave the weaver
 bundle exec ruby bin/lmt --file src/lmt/lmt.rb.lmd --output lib/lmt/lmt.rb; bundle exec ruby bin/lmw --file src/lmt/lmw.rb.lmd --output doc/lmt/lmw.rb.md
 ```
 
+Since this is a self-bootstraping program, it is both tested and built by running itself on itself.  This means that if you add a bug, it won't run.  To fix this, check out the most recent version of the output file out of git.
+
+``` bash
+git co -- src/lmt/lmt.rb
+```
+
 ## Prior Art
 
 Some related and similar tools that the reader might find interesting:

data/Rakefile

@@ -42,6 +42,7 @@ task :install => :build
 
 task :build => :tangle
 task :build => :weave
+task :weave => :tangle
 
 lmd_files = Rake::FileList['src/**/*.lmd']
 outputs = lmd_files.pathmap('%{^src,lib}X')
@@ -53,10 +54,10 @@ task :weave => docs
 lmd_files.zip(outputs, docs).each do |lmd_file, output, doc|
   directory output_dir = output.pathmap('%d')
   directory doc_dir = doc.pathmap('%d')
-  file output => [output_dir, lmd_file] do
-    sh "ruby bin/lmt --file #{lmd_file} --output #{output}"
+  file output => [output_dir, lmd_file] do #FIXME LMT steps need to be aware of includes
+    sh "ruby bin/lmt --file #{lmd_file} --output #{output} --dev"
   end
-  file doc => [doc_dir, lmd_file] do
+  file doc => ["lib/lmt/lmw.rb", doc_dir, lmd_file] do
     sh "ruby bin/lmw --file #{lmd_file} --output #{doc}"
   end
 end

data/doc/lmt/lmt.rb.md

@@ -24,15 +24,15 @@ In order to be useful for literate programming we need a few features:
 4. The ability to to identify code blocks which will be expanded when referenced
 5. The ability to append to or replace code blocks
 6. The ability to include another file.
+7. The ability to extend the tangler with Ruby code from a block.
+8. Simple conditional logic to enable output only under certain circumstances
 
 There are also a few potentially useful features that are not implemented but might be in the future:
 
-1. The ability to extend the tangler with Ruby code from a block.
-2. The ability to write out other files.
-3. Source mapping
-4. Further source verification.  For instance, all instances of the same block should be in the same language.  Also, detect and prevent double inclusion.
-
-Also, the only filter currently existing just escapes strings for ruby code.  There are many more that could be useful.
+1. The ability to write out other files.
+2. Source mapping
+3. Further source verification.  For instance, all instances of the same block should be in the same language.  Also, detect and prevent double inclusion.
+4. include path semantics.
 
 ### Blocks
 
@@ -40,7 +40,7 @@ Markdown already supports code blocks expressed with code fences starting with t
 
 There are two types of blocks: the default block and macro blocks.
 
-Ouput begins with the default block.  It is simply a markdown code block which has no macro name.  with no further information.  It looks like this.
+Output begins with the default block.  It is simply a markdown code block which has no macro name.  with no further information.  It looks like this.
 
 ###### Output Block
 
@@ -136,7 +136,11 @@ There are a few built in filters:
 
 ``` ruby
 {
-  'ruby_escape' => ⦅ruby_escape⦆
+  'ruby_escape' => ⦅ruby_escape⦆,
+  'double_quote' => ⦅double_quote⦆,
+  'add_comma' => ⦅add_comma⦆,
+  'indent_continuation' => ⦅indent_continuation⦆,
+  'indent_lines' => ⦅indent_lines⦆
 }
 ```
 
@@ -154,6 +158,65 @@ included_string = "I am in lmt.lmd"
 
 **See include:** [lmt_include.lmd](include_file)
 
+### Extension
+
+In order to extend the tangler, it must be possible to mark a block for evaluation.  To do so we will extend the mechanism to indicate replacement.  Let's use `!`.  A block simply named `!` will just be executed within a contained scope.  Within this scope, it will be possible to access the map of filters through the `@filters` variable.  All of these blocks are executed and removed from the stream before any further processing is done.
+
+However, after blocks have been parsed, the map of blocks will be passed to the `parse_hook` method which an extension may define.  It will be passed a two arguments.  The first is an array with the lines of the main block, the second is a map of block name to line arrays.  It is expected to return the same data structure in a two value array.  An example parse hook which adds a block to the list of know blocks follows:
+
+###### Execute Extension Block
+
+``` ruby
+def parse_hook(main_block, blocks)
+  blocks["from_extension"] = ["from_extension = true\n"]
+  [main_block, blocks]
+end
+```
+
+### Conditional Output
+
+Under certain circumstances it is useful to have certain output only happen under certain circumstances.  For instance, a file prepared for Windows might have slightly different content than the same file prepared for Linux.  In order to enable this, a variable may be set within an extension block and then output may be enabled / disabled using directives based on them.
+
+###### Execute Extension Block
+
+``` ruby
+@a_variable = true
+@another_variable = false
+```
+
+We can then disable and enable output using the if, else, elsif, and end directives.  The if directive takes a line of ruby code, executes it.
+
+! if @a_variable
+
+Since a_variable is true, the next block will be processed.
+
+###### Code Block: Conditional Output
+
+``` ruby
+conditional_output_else = true
+conditional_output_elsif = true
+```
+
+! elsif @another_variable
+
+###### Code Block: Conditional Output
+
+``` ruby
+conditional_output_elsif = false
+```
+
+! else
+
+And the following block will have no effect.
+
+###### Code Block: Conditional Output
+
+``` ruby
+conditional_output_else = false
+```
+
+! end
+
 ### Self Test
 
 Of course, we will also need a testing procedure.  Since this is written as a literate program, our test procedure is: can we tangle ourself.  If the output of the tangler run on this file can tangle this file, then we know that the tangler works.
@@ -234,6 +297,8 @@ class Tangle
 
   ⦅tangle_class⦆
 
+  ⦅context_class⦆
+
   ⦅option_verification⦆
 
   description "⦅description⦆"
@@ -257,6 +322,7 @@ The main body will first test itself then, invoke the library component, which i
 ###### Code Block: Main Body
 
 ``` ruby
+@dev = options[:dev]
 self_test()
 tangler = Tangle::Tangler.new(options[:file])
 tangler.tangle()
@@ -283,16 +349,11 @@ The tangler is defined within a class that contains the tangling implementation.
 
 ``` ruby
 class Tangler
-  class << self
-    attr_reader :filters
-  end
-
-  @filters = ⦅filter_list⦆
-
   ⦅initializer⦆
   ⦅tangle⦆
   ⦅read_file⦆
   ⦅include_includes⦆
+  ⦅handle_extensions_and_conditionals⦆
   ⦅parse_blocks⦆
   ⦅expand_macros⦆
   ⦅apply_filters⦆
@@ -300,7 +361,12 @@ class Tangler
   ⦅write⦆
 
   private
+  def filters_map
+    @extension_context.filters
+  end
   ⦅tangle_class_privates⦆
+
+  ⦅conditional_processor⦆
 end
 ```
 
@@ -312,6 +378,8 @@ The initializer takes in the input file and sets up our state.  We are keeping t
 
 ``` ruby
 def initialize(input)
+  @extension_context = Context.new()
+  @extension_context.filters = ⦅filter_list⦆
   @input = input
   @block = ""
   @blocks = {}
@@ -322,14 +390,15 @@ end
 
 ### Tangle
 
-Now we have the basic tangle process wherein a file is read, includes are substituted, the blocks extracted, macros expanded recursively, and escaped double parentheses unescaped.  If there is no default block, then there is no further work to be done.
+Now we have the basic tangle process wherein a file is read, includes are substituted, extensions and conditionals are processed, the blocks extracted, the extension hook called, macros expanded recursively, and escaped double parentheses unescaped.  If there is no default block, then there is no further work to be done.
 
 ###### Code Block: Tangle
 
 ``` ruby
 def tangle()
-  contents = include_includes(read_file(@input))
-  @block, @blocks = parse_blocks(contents)
+  contents =  handle_extensions_and_conditionals(
+      include_includes(read_file(@input)))
+  @block, @blocks = @extension_context.parse_hook(*parse_blocks(contents))
   if @block
     @block = expand_macros(@block)
     @block = unescape_double_parens(@block)
@@ -381,6 +450,122 @@ end
 
 ```
 
+### Evaling the Extensions and Processing the Conditionals
+
+The extensions are executed within the following context.  This context is also
+used to evaluate conditionals.
+
+###### Code Block: Context Class
+
+``` ruby
+class Context
+  attr_accessor :filters
+
+  def get_binding
+    binding
+  end
+
+  def parse_hook(main_block, blocks)
+    [main_block, blocks]
+  end
+end
+
+```
+
+Because conditional processing must occur concurrently with bock evaling, we have to build up each block and eval it the moment it is complete.  To do so, we find all the lines that are not in an extension block.  When we enter a new extension block, we clear the current extension block, and when we leave an extension block, we eval it.
+
+###### Code Block: Handle Extensions And Conditionals
+
+``` ruby
+def handle_extensions_and_conditionals(lines)
+  extension_expression = ⦅extension_expression⦆
+  condition_processor = ConditionalProcessor.new(@extension_context)
+  extension_exit_expression = /```/
+  in_extension_block = false
+  current_extension_block = []
+
+  other_lines = lines.lazy
+    .find_all do |line|
+      condition_processor.should_output(line)
+    end.find_all do |line|
+      unless in_extension_block
+        in_extension_block = line =~ extension_expression
+        if in_extension_block
+          current_extension_block = []
+        end
+        !in_extension_block
+      else
+        in_extension_block = !(line =~ extension_exit_expression)
+        if in_extension_block
+          current_extension_block << line
+        else
+          @extension_context.get_binding.eval(current_extension_block.join)
+        end
+        false
+      end
+    end.force
+
+  condition_processor.check_block_balance()
+
+  other_lines
+end
+
+```
+
+### Processing The Conditionals
+
+To process the conditionals, we need a stack.  Given that we are handling if, elsif, and else, we will need to track: 1) the type of statement (in case we want to add loops later), 2) the state before we encounter the if and, 3) if the else should be executed.  For if statements, we can store these in an array like `[type, prior_state, execute_else]`
+
+Since this process happens concurrently with evaling the included blocks, it's process is represented by a class.  Should_output is the inside of the filter statement which is used to filter the lines.
+
+###### Code Block: Conditional Processor
+
+``` ruby
+class ConditionalProcessor
+
+  def initialize(extension_context)
+    @if_expression = ⦅if_expression⦆
+    @elsif_expression = ⦅elsif_expression⦆
+    @else_expression = ⦅else_expression⦆
+    @end_expression = ⦅end_expression⦆
+    @output_enabled = true
+    @stack = []
+    @extension_context = extension_context
+  end
+
+  def should_output(line)
+    case line
+      when @if_expression
+        condition = $1
+        prior_state = @output_enabled
+        @output_enabled = !!@extension_context.get_binding.eval(condition)
+        @stack.push([:if, prior_state, !@output_enabled])
+      when @elsif_expression
+        throw "elsif statement missing if"  if @stack.empty?
+        condition = $1
+        type, prior_state, execute_else = @stack.pop()
+        @output_enabled = execute_else && !!@extension_context.get_binding.eval(condition)
+        @stack.push([type, prior_state, execute_else && !@output_enabled])
+      when @else_expression
+        throw "else statement missing if" if @stack.empty?
+        type, prior_state, execute_else = @stack.pop()
+        @output_enabled = execute_else
+        @stack.push([type, prior_state, execute_else])
+      when @end_expression
+        throw "end statement missing begin" if @stack.empty?
+        type, prior_state, execute_else = @stack.pop()
+        @output_enabled = prior_state
+    end
+    @output_enabled
+  end
+  
+  def check_block_balance
+    throw "unbalanced blocks" unless @stack.empty?
+  end
+end
+
+```
+
 ### Parsing The Blocks
 
 Now we get to the meat of the algorithm.  This uses the regular expression in [lmt_expressions](lmt_expressions.lmd#The-Code-Block-Expression)
@@ -604,14 +789,13 @@ Filters are applied by the following method:
 ``` ruby
 def apply_filters(strings, filters)
   filters.map do |filter_name|
-    Tangler.filters[filter_name]
+    filters_map[filter_name]
   end.inject(strings) do |strings, filter|
     filter.filter(strings)
   end
 end
 ```
 
-
 ### Ruby Escape
 
 Ruby escape escapes strings appropriately for Ruby.  
@@ -624,6 +808,58 @@ LineFilter.new do |line|
 end
 ```
 
+### Double Quote
+
+Double quote surrounds strings in double quotes
+
+###### Code Block: Double Quote
+
+``` ruby
+LineFilter.new do |line|
+  before_white = /^\s*/.match(line)[0]
+  after_white = /\s*$/.match(line)[0]
+  "#{before_white}\"#{line.strip}\"#{after_white}"
+end
+```
+
+### Add Commas
+
+Add commas adds comma to the end of each line.
+
+###### Code Block: Add Comma
+
+``` ruby
+LineFilter.new do |line|
+  before_white = /^\s*/.match(line)[0]
+  after_white = /\s*$/.match(line)[0]
+  "#{before_white}#{line.strip},#{after_white}"
+end
+```
+
+### Indent continuation
+
+Adds two spaces to the front of each line after the first
+
+###### Code Block: Indent Continuation
+
+``` ruby
+Filter.new do |lines|
+  [lines[0], *lines[1..-1].map {|l| "  #{l}"}]
+end
+```
+
+### Indent Lines
+
+Adds two spaces to the front of each line
+
+###### Code Block: Indent Lines
+
+``` ruby
+LineFilter.new do |line|
+  "  #{line}"
+end
+```
+
 ## Option Verification
 
 Option verification is described here:
@@ -648,6 +884,8 @@ Then we need the tests we are doing.  The intentionally empty block is included
 ⦅test_filters⦆
 ⦅test_inclusion⦆
 ⦅intentionally_empty_block⦆
+⦅test_extensions⦆
+⦅test_conditional_output⦆
 ```
 
 ### Testing: Macros
@@ -694,11 +932,30 @@ foo
 
 At the [top of the file](Filters) we described the usage of filters.  Let's make sure that works.  The extra `.?` in the regular expression is a workaround for an editor bug in Visual Studio Code, where, apparently, `/\\/` escapes the `/` rather than the `\`.... annoying.
 
+###### Code Block: Some Text
+
+``` text
+some text
+```
+
+###### Code Block: A List
+
+``` text
+item 1
+item 2
+```
+
 ###### Code Block: Test Filters
 
 ``` ruby
 ⦅filter_use_description⦆
 report_self_test_failure("ruby escape doesn't escape backslash") unless string_with_backslash =~ /\\.?/
+some_text = ⦅some_text | double_quote⦆
+report_self_test_failure("Double quote doesn't double quote") unless some_text == "⦅some_text⦆"
+some_indented_text = "⦅some_text | indent_lines⦆"
+report_self_test_failure("Indent lines should add two spaces to lines") unless some_indented_text == "  ⦅some_text⦆"
+items = [⦅a_list | double_quote | add_comma | indent_continuation⦆]
+report_self_test_failure("Add comma isn't adding commas") unless items == ["item 1", "item 2"]
 ```
 
 ### Testing: Inclusion
@@ -710,6 +967,169 @@ report_self_test_failure("ruby escape doesn't escape backslash") unless string_w
 report_self_test_failure("included replacements should replace blocks") unless included_string == "I came from lmt_include.lmd"
 ```
 
+### Testing: Extensions
+
+###### Code Block: Test Extensions
+
+``` ruby
+⦅from_extension⦆
+report_self_test_failure("extension hook should be able to add blocks") unless from_extension
+```
+
+### Testing: Conditional Output
+
+In the description, the if statement was to be executed.
+
+###### Code Block: Test Conditional Output
+
+``` ruby
+⦅conditional_output⦆
+report_self_test_failure("conditional output elseif should not be output when elseif is false") unless conditional_output_elsif
+report_self_test_failure("conditional output else should not be output when if true") unless conditional_output_else
+```
+
+#### If and elsif
+
+Neither the elsif or elsif statement are output when if is true.
+
+###### Code Block: Test Conditional Output
+
+``` ruby
+⦅conditional_output_if_and_elsif⦆
+report_self_test_failure("conditional output elsif should not be output even if true when is also true") unless conditional_output_elsif
+report_self_test_failure("conditional output else should not be output when if is true (if and elseif)") unless conditional_output_else
+```
+
+###### Execute Extension Block
+
+``` ruby
+@a_variable = true
+@another_variable = true
+```
+
+! if @a_variable
+
+###### Code Block: Conditional Output If And Elsif
+
+``` ruby
+conditional_output_else = true
+conditional_output_elsif = true
+```
+
+! elsif @another_variable
+
+###### Code Block: Conditional Output If And Elsif
+
+``` ruby
+conditional_output_elsif = false
+```
+
+! else
+
+###### Code Block: Conditional Output If And Elsif
+
+``` ruby
+conditional_output_else = false
+```
+
+! end
+
+#### Elsif
+
+When the if is false but the elsif true, only the elsif is output.
+
+###### Code Block: Test Conditional Output
+
+``` ruby
+conditional_output_if = true
+⦅conditional_output_elsif⦆
+report_self_test_failure("conditional output if should not be output when false") unless conditional_output_if
+report_self_test_failure("conditional output elseif should be output when elseif is true") unless conditional_output_elsif
+report_self_test_failure("conditional output else should not be output when elseif is true") unless conditional_output_else
+```
+
+###### Execute Extension Block
+
+``` ruby
+@a_variable = false
+@another_variable = true
+```
+
+! if @a_variable
+
+###### Code Block: Conditional Output Elsif
+
+``` ruby
+conditional_output_if = false
+```
+
+! elsif @another_variable
+
+###### Code Block: Conditional Output Elsif
+
+``` ruby
+conditional_output_else = true
+conditional_output_elsif = true
+```
+
+! else
+
+###### Code Block: Conditional Output Elsif
+
+``` ruby
+conditional_output_else = false
+```
+
+! end
+
+#### Else
+
+The else is output when none of the if or elsif statements are true.
+
+###### Code Block: Test Conditional Output
+
+``` ruby
+conditional_output_if = true
+conditional_output_elsif = true
+⦅conditional_output_else⦆
+report_self_test_failure("conditional output if should not be output when false") unless conditional_output_if
+report_self_test_failure("conditional output elseif should not be output when elseif is false") unless conditional_output_elsif
+report_self_test_failure("conditional output else should be output when neither if nor elseif is true") unless conditional_output_else
+```
+
+###### Execute Extension Block
+
+``` ruby
+@a_variable = false
+@another_variable = false
+```
+
+! if @a_variable
+
+###### Code Block: Conditional Output Else
+
+``` ruby
+conditional_output_if = false
+```
+
+! elsif @another_variable
+
+###### Code Block: Conditional Output Else
+
+``` ruby
+conditional_output_elsif = false
+```
+
+! else
+
+###### Code Block: Conditional Output Else
+
+``` ruby
+conditional_output_else = true
+```
+
+! end
+
 ### Regressions
 
 Some regressions / edge cases that we need to watch for.  These should not break our tangle operation.