markdown_exec 3.0.8 → 3.1.0

data/examples/imports/template_mustache.md

@@ -0,0 +1,22 @@
+# Organism Profile (Template Delimiter Mode)
+
+**Common Name:** {{COMMON_NAME}}  
+**Species:** {{SPECIES}}  
+**Genus:** {{GENUS}}  
+**Family:** {{FAMILY}}  
+**Order:** {{ORDER}}  
+**Class:** {{CLASS}}  
+**Year Discovered:** {{YEAR_DISCOVERED}}
+
+```bash
+echo "Organism {{COMMON_NAME}} belongs to {{SPECIES}} and was discovered in {{YEAR_DISCOVERED}}"
+export ORGANISM_NAME="{{COMMON_NAME}}"
+export SCIENTIFIC_NAME="{{SPECIES}}"
+export TAXONOMIC_GENUS="{{GENUS}}"
+export TAXONOMIC_FAMILY="{{FAMILY}}"
+export TAXONOMIC_ORDER="{{ORDER}}"
+export TAXONOMIC_CLASS="{{CLASS}}"
+export DISCOVERY_YEAR="{{YEAR_DISCOVERED}}"
+```
+
+Biological organism template using **{{}} delimiters** - requires `use_template_delimiters: true` configuration. 

data/examples/imports/template_vars.md

@@ -0,0 +1,22 @@
+# Organism Profile (Raw Replacement)
+
+**Common Name:** COMMON_NAME  
+**Species:** SPECIES  
+**Genus:** GENUS  
+**Family:** FAMILY  
+**Order:** ORDER  
+**Class:** CLASS  
+**Year Discovered:** YEAR_DISCOVERED
+
+```bash
+echo "Organism COMMON_NAME belongs to SPECIES and was discovered in YEAR_DISCOVERED"
+export ORGANISM_NAME="COMMON_NAME"
+export SCIENTIFIC_NAME="SPECIES"
+export TAXONOMIC_GENUS="GENUS"
+export TAXONOMIC_FAMILY="FAMILY"
+export TAXONOMIC_ORDER="ORDER"
+export TAXONOMIC_CLASS="CLASS"
+export DISCOVERY_YEAR="YEAR_DISCOVERED"
+```
+
+Biological organism template using **raw replacement** - placeholders are just the taxonomic key names. 

data/examples/raw_replacement_demo.md

@@ -0,0 +1,42 @@
+# Raw Replacement Demo
+
+This demonstrates the primary **raw replacement** mode of the enhanced `@import` functionality using biological entity data.
+
+## Tapanuli Orangutan Classification
+
+@import imports/organism_template.md COMMON_NAME="Tapanuli Orangutan" SPECIES="Pongo tapanuliensis" GENUS=Pongo FAMILY=Hominidae ORDER=Primates CLASS=Mammalia YEAR_DISCOVERED=2017
+
+## Psychedelic Frogfish Classification
+
+@import imports/organism_template.md COMMON_NAME="Psychedelic Frogfish" SPECIES="Histiophryne psychedelica" GENUS=Histiophryne FAMILY=Antennariidae ORDER=Lophiiformes CLASS=Actinopterygii YEAR_DISCOVERED=2009
+
+## Ruby Seadragon Classification
+
+@import imports/organism_template.md COMMON_NAME="Ruby Seadragon" SPECIES="Phyllopteryx dewysea" GENUS=Phyllopteryx FAMILY=Syngnathidae ORDER=Syngnathiformes CLASS=Actinopterygii YEAR_DISCOVERED=2015
+
+## How It Works
+
+In raw replacement mode (the default), placeholders in template files are just the key names:
+
+- `COMMON_NAME` gets replaced with the actual common name
+- `SPECIES` gets replaced with the scientific species name  
+- `GENUS` gets replaced with the taxonomic genus
+- `FAMILY` gets replaced with the taxonomic family
+- etc.
+
+This is different from template delimiter modes that use `${SPECIES}` or `{{GENUS}}`.
+
+## Benefits of Raw Replacement
+
+1. **Simple syntax** - No need for special delimiters
+2. **Clean templates** - Templates are more readable
+3. **Direct replacement** - What you see is what gets replaced
+4. **Scientific data friendly** - Works well with taxonomic classifications
+
+## Word Boundary Protection
+
+Raw replacement uses word boundaries, so:
+
+- `SPECIES` in "Species: SPECIES" gets replaced ✓
+- `SPECIES` in "SUBSPECIES=unknown" does NOT get replaced ✓
+- Partial matches are avoided automatically 

data/examples/recent_discoveries_demo.md

@@ -0,0 +1,43 @@
+# Recent Biological Discoveries Demo
+
+This demonstrates the `@import` functionality with **real biological discoveries** from the last 30 years.
+
+## Millipede Discovery - California
+
+@import imports/organism_template.md COMMON_NAME="Illacme tobini Millipede" SPECIES="Illacme tobini" GENUS=Illacme FAMILY=Siphonorhinidae ORDER=Siphonophorida CLASS=Diplopoda YEAR_DISCOVERED=2016
+
+## Fascinating Frogfish - Indonesia
+
+@import imports/organism_template.md COMMON_NAME="Psychedelic Frogfish" SPECIES="Histiophryne psychedelica" GENUS=Histiophryne FAMILY=Antennariidae ORDER=Lophiiformes CLASS=Actinopterygii YEAR_DISCOVERED=2009
+
+## Endangered Primate - Sumatra
+
+@import imports/organism_template.md COMMON_NAME="Tapanuli Orangutan" SPECIES="Pongo tapanuliensis" GENUS=Pongo FAMILY=Hominidae ORDER=Primates CLASS=Mammalia YEAR_DISCOVERED=2017
+
+## Unique Snake Species
+
+@import imports/organism_template.md COMMON_NAME="Cappuccino Snake" SPECIES="Hydrodynastes bicinctus" GENUS=Hydrodynastes FAMILY=Colubridae ORDER=Squamata CLASS=Reptilia YEAR_DISCOVERED=2021
+
+## Plant Discovery - Japan
+
+@import imports/organism_template.md COMMON_NAME="Spiny Dandelion" SPECIES="Taraxacum japonicum" GENUS=Taraxacum FAMILY=Asteraceae ORDER=Asterales CLASS=Magnoliopsida YEAR_DISCOVERED=2022
+
+## Deep Sea Discovery
+
+@import imports/organism_template.md COMMON_NAME="Yeti Crab" SPECIES="Kiwa hirsuta" GENUS=Kiwa FAMILY=Kiwaidae ORDER=Decapoda CLASS=Malacostraca YEAR_DISCOVERED=2005
+
+## About These Discoveries
+
+These organisms represent the incredible biodiversity still being discovered:
+
+- **Tapanuli Orangutan** (2017) - The 8th known species of great ape
+- **Psychedelic Frogfish** (2009) - Named for its vibrant, psychedelic coloration
+- **Illacme tobini** (2016) - A millipede species with an extraordinary number of legs
+- **Yeti Crab** (2005) - Found in deep-sea hydrothermal vents
+- **Spiny Dandelion** (2022) - One of the most recent plant discoveries
+
+Each entry demonstrates how the template substitution system can handle:
+- Complex scientific names with multiple words
+- Taxonomic hierarchy data
+- Mixed data types (names, years, classifications)
+- Quoted values for multi-word common names 

data/examples/template_syntax_demo.md

@@ -0,0 +1,24 @@
+# Template Syntax Demo
+
+This file shows the primary **raw replacement** mode and optional template delimiter modes using biological entity data.
+
+## Primary Mode: Raw Replacement (Default)
+
+@import imports/template_vars.md COMMON_NAME="Mythical Monkey" SPECIES="Cercopithecus lomamiensis" GENUS=Cercopithecus FAMILY=Cercopithecidae ORDER=Primates CLASS=Mammalia YEAR_DISCOVERED=2012
+
+## Template Delimiter Mode (Optional)
+
+@import imports/template_mustache.md COMMON_NAME="Ecuadorian Glassfrog" SPECIES="Hyalinobatrachium yaku" GENUS=Hyalinobatrachium FAMILY=Centrolenidae ORDER=Anura CLASS=Amphibia YEAR_DISCOVERED=2017
+
+*Note: Template delimiter mode requires `use_template_delimiters: true` configuration*
+
+## Complex Example - Plant Species
+
+@import imports/mixed_template.md COMMON_NAME="Spiny Dandelion" SPECIES="Taraxacum japonicum" GENUS=Taraxacum FAMILY=Asteraceae ORDER=Asterales CLASS=Magnoliopsida YEAR_DISCOVERED=2022
+
+## How It Works
+
+- **Raw replacement** (default): Placeholders are just key names like `COMMON_NAME`, `SPECIES`, `GENUS`
+- **Template delimiters** (optional): Placeholders use `${SPECIES}` or `{{GENUS}}` syntax
+- Raw replacement is simpler and more direct
+- Template delimiters provide explicit boundaries when needed for taxonomic data 

data/lib/cached_nested_file_reader.rb

@@ -11,10 +11,12 @@ require_relative 'exceptions'
 require_relative 'find_files'
 
 ##
-# The CachedNestedFileReader class provides functionality to read file lines with the ability
-# to process '#import filename' directives. When such a directive is encountered in a file,
-# the corresponding 'filename' is read and its contents are inserted at that location.
-# This class caches read files to avoid re-reading the same file multiple times.
+# The CachedNestedFileReader class provides functionality to read file
+# lines with the ability to process '#import filename' directives. When
+# such a directive is encountered in a file, the corresponding 'filename'
+# is read and its contents are inserted at that location.
+# This class caches read files to avoid re-reading the same file multiple
+# times.
 # It allows clients to read lines with or without providing a block.
 #
 class CachedNestedFileReader
@@ -41,11 +43,14 @@ class CachedNestedFileReader
 
   # yield each line to the block
   # return the processed lines
-  def readlines(filename, depth = 0, context: '', import_paths: nil,
-                indention: '', &block)
-    if @file_cache.key?(filename)
-      @file_cache[filename].each(&block) if block
-      return @file_cache[filename]
+  def readlines(
+    filename, depth = 0, context: '', import_paths: nil,
+    indention: '', substitutions: {}, use_template_delimiters: false, &block
+  )
+    cache_key = build_cache_key(filename, substitutions)
+    if @file_cache.key?(cache_key)
+      @file_cache[cache_key].each(&block) if block
+      return @file_cache[cache_key]
     end
     raise Errno::ENOENT, filename unless filename
 
@@ -54,35 +59,135 @@ class CachedNestedFileReader
     File.readlines(filename, chomp: true).each.with_index do |line, ind|
       if Regexp.new(@import_pattern) =~ line
         name_strip = $~[:name].strip
+        params_string = $~[:params] || ''
         import_indention = indention + $~[:indention]
-        included_file_path = if name_strip =~ %r{^/}
-                               name_strip
-                             elsif import_paths
-                               find_files(name_strip,
-                                          import_paths + [directory_path])&.first
-                             else
-                               File.join(directory_path, name_strip)
-                             end
+
+        # Parse parameters for text substitution
+        import_substitutions = parse_import_params(params_string)
+        merged_substitutions = substitutions.merge(import_substitutions)
+
+        included_file_path =
+          if name_strip =~ %r{^/}
+            name_strip
+          elsif import_paths
+            find_files(name_strip,
+                       import_paths + [directory_path])&.first
+          else
+            File.join(directory_path, name_strip)
+          end
 
         raise Errno::ENOENT, name_strip unless included_file_path
 
-        processed_lines += readlines(included_file_path, depth + 1,
-                                     context: "#{filename}:#{ind + 1}",
-                                     import_paths: import_paths,
-                                     indention: import_indention,
-                                     &block)
+        imported_lines = readlines(
+          included_file_path, depth + 1,
+          context: "#{filename}:#{ind + 1}",
+          import_paths: import_paths,
+          indention: import_indention,
+          substitutions: merged_substitutions,
+          use_template_delimiters: use_template_delimiters,
+          &block
+        )
+
+        # Apply text substitutions to imported content
+        processed_imported_lines = apply_substitutions(
+          imported_lines,
+          import_substitutions, use_template_delimiters
+        )
+        processed_lines += processed_imported_lines
       else
-        nested_line = NestedLine.new(line, depth, indention, filename, ind)
+        # Apply substitutions to the current line
+        substituted_line = apply_line_substitutions(line, substitutions,
+                                                    use_template_delimiters)
+        nested_line = NestedLine.new(substituted_line, depth, indention,
+                                     filename, ind)
         processed_lines.push(nested_line)
         block&.call(nested_line)
       end
     end
 
-    @file_cache[filename] = processed_lines
-  rescue Errno::ENOENT => err_filename
-    warn_format('readlines', "#{err_filename} @@ #{context}",
+    @file_cache[cache_key] = processed_lines
+  rescue Errno::ENOENT => err
+    warn_format('readlines', "#{err} @@ #{context}",
                 { abort: true })
   end
+
+  private
+
+  # Parse key=value parameters from the import line
+  def parse_import_params(params_string)
+    return {} if params_string.nil? || params_string.strip.empty?
+
+    params = {}
+    # Match key=value pairs, handling quoted values
+    params_string.scan(
+      /([A-Za-z_]\w*)=(?:"([^"]*)"|'([^']*)'|(\S+))/
+    ) do |key, quoted_double, quoted_single, unquoted|
+      value = quoted_double || quoted_single || unquoted
+      params[key] = value
+    end
+    params
+  end
+
+  # Apply text substitutions to a collection of NestedLine objects
+  def apply_substitutions(
+    lines, substitutions, use_template_delimiters = false
+  )
+    return lines if substitutions.empty?
+
+    lines.map do |nested_line|
+      substituted_text = apply_line_substitutions(
+        nested_line.text,
+        substitutions, use_template_delimiters
+      )
+      NestedLine.new(substituted_text, nested_line.depth, nested_line.indention,
+                     nested_line.filename, nested_line.index)
+    end
+  end
+
+  # Apply text substitutions to a single line
+  def apply_line_substitutions(line, substitutions,
+                               use_template_delimiters = false)
+    return line if substitutions.empty?
+
+    substituted_line = line.dup
+    if use_template_delimiters
+      # Replace template-style placeholders: ${KEY} or {{KEY}}
+      substitutions.each do |key, value|
+        substituted_line = substituted_line.gsub(/\$\{#{Regexp.escape(key)}\}/,
+                                                 value)
+        substituted_line = substituted_line.gsub(
+          /\{\{#{Regexp.escape(key)}\}\}/, value
+        )
+      end
+    else
+      # use temporary placeholders to avoid double replacement
+      temp_placeholders = {}
+
+      # Replace each key with a unique temporary placeholder
+      substitutions.each_with_index do |(key, value), index|
+        temp_placeholder = "__MDE_TEMP_#{index}__"
+        # pattern = /\b#{Regexp.escape(key)}\b/
+        pattern = Regexp.new(Regexp.escape(key))
+        substituted_line = substituted_line.gsub(pattern, temp_placeholder)
+        temp_placeholders[temp_placeholder] = value
+      end
+
+      # Replace temporary placeholders with actual values
+      temp_placeholders.each do |placeholder, value|
+        substituted_line = substituted_line.gsub(placeholder, value)
+      end
+
+    end
+    substituted_line
+  end
+
+  # Build cache key that includes substitutions to avoid conflicts
+  def build_cache_key(filename, substitutions)
+    return filename if substitutions.empty?
+
+    substitution_hash = substitutions.sort.to_h.hash
+    "#{filename}##{substitution_hash}"
+  end
 end
 
 return if $PROGRAM_NAME != __FILE__
@@ -97,9 +202,11 @@ class CachedNestedFileReaderTest < Minitest::Test
     @file2.rewind
 
     @file1 = Tempfile.new('test1.txt')
-    @file1.write("Line1\nLine2\n #insert #{@file2.path}\nLine3")
+    @file1.write("Line1\nLine2\n @import #{@file2.path}\nLine3")
     @file1.rewind
-    @reader = CachedNestedFileReader.new(import_pattern: /^(?<indention> *)#insert (?'name'.+)$/)
+    @reader = CachedNestedFileReader.new(
+      import_pattern: /^(?<indention> *)@import +(?<name>\S+)(?<params>(?: +[A-Za-z_]\w*=(?:"[^"]*"|'[^']*'|\S+))*) *$/
+    )
   end
 
   def teardown
@@ -121,6 +228,45 @@ class CachedNestedFileReaderTest < Minitest::Test
                  result
   end
 
+  def test_readlines_with_imports_and_substitutions
+    file_with_substitution = Tempfile.new('test_substitution.txt')
+    file_with_substitution.write("Server: HOST\nPort: PORT")
+    file_with_substitution.rewind
+
+    file_importing = Tempfile.new('test_importing.txt')
+    file_importing.write("Config:\n @import #{file_with_substitution.path} HOST=localhost PORT=8080\nEnd")
+    file_importing.rewind
+
+    result = @reader.readlines(file_importing.path).map(&:to_s)
+    assert_equal ['Config:', ' Server: localhost', ' Port: 8080', 'End'],
+                 result
+
+    file_with_substitution.close
+    file_with_substitution.unlink
+    file_importing.close
+    file_importing.unlink
+  end
+
+  def test_readlines_with_template_delimiters
+    file_with_template = Tempfile.new('test_template.txt')
+    file_with_template.write("API_URL=${API_URL}\nVERSION={{VERSION}}")
+    file_with_template.rewind
+
+    file_importing = Tempfile.new('test_importing.txt')
+    file_importing.write("Config:\n @import #{file_with_template.path} API_URL=https://api.example.com VERSION=1.2.3\nEnd")
+    file_importing.rewind
+
+    result = @reader.readlines(file_importing.path,
+                               use_template_delimiters: true).map(&:to_s)
+    assert_equal ['Config:', ' API_URL=https://api.example.com', ' VERSION=1.2.3', 'End'],
+                 result
+
+    file_with_template.close
+    file_with_template.unlink
+    file_importing.close
+    file_importing.unlink
+  end
+
   def test_caching_functionality
     # First read
 

data/lib/command_result.rb

@@ -20,6 +20,7 @@ class CommandResult
     @attributes = {}
     @attributes[:exit_status] = 0
     @attributes[:stdout] = ''
+    @attributes[:warning] = ''
     attributes.each { |name, value| @attributes[name] = value }
   end
 
@@ -34,13 +35,13 @@ class CommandResult
 
   # def new_lines
   #   value = @attributes[:new_lines]
-  #   ww caller.deref[0..4], value    
+  #   ww caller.deref[0..4], value
   #   value
   # end
 
   # # trap assignment to new_lines
   # def new_lines=(value)
-  #   ww caller.deref[0..4], value    
+  #   ww caller.deref[0..4], value
   #   @attributes[:new_lines] = value
   # end
 

data/lib/constants.rb

@@ -72,6 +72,11 @@ end
 
 LoadFileLinkState = Struct.new(:load_file, :link_state)
 
+class LoadMode
+  APPEND = :append
+  REPLACE = :replace
+end
+
 class MenuOptions
   YES = 1
   NO = 2

data/lib/evaluate_shell_expressions.rb

@@ -45,7 +45,6 @@ def evaluate_shell_expressions(initial_code, expressions, shell: '/bin/bash',
   end
 
   result_hash
-
 rescue StandardError
   ww $@, $!, caller.deref
   ww initial_code, expressions

data/lib/exceptions.rb

@@ -5,11 +5,19 @@
 require_relative 'ansi_string'
 
 module Exceptions
-  def self.error_handler(name = '', opts = {}, backtrace: $@, format_string: "\nError: %{name} -- %{message}", color_symbol: :red, take_count: 16)
+  def self.error_handler(
+    name = '',
+    opts = {},
+    backtrace: $@,
+    color_symbol: :red,
+    format_string: "\nError: %{name} -- %{message}",
+    show_backtrace: false,
+    take_count: 16
+  )
     warn(error = AnsiString.new(format(format_string,
                                        { name: name,
                                          message: $! })).send(color_symbol))
-    if backtrace
+    if show_backtrace && backtrace
       warn(backtrace.select do |s|
              s.include? 'markdown_exec'
            end.reject { |s| s.include? 'vendor' }.take(take_count).map.with_index { |line, ind| " *   #{ind}: #{line}" })

data/lib/fcb.rb

@@ -23,6 +23,7 @@ def parse_yaml_of_ux_block(
     default: export['default'],
     echo: export['echo'],
     exec: export['exec'],
+    force: export['force'],
     init: export['init'],
     menu_format: export['format'] || export['menu_format'] || menu_format,
     name: name,
@@ -156,20 +157,25 @@ module MarkdownExec
       @attrs[:id] = id
 
       if @attrs[:type] == BlockType::UX
-        case data = YAML.load(@attrs[:body].join("\n"))
-        when Hash
-          export = parse_yaml_of_ux_block(
-            data,
-            menu_format: menu_format,
-            prompt: prompt
-          )
-
-          @attrs[:center] = table_center
-          oname = @attrs[:oname] = format(export.menu_format, export.to_h)
-          @attrs[:readonly] = export.readonly
-        else
-          # triggered by an empty or non-YAML block
-          return NullResult.new(message: 'Invalid YAML', data: data)
+        begin
+          case data = YAML.load(@attrs[:body].join("\n"))
+          when Hash
+            export = parse_yaml_of_ux_block(
+              data,
+              menu_format: menu_format,
+              prompt: prompt
+            )
+
+            @attrs[:center] = table_center
+            oname = @attrs[:oname] = format(export.menu_format, export.to_h)
+            @attrs[:readonly] = export.readonly
+          else
+            # triggered by an empty or non-YAML block
+            return NullResult.new(message: 'Invalid YAML', data: data)
+          end
+        rescue StandardError
+          wwe 'Error processing block for menu', 'body:', @attrs[:body],
+              'data', data, 'export', export
         end
       end