markdown_exec 3.5.1 → 3.6.0

data/docs/getting-started.md

@@ -0,0 +1,209 @@
+# Getting Started with MarkdownExec
+
+A quick guide to get you up and running with MarkdownExec.
+
+## Installation
+
+```bash
+gem install markdown_exec
+```
+
+## Quick Start
+
+### Interactive Mode (Recommended)
+
+The simplest way to use MarkdownExec is in interactive mode:
+
+```bash
+# Process README.md in current directory
+mde
+
+# Process a specific markdown file
+mde my-document.md
+
+# Select from markdown files in current folder
+mde .
+```
+
+### Command Line Mode
+
+Execute specific blocks directly without the interactive menu:
+
+```bash
+# Execute a specific block
+mde my-document.md my-block-name
+
+# Or use the flag
+mde my-document.md --block-name my-block-name
+```
+
+## Basic Concepts
+
+### Named Code Blocks
+
+Create executable code blocks with names:
+
+```bash :setup
+echo "Setting up environment"
+```
+
+The block name (`setup`) is extracted from the fenced code block start line.
+
+### Block Dependencies
+
+Blocks can require other blocks to run first:
+
+```bash :deploy +setup +test
+echo "Deploying after setup and test"
+```
+
+The `+setup` and `+test` indicate that those blocks must execute first.
+
+### Interactive Forms (UX Blocks)
+
+Create interactive forms for user input:
+
+```ux
+name: USER_NAME
+prompt: Enter your name
+init: Guest
+```
+
+### Cross-Document Navigation
+
+Navigate between documents while maintaining context:
+
+```link
+file: next-document.md
+vars:
+  current_user: ${USER_NAME}
+```
+
+## Common Usage Patterns
+
+### Quick Execution
+
+```bash
+# Execute a specific block directly
+mde README.md deploy
+
+# With approval (review script before execution)
+mde README.md deploy --user-must-approve 1
+```
+
+### Interactive Development
+
+```bash
+# Open document in interactive mode
+mde README.md
+
+# Search and open a document
+mde --open "setup"
+```
+
+### Discovery
+
+```bash
+# List available documents
+mde --list-docs
+
+# List blocks in a document
+mde README.md --list-blocks
+
+# Search for content
+mde --find "database"
+```
+
+### Script Management
+
+```bash
+# Save scripts and output
+mde README.md -b deploy --save-executed-script 1 --save-execution-output 1
+```
+
+## Configuration
+
+### Environment Variables
+
+Set options via environment variables:
+
+```bash
+export MDE_USER_MUST_APPROVE=1
+export MDE_SAVE_EXECUTED_SCRIPT=1
+mde README.md deploy
+```
+
+### Configuration Files
+
+Create a `.mde.yml` file in your project:
+
+```yaml
+# .mde.yml
+save_executed_script: true
+user_must_approve: true
+output_script: true
+```
+
+## Next Steps
+
+- [CLI Reference](cli-reference.md) - Complete command-line reference
+- [Execution Control](execution-control.md) - Control how scripts execute
+- [Block Execution Modes](block-execution-modes.md) - Configure execution modes
+- [Block Filtering](block-filtering.md) - Filter and select blocks
+- [UX Blocks](ux-blocks.md) - Interactive form system
+- [Import Options](import-options.md) - Template system with @import
+- [Tab Completion](tab-completion.md) - Set up tab completion
+
+## Example Workflow
+
+Here's a complete example demonstrating key features:
+
+### Step 1: User Input
+
+```ux :user-setup
+name: USER_NAME
+prompt: Enter your name
+init: Guest
+```
+
+```ux :environment
+name: ENVIRONMENT
+allow:
+  - development
+  - staging
+  - production
+prompt: Select environment
+```
+
+### Step 2: Automated Setup
+
+```bash :setup +user-setup +environment
+echo "Setting up for user: ${USER_NAME}"
+echo "Environment: ${ENVIRONMENT}"
+```
+
+### Step 3: Conditional Logic
+
+```bash :deploy +setup
+if [ "${ENVIRONMENT}" = "production" ]; then
+  echo "Deploying to production with extra safety checks"
+else
+  echo "Deploying to ${ENVIRONMENT}"
+fi
+```
+
+### Step 4: Cross-Document Navigation
+
+```link
+file: next-workflow.md
+vars:
+  user: ${USER_NAME}
+  env: ${ENVIRONMENT}
+```
+
+## Getting Help
+
+- Run `mde --help` for command-line help
+- See [CLI Reference](cli-reference.md) for detailed option documentation
+- Check the [README](../README.md) for feature overview
+

data/docs/import-options.md

@@ -0,0 +1,391 @@
+# Import Options
+
+This document describes the configuration options that control how `@import` directives work in markdown_exec, including file path resolution, parameter parsing, and text substitution.
+
+## Overview
+
+The `@import` directive allows you to include content from other markdown files with parameter-based text substitution. Import options control:
+
+- **File path resolution**: Where to search for imported files
+- **Directive parsing**: How `@import` lines are recognized and parsed
+- **Parameter processing**: How parameters are extracted and processed
+- **Text substitution**: How parameter values are substituted into imported content
+
+## Options Reference
+
+### `import_paths`
+
+**Environment Variable**: `MDE_IMPORT_PATHS`  
+**Type**: String (colon-separated paths)  
+**Default**: Empty (uses document directory)
+
+**Description**: Colon-separated list of directory paths to search when resolving imported file names. When an `@import` directive specifies a relative filename, markdown_exec searches these paths in order, then falls back to the current document's directory.
+
+**Usage**:
+```bash
+# Set via environment variable
+export MDE_IMPORT_PATHS="docs/templates:examples/shared:common"
+
+# Or in document opts block
+```opts :(document_opts)
+import_paths: "docs/templates:examples/shared:common"
+```
+
+**Path Resolution Order**:
+1. Absolute paths (starting with `/`) are used as-is
+2. If `import_paths` is set, search each path in order
+3. Finally, search in the current document's directory
+
+**Example**:
+```markdown
+@import template.md
+```
+
+With `import_paths="templates:shared"`, markdown_exec searches:
+1. `templates/template.md`
+2. `shared/template.md`
+3. `./template.md` (current document directory)
+
+---
+
+### `import_directive_line_pattern`
+
+**Environment Variable**: `MDE_IMPORT_PATTERN`  
+**Type**: Regular Expression (String)  
+**Default**: `^(?<indention> *)@import +(?<name>\S+)(?<params>(?: +\w+(?::(?:[ceqv]|[ceqv]{2})=|=)(?:"[^"]*"|'[^']*'|\S+))*) *\\?$`
+
+**Description**: Regular expression pattern that matches `@import` directive lines. This pattern captures:
+- `indention`: Leading whitespace (preserved for indentation)
+- `name`: The filename to import
+- `params`: Optional space-separated parameters with various value formats
+- Optional line continuation marker (`\` at end of line)
+
+**Pattern Components**:
+- `^(?<indention> *)` - Captures leading spaces for indentation preservation
+- `@import +` - Matches the literal `@import` followed by spaces
+- `(?<name>\S+)` - Captures the filename (non-whitespace)
+- `(?<params>...)` - Captures optional parameters (see parameter syntax below)
+- ` *\\?$` - Optional trailing spaces and line continuation marker
+
+**Parameter Syntax in Pattern**:
+- `KEY=value` - Simple assignment
+- `KEY:"quoted value"` - Double-quoted value
+- `KEY:'quoted value'` - Single-quoted value
+- `KEY:ceqv=value` - Parameter type indicators (see parameter symbols below)
+
+**Example Matches**:
+```markdown
+@import template.md
+@import template.md KEY=value
+@import template.md KEY="value with spaces" OTHER=123
+@import template.md KEY:q="quoted" VAR:v=existing_var \
+  CONTINUED:c="command"
+```
+
+**Note**: Modifying this pattern requires understanding regex syntax and may break existing imports. Only change if you need custom import directive syntax.
+
+---
+
+### `import_directive_parameter_scan`
+
+**Environment Variable**: `MDE_IMPORT_PATTERN_SCAN`  
+**Type**: Regular Expression (String)  
+**Default**: `(\w+)(:[ceqv]{1,2}=|=)(?:"([^"]*)"|'([^']*)'|(\S+))`
+
+**Description**: Regular expression pattern that extracts individual parameters from the `params` portion of an `@import` line. This pattern is applied to the captured `params` group from `import_directive_line_pattern`.
+
+**Pattern Components**:
+- `(\w+)` - Captures the parameter key (word characters)
+- `(:[ceqv]{1,2}=|=)` - Captures the operator (parameter type indicator or simple `=`)
+- `(?:"([^"]*)"|'([^']*)'|(\S+))` - Captures the value from:
+  - Double-quoted string: `"value"`
+  - Single-quoted string: `'value'`
+  - Unquoted value: `value`
+
+**Capture Groups**:
+1. Parameter key (e.g., `KEY`)
+2. Operator (e.g., `=`, `:c=`, `:q=`, `:v=`, `:e=`)
+3. Double-quoted value (if used)
+4. Single-quoted value (if used)
+5. Unquoted value (if used)
+
+**Example**:
+For the line: `@import file.md KEY:q="value" VAR:v=existing OTHER=123`
+
+The pattern extracts:
+- `KEY`, `:q=`, `"value"` (from group 3)
+- `VAR`, `:v=`, `existing` (from group 5)
+- `OTHER`, `=`, `123` (from group 5)
+
+**Note**: This pattern works in conjunction with `import_directive_line_pattern`. Only modify if you need custom parameter parsing.
+
+---
+
+### `import_parameter_variable_assignment`
+
+**Environment Variable**: `MDE_IMPORT_PARAMETER_VARIABLE_ASSIGNMENT`  
+**Type**: Format String  
+**Default**: `"%{key}=%{value}"`
+
+**Description**: Format string used to generate shell variable assignments when parameters require command substitution or expression evaluation. This format is used when a parameter uses `:c=` (command) or `:e=` (expression) operators that need to create shell variables.
+
+**Format Placeholders**:
+- `%{key}` - The parameter name
+- `%{value}` - The shell code/expression to assign
+
+**Example**:
+When processing `KEY:c="ls -la"`, if a new variable needs to be created, the format generates:
+```bash
+KEY=$(ls -la)
+```
+
+**Usage Context**:
+This format is used internally when `ParameterExpansion` determines that a parameter requires a new shell variable to be created (e.g., for command substitution results).
+
+**Note**: This is an internal format string. Only modify if you need custom variable assignment syntax in generated shell code.
+
+---
+
+### `import_symbol_command_substitution`
+
+**Environment Variable**: `MDE_IMPORT_SYMBOL_COMMAND_SUBSTITUTION`  
+**Type**: String  
+**Default**: `:c=`
+
+**Description**: Symbol/operator that indicates a parameter value should be treated as a shell command to execute. The output of the command is used as the substitution value.
+
+**Usage**:
+```markdown
+@import template.md TIMESTAMP:c="date '+%Y-%m-%d'"
+```
+
+This executes `date '+%Y-%m-%d'` and uses its output as the value for `TIMESTAMP` in the imported template.
+
+**Behavior**:
+- The command is executed in a shell context
+- The command output (stdout) is captured and used as the value
+- If the command creates a new variable, `import_parameter_variable_assignment` format is used
+- Command output is typically chomped (trailing newline removed)
+
+**Example**:
+```markdown
+@import report.md GENERATED:c="date '+%Y-%m-%d %H:%M:%S'" LATEST:c="ls -t *.md | head -n 1"
+```
+
+**Note**: This symbol is part of the parameter expansion system. See `lib/parameter_expansion.rb` for full details on command substitution behavior.
+
+---
+
+### `import_symbol_evaluated_expression`
+
+**Environment Variable**: `MDE_IMPORT_SYMBOL_EVALUATED_EXPRESSION`  
+**Type**: String  
+**Default**: `:e=`
+
+**Description**: Symbol/operator that indicates a parameter value should be treated as a shell expression to evaluate. The expression is evaluated with variable expansion, and the result is used as the substitution value.
+
+**Usage**:
+```markdown
+@import template.md FILENAME:e="report-${COMMON_NAME// /_}-$(date +%Y%m%d).md"
+```
+
+This evaluates the shell expression with variable expansion and uses the result for `FILENAME`.
+
+**Behavior**:
+- The expression is evaluated in a shell context with variable expansion
+- Shell parameter expansion syntax (`${VAR}`, `${VAR:-default}`) is supported
+- Command substitution (`$(command)`) within expressions is supported
+- The evaluated result is used as the literal substitution value
+
+**Example**:
+```markdown
+@import template.md \
+  DESCRIPTION:e="Classification report for ${COMMON_NAME} (${PLAIN_SPECIES})" \
+  YEAR:e="${DISCOVERY_YEAR:-Unknown}"
+```
+
+**Note**: This symbol is part of the parameter expansion system. See `lib/parameter_expansion.rb` for full details on expression evaluation behavior.
+
+---
+
+### `import_symbol_raw_literal`
+
+**Environment Variable**: `MDE_IMPORT_SYMBOL_RAW_LITERAL`  
+**Type**: String  
+**Default**: `=`
+
+**Description**: Symbol/operator that indicates a parameter value should be used as a raw literal string without any processing. This is the default behavior when no type indicator is specified.
+
+**Usage**:
+```markdown
+@import template.md SPECIES=Pongo FAMILY=Hominidae
+```
+
+or explicitly:
+```markdown
+@import template.md SPECIES:=Pongo FAMILY:=Hominidae
+```
+
+**Behavior**:
+- The value is used exactly as provided
+- No shell evaluation or variable expansion
+- No quoting is added automatically
+- This is the simplest and most common parameter type
+
+**Example**:
+```markdown
+@import organism.md \
+  COMMON_NAME=Tapanuli \
+  SPECIES="Pongo tapanuliensis" \
+  GENUS=Pongo \
+  YEAR_DISCOVERED=2017
+```
+
+**Note**: This is the default behavior. The `=` operator can be omitted in most cases, but explicit `:=` makes the intent clear.
+
+---
+
+### `import_symbol_force_quoted_literal`
+
+**Environment Variable**: `MDE_IMPORT_SYMBOL_FORCE_QUOTED_LITERAL`  
+**Type**: String  
+**Default**: `:q=`
+
+**Description**: Symbol/operator that indicates a parameter value should be treated as a literal string and automatically wrapped in double quotes during substitution. This ensures the value is treated as a single string even if it contains spaces or special characters.
+
+**Usage**:
+```markdown
+@import template.md COMMON_NAME:q="Tapanuli Orangutan" SPECIES:q="Pongo tapanuliensis"
+```
+
+**Behavior**:
+- The value is wrapped in double quotes: `"value"`
+- Useful for values with spaces or special characters
+- Prevents shell interpretation of the value
+- The quotes are added during substitution, not in the parameter itself
+
+**Example**:
+```markdown
+@import organism.md \
+  COMMON_NAME:q="Tapanuli Orangutan" \
+  SPECIES:q="Pongo tapanuliensis" \
+  DESCRIPTION:q="Endangered primate species"
+```
+
+**Comparison**:
+- `COMMON_NAME=Tapanuli Orangutan` - May be interpreted as two values
+- `COMMON_NAME:q="Tapanuli Orangutan"` - Guaranteed to be one quoted value
+
+**Note**: The value in the `@import` line can be quoted or unquoted; the `:q=` operator ensures the substituted value is always quoted.
+
+---
+
+### `import_symbol_variable_reference`
+
+**Environment Variable**: `MDE_IMPORT_SYMBOL_VARIABLE_REFERENCE`  
+**Type**: String  
+**Default**: `:v=`
+
+**Description**: Symbol/operator that indicates a parameter value should be treated as a reference to an existing variable from the importing document. The variable's value is looked up and used for substitution.
+
+**Usage**:
+```markdown
+@import template.md GENUS:v=orangutan_genus FAMILY:v=hominidae_family
+```
+
+This looks up the values of `orangutan_genus` and `hominidae_family` from the current document's variables and uses them for substitution.
+
+**Behavior**:
+- The parameter value is treated as a variable name
+- The variable is looked up in the current document's context
+- If the variable exists, its value is used
+- If the variable doesn't exist, the substitution may fail or use an empty value
+- Variable references use shell syntax: `${VAR_NAME}`
+
+**Example**:
+```markdown
+# In the importing document
+```vars
+orangutan_genus: Pongo
+discovery_year_2017: 2017
+```
+
+@import organism.md \
+  GENUS:v=orangutan_genus \
+  YEAR_DISCOVERED:v=discovery_year_2017
+```
+
+**Note**: Variable references are resolved from the importing document's variable context, not from shell environment variables.
+
+---
+
+## Parameter Type Summary
+
+The import system supports five parameter processing types:
+
+| Symbol | Operator | Type | Description |
+|--------|----------|------|-------------|
+| `=` | `=` | Raw Literal | Direct text replacement (default) |
+| `:q=` | `:q=` | Force-Quoted Literal | Literal value wrapped in quotes |
+| `:v=` | `:v=` | Variable Reference | Reference to existing variable |
+| `:e=` | `:e=` | Evaluated Expression | Shell expression with variable expansion |
+| `:c=` | `:c=` | Command Substitution | Execute command and use output |
+
+## Examples
+
+### Basic Import
+```markdown
+@import shared/header.md
+```
+
+### Import with Simple Parameters
+```markdown
+@import template.md TITLE="My Document" AUTHOR=John YEAR=2024
+```
+
+### Import with Parameter Types
+```markdown
+@import organism.md \
+  COMMON_NAME:q="Tapanuli Orangutan" \
+  SPECIES:q="Pongo tapanuliensis" \
+  GENUS:v=orangutan_genus \
+  YEAR_DISCOVERED:v=discovery_year \
+  TIMESTAMP:c="date '+%Y-%m-%d %H:%M:%S'" \
+  FILENAME:e="report-${COMMON_NAME// /_}.md"
+```
+
+### Import with Line Continuation
+```markdown
+@import template.md \
+  PARAM1=value1 \
+  PARAM2=value2 \
+  PARAM3=value3
+```
+
+## Related Options
+
+- `menu_include_imported_blocks` - Control whether imported blocks appear in menus
+- `menu_include_imported_notes` - Control whether imported notes appear in menus
+- `menu_import_level_match` - Filter imported blocks by import level
+
+## Implementation Details
+
+The import system is implemented in:
+- `lib/cached_nested_file_reader.rb` - Main import processing
+- `lib/parameter_expansion.rb` - Parameter expansion logic
+- `lib/find_files.rb` - File path resolution
+
+For more details on parameter expansion, see the comprehensive documentation in `lib/parameter_expansion.rb`.
+
+## Related Documentation
+
+- [Block Naming Patterns](block-naming-patterns.md) - How imported blocks are named
+- [Block Filtering](block-filtering.md) - Filtering imported blocks
+- [CLI Reference](cli-reference.md) - Command-line usage
+
+## Additional Resources
+
+- [Import Substitution README](../IMPORT_SUBSTITUTION_README.md)
+- [Examples: Import with Substitution](../examples/import_with_substitution_demo.md)
+- [Parameter Expansion Documentation](../lib/parameter_expansion.rb)
+

data/docs/shell-script-evaluation.md

@@ -0,0 +1,78 @@
+/ v2025-12-07
+/---
+
+# App configuration
+
+## CLI loading of app configuration
+
+A file is read into the app configuration using the `--config` command line option.
+
+## Per-document app configuration
+
+An OPTS block that is automatically loaded when the document is loaded, modifies the app configuration.
+
+## Activated block effect on app configuration
+
+An OPTS block that is activated by the user modifies the app configuration.
+
+/---
+
+# Context code
+
+The context code is empty when the app starts.
+
+## CLI loading of context code
+
+A file is loaded as the context code using the `--load-code` command line option.
+
+## Automatic blocks
+
+Some blocks are automatically processed when the document is loaded.
+
+A VARS block that is automatically loaded when the document is loaded, is appended to the context code.  Display variables set per the app configuration.
+
+A UX block that is automatically loaded, is appended to the context code.  Display variables set per the app configuration.
+
+A shell block that is automatically loaded is appended to the context code.
+
+## Activated VARS block
+
+The variables named, converted to shell code to set to the corresponding values, are appended to the context code.
+The app's environment is also updated with the specified variables and values.
+
+## Activated UX block
+
+Variables named, converted to shell code to set to the corresponding values, are appended to the context code.
+The app's environment is also updated with the specified variables and values.
+
+## Activated shell block
+
+The activated and required shell blocks marked as context are appended to the context code.
+
+/---
+
+# Transient code
+
+The transient code is empty between block activations.
+
+## Activated shell block
+
+The required blocks, the transient code for this block, are executed in a shell script and its output is displayed and logged.
+
+/---
+
+# Shell script composition
+
+The shell script is composed of the context code followed by the transient code. The context code will include variables or context code in the required blocks.
+
+## Script output
+
+The output of the executed script (STDOUT and STDERR) is displayed in the console and logged to a file.
+
+/---
+
+## Shell block activation
+
+The activated and required shell blocks, not marked as context by default, are are the transient code.
+The script is composed and executed.
+Display shell output per the app configuration.

data/docs/tab-completion.md

@@ -1,5 +1,7 @@
 # Tab Completion
 
+This document describes how to install and use tab completion for MarkdownExec.
+
 ## Install tab completion
 
 Append a command to load the completion script to your shell configuration file. `mde` must be executable for the command to be composed correctly.
@@ -31,3 +33,8 @@ In the table below, tab is indicated by `!`
 | `mde --user-must-approve !` | `mde --user-must-approve .BOOL.`|
 | `mde --user-must-approve .BOOL.!` | `mde --user-must-approve 1` |
 
+## Related Documentation
+
+- [CLI Reference](cli-reference.md) - Complete command-line reference
+- [Getting Started](getting-started.md) - Quick start guide
+