markdown_exec 3.5.1 → 3.5.2

data/docs/block-scanning-patterns.md

@@ -0,0 +1,248 @@
+# Block Scanning Pattern Options
+
+This document describes the configuration options that control how markdown_exec scans block content for special patterns and directives.
+
+## Overview
+
+Block scanning patterns are regex patterns used to extract special information from fenced code blocks. These patterns identify:
+- Call names in block start lines
+- Required dependencies in block bodies
+- Stdin/stdout redirections
+- Default block types
+- Format strings for special block types
+
+## Configuration Options
+
+### block_calls_scan
+
+- **Option**: `block_calls_scan`
+- **Environment Variable**: `MDE_BLOCK_CALLS_SCAN`
+- **Default**: `"%\\([^\\)]+\\)"`
+- **Description**: Regex pattern to extract call names from block start lines (e.g., %call_name)
+
+This pattern is used to extract call names from the fenced code block start line. When a block has a call name like `%my_call`, this pattern matches it and extracts `my_call` for use in block execution and dependency tracking.
+
+**Example:**
+```markdown
+```bash :my_block %my_call
+echo "This block has a call name"
+```
+```
+
+The pattern `%\\([^\\)]+\\)` matches `%my_call` and extracts `my_call` as the call name.
+
+### block_required_scan
+
+- **Option**: `block_required_scan`
+- **Environment Variable**: `MDE_BLOCK_REQUIRED_SCAN`
+- **Default**: `"\\+\\S+"`
+- **Description**: Regex pattern to scan block body for required dependencies (e.g., +blockname)
+
+This pattern scans the block body content to identify required dependencies. When a block contains text like `+dependency_name`, it indicates that the block depends on another block named `dependency_name`.
+
+**Example:**
+```markdown
+```bash :my_block
+# This block requires another block
++setup_block
++config_block
+echo "Running after dependencies"
+```
+```
+
+The pattern `\\+\\S+` matches `+setup_block` and `+config_block`, indicating these blocks must be executed before `my_block`.
+
+### block_stdin_scan
+
+- **Option**: `block_stdin_scan`
+- **Environment Variable**: `MDE_BLOCK_STDIN_SCAN`
+- **Default**: `"<(?<full>(?<type>\\$)?(?<name>[A-Za-z_\\-\\.\\w]+))"`
+- **Description**: Regex pattern to scan block body for stdin redirection (e.g., <variable or <$variable)
+
+This pattern identifies stdin redirection directives in block content. It matches patterns like `<variable` or `<$variable` to indicate that the block should read from a variable or file.
+
+**Example:**
+```markdown
+```bash :my_block
+# Read from a variable
+<$input_data
+process_data
+```
+```
+
+The pattern matches `<input_data` or `<$input_data` and extracts the variable name for stdin redirection.
+
+### block_stdout_scan
+
+- **Option**: `block_stdout_scan`
+- **Environment Variable**: `MDE_BLOCK_STDOUT_SCAN`
+- **Default**: `">(?<full>(?<type>\\$)?(?<name>[A-Za-z_\\-\\.\\w]+))"`
+- **Description**: Match to place block body into a file or a variable
+
+This pattern identifies stdout redirection directives in block content. It matches patterns like `>variable` or `>$variable` to indicate where the block's output should be directed.
+
+**Example:**
+```markdown
+```bash :my_block
+# Write output to a variable
+generate_data >output_var
+```
+```
+
+The pattern matches `>output_var` or `>$output_var` and extracts the variable name for stdout redirection.
+
+### block_type_default
+
+- **Option**: `block_type_default`
+- **Environment Variable**: `MDE_BLOCK_TYPE_DEFAULT`
+- **Default**: `bash`
+- **Description**: Default shell type for blocks when no type is specified in the fenced code block
+
+When a fenced code block doesn't specify a type (e.g., just ` ``` ` instead of ` ```bash `), this option determines the default shell type to use for execution.
+
+**Example:**
+```markdown
+```
+echo "This uses the default shell type"
+```
+```
+
+If `block_type_default` is set to `bash`, this block will be executed as a bash script.
+
+**Valid values:**
+- `bash` - Bash shell (default)
+- `sh` - POSIX shell
+- `fish` - Fish shell
+
+### block_type_port_set_format
+
+- **Option**: `block_type_port_set_format`
+- **Environment Variable**: `MDE_BLOCK_TYPE_PORT_SET_FORMAT`
+- **Default**: `": ${%{key}:=%{value}}"`
+- **Description**: Format string for generating shell variable assignments from PORT block content
+
+This format string is used when processing PORT-type blocks to generate shell commands that set environment variables. The format uses placeholders:
+- `%{key}` - The variable name
+- `%{value}` - The variable value
+
+**Example:**
+```markdown
+```port
+PATH HOME USER
+```
+```
+
+With the default format `": ${%{key}:=%{value}}"`, this would generate:
+```bash
+: ${PATH:=/usr/bin:/bin}
+: ${HOME:=/home/user}
+: ${USER:=username}
+```
+
+The format uses shell parameter expansion syntax (`${VAR:=default}`) to set variables with default values if they're not already set.
+
+## How It Works
+
+### Scanning Process
+
+1. **Start Line Scanning**: When a fenced code block is parsed, `block_calls_scan` is applied to the start line to extract call names.
+
+2. **Body Content Scanning**: After parsing the block body, the following patterns are applied:
+   - `block_required_scan` - Scans for dependency markers
+   - `block_stdin_scan` - Scans for stdin redirection
+   - `block_stdout_scan` - Scans for stdout redirection
+
+3. **Type Determination**: If no type is specified in the fenced code block delimiter, `block_type_default` is used.
+
+4. **PORT Block Processing**: For PORT-type blocks, `block_type_port_set_format` is used to generate shell variable assignment commands.
+
+### Execution Order
+
+1. Block start line is parsed for call names
+2. Block body is scanned for required dependencies
+3. Block body is scanned for stdin/stdout redirections
+4. Block type is determined (from delimiter or default)
+5. For PORT blocks, format string is applied to generate commands
+
+## Usage Examples
+
+### Custom Call Pattern
+
+To use a different call pattern (e.g., `@call_name` instead of `%call_name`):
+
+```yaml
+block_calls_scan: "@\\([^\\)]+\\)"
+```
+
+### Custom Required Dependency Pattern
+
+To use a different dependency marker (e.g., `requires:blockname`):
+
+```yaml
+block_required_scan: "requires:(\\S+)"
+```
+
+### Custom Default Shell Type
+
+To use `sh` as the default instead of `bash`:
+
+```yaml
+block_type_default: sh
+```
+
+### Custom PORT Format
+
+To use a different format for PORT blocks (e.g., simple assignment):
+
+```yaml
+block_type_port_set_format: "export %{key}=%{value}"
+```
+
+This would generate:
+```bash
+export PATH=/usr/bin:/bin
+export HOME=/home/user
+export USER=username
+```
+
+## Related Configuration
+
+These options work together with:
+
+- **Block Naming Patterns** (`block_name_match`, `block_name_nick_match`, etc.) - Control how block names are extracted
+- **Block Execution Modes** (`block_batch_match`, `block_interactive_match`) - Control how blocks are executed
+- **Block Filtering** (`exclude_by_name_regex`, `select_by_name_regex`) - Control which blocks are included/excluded
+
+## Technical Details
+
+### Pattern Matching
+
+All scan patterns use Ruby regex syntax. Special characters must be escaped:
+- `\\(` for literal `(`
+- `\\)` for literal `)`
+- `\\+` for literal `+`
+- `\\$` for literal `$`
+
+### Named Capture Groups
+
+Some patterns use named capture groups:
+- `block_stdin_scan` captures `full`, `type`, and `name`
+- `block_stdout_scan` captures `full`, `type`, and `name`
+
+These capture groups allow the code to extract specific parts of the matched pattern.
+
+### Default Values
+
+The default patterns are designed to match common conventions:
+- `%call_name` for call names
+- `+dependency` for required dependencies
+- `<variable` for stdin redirection
+- `>variable` for stdout redirection
+
+## Related Documentation
+
+- [Block Naming Patterns](block-naming-patterns.md) - How block names are parsed and matched
+- [Block Execution Modes](block-execution-modes.md) - How blocks are executed
+- [Block Filtering](block-filtering.md) - How blocks are filtered and selected
+- [CLI Reference](cli-reference.md) - Command-line usage
+

data/docs/cli-reference.md

@@ -0,0 +1,370 @@
+# CLI Reference
+
+Quick reference for command-line usage of MarkdownExec. For detailed option documentation, see the specialized guides linked below.
+
+## Overview
+
+MarkdownExec provides both interactive and command-line modes for executing code blocks from markdown documents. Most options can be specified via:
+- Command-line flags (e.g., `--filename`, `-f`)
+- Environment variables (e.g., `MDE_FILENAME`)
+- Configuration files (`.mde.yml`)
+
+## Basic Usage
+
+```bash
+# Interactive mode - process README.md in current directory
+mde
+
+# Process a specific markdown file
+mde my-document.md
+
+# Execute a specific block directly (non-interactive)
+mde my-document.md my-block-name
+
+# Execute a specific block using option flag
+mde my-document.md --block-name my-block-name
+mde my-document.md -b my-block-name
+```
+
+## File and Document Selection
+
+### `filename` / `-f`
+
+- **Option**: `filename`
+- **Long Name**: `--filename`
+- **Short Name**: `-f`
+- **Environment Variable**: `MDE_FILENAME`
+- **Default**: None
+
+Specifies which markdown file to process. Can be a relative path to a markdown file.
+
+**Usage:**
+```bash
+mde --filename my-document.md
+mde -f my-document.md
+mde my-document.md  # Positional argument (position 0)
+```
+
+**Note**: If a filename is provided as a positional argument (first argument without a flag), it's automatically treated as the filename.
+
+### `path` / `-p`
+
+- **Option**: `path`
+- **Long Name**: `--path`
+- **Short Name**: `-p`
+- **Environment Variable**: `MDE_PATH`
+- **Default**: `"."`
+
+Specifies the directory path where markdown documents are located.
+
+**Usage:**
+```bash
+mde --path /path/to/documents
+mde -p ./docs
+mde /path/to/documents  # Positional argument if directory exists
+```
+
+### `block-name` / `-b`
+
+- **Option**: `block_name`
+- **Long Name**: `--block-name`
+- **Short Name**: `-b`
+- **Environment Variable**: `MDE_BLOCK_NAME`
+- **Default**: None
+
+Specifies which code block to execute directly, bypassing the interactive menu.
+
+**Usage:**
+```bash
+mde my-document.md --block-name my-block
+mde my-document.md -b my-block
+mde my-document.md my-block  # Positional argument (position 1)
+```
+
+**Note**: If a block name is provided as a positional argument (second argument), it's automatically treated as the block name. Use `.` as the block name to force menu display.
+
+### `hide-shebang`
+
+- **Option**: `hide_shebang`
+- **Long Name**: `--hide-shebang` / `--no-hide-shebang`
+- **Environment Variable**: `MDE_HIDE_SHEBANG`
+- **Default**: `true`
+
+Controls whether shebang lines (lines starting with `#!`) are hidden from document output. When enabled (default), shebang lines are extracted from input files during processing and not displayed as part of the document. This is useful when markdown files include shebang lines for direct execution (e.g., `#!/usr/bin/env mde`) but you don't want them to appear in the rendered document.
+
+**Usage:**
+```bash
+# Hide shebang lines (default behavior)
+mde my-document.md
+mde my-document.md --hide-shebang
+MDE_HIDE_SHEBANG=true mde my-document.md
+
+# Show shebang lines in output
+mde my-document.md --no-hide-shebang
+MDE_HIDE_SHEBANG=false mde my-document.md
+```
+
+**Note**: The option applies to both the main document and any imported files (via `@import` directives). Shebang lines are detected and filtered during the cached nested read process.
+
+## Discovery Commands
+
+### `list-blocks`
+
+- **Option**: `list_blocks`
+- **Long Name**: `--list-blocks`
+
+Displays all available code blocks in the specified document(s). Useful for discovering what blocks are available without entering interactive mode.
+
+**Usage:**
+```bash
+mde my-document.md --list-blocks
+```
+
+### `list-docs`
+
+- **Option**: `list_docs`
+- **Long Name**: `--list-docs`
+
+Lists all markdown documents found in the current directory (or specified path).
+
+**Usage:**
+```bash
+mde --list-docs
+mde --path /path/to/docs --list-docs
+```
+
+### `find` / `?`
+
+- **Option**: `find`
+- **Long Name**: `--find`
+- **Short Name**: `?`
+
+Searches for a keyword or pattern across markdown documents, block names, and file contents. Displays matching results and allows selection.
+
+**Usage:**
+```bash
+mde --find "search-term"
+mde ? "search-term"
+mde "search-term"  # Positional argument if file doesn't exist
+```
+
+### `open` / `o`
+
+- **Option**: `open`
+- **Long Name**: `--open`
+- **Short Name**: `o`
+
+Similar to `find`, but automatically presents a selection menu and opens the user's choice. Combines search and open in one step.
+
+**Usage:**
+```bash
+mde --open "search-term"
+mde -o "search-term"
+```
+
+## Save Options
+
+### `save-executed-script`
+
+- **Option**: `save_executed_script`
+- **Long Name**: `--save-executed-script`
+- **Environment Variable**: `MDE_SAVE_EXECUTED_SCRIPT`
+- **Default**: `false`
+
+When enabled, saves the generated script to a file before execution. Scripts are saved with timestamps and metadata for later review or replay.
+
+**Usage:**
+```bash
+mde my-document.md --save-executed-script 1
+```
+
+**Saved Script Location:**
+- Default folder: `logs/` (configurable via `saved_script_folder`)
+- Filename format: `mde_TIMESTAMP_DOCUMENT~BLOCK.sh` (configurable via `saved_asset_format`)
+
+### `save-execution-output`
+
+- **Option**: `save_execution_output`
+- **Long Name**: `--save-execution-output`
+- **Environment Variable**: `MDE_SAVE_EXECUTION_OUTPUT`
+- **Default**: `false`
+
+When enabled, saves the standard output (stdout) from script execution to a file. Useful for logging execution results.
+
+**Usage:**
+```bash
+mde my-document.md --save-execution-output 1
+```
+
+## Utility Commands
+
+### `help` / `-h`
+
+- **Option**: `help`
+- **Long Name**: `--help`
+- **Short Name**: `-h`
+
+Displays the help message with usage information and available options.
+
+**Usage:**
+```bash
+mde --help
+mde -h
+```
+
+### `version` / `-v`
+
+- **Option**: `version`
+- **Long Name**: `--version`
+- **Short Name**: `-v`
+
+Displays the current version of MarkdownExec.
+
+**Usage:**
+```bash
+mde --version
+mde -v
+```
+
+### `debug` / `-d`
+
+- **Option**: `debug`
+- **Long Name**: `--debug`
+- **Short Name**: `-d`
+- **Environment Variable**: `MDE_DEBUG`
+- **Default**: `false`
+
+Enables debug output, providing detailed information about internal operations, option parsing, and execution flow. Useful for troubleshooting.
+
+**Usage:**
+```bash
+mde my-document.md --debug 1
+mde my-document.md -d 1
+```
+
+### `config`
+
+- **Option**: `config`
+- **Long Name**: `--config`
+- **Default**: `"."`
+
+Specifies the path to a configuration file (`.mde.yml`). Configuration files allow you to set default options for a project or directory.
+
+**Usage:**
+```bash
+mde --config /path/to/.mde.yml
+mde --config .
+```
+
+**Note**: Configuration files are automatically loaded from the current directory (`.mde.yml`) if they exist. Use `--config` to specify a different location.
+
+### `exit` / `x`
+
+- **Option**: `exit`
+- **Long Name**: `--exit`
+- **Short Name**: `x`
+
+Exits the application immediately. Useful in scripts or when you want to exit without processing.
+
+**Usage:**
+```bash
+mde --exit
+mde -x
+```
+
+### `load-code` / `l`
+
+- **Option**: `load_code`
+- **Long Name**: `--load-code`
+- **Short Name**: `l`
+- **Environment Variable**: `MDE_LOAD_CODE`
+
+Loads code from an external file into the document context. The loaded code is treated as inherited lines that can be referenced by blocks.
+
+**Usage:**
+```bash
+mde my-document.md --load-code /path/to/code.sh
+mde my-document.md -l /path/to/code.sh
+```
+
+## Common Usage Patterns
+
+### Quick Execution
+
+```bash
+# Execute a specific block directly
+mde README.md deploy
+
+# With approval (see execution-control.md for details)
+mde README.md deploy --user-must-approve 1
+```
+
+### Interactive Development
+
+```bash
+# Open document in interactive mode
+mde README.md
+
+# Open with specific block pre-selected
+mde README.md --block-name .
+
+# Search and open
+mde --open "setup"
+```
+
+### Script Management
+
+```bash
+# Save scripts and output
+mde README.md -b deploy --save-executed-script 1 --save-execution-output 1
+```
+
+### Discovery and Exploration
+
+```bash
+# List available documents
+mde --list-docs
+
+# List blocks in a document
+mde README.md --list-blocks
+
+# Search for content
+mde --find "database"
+```
+
+## Configuration
+
+### Environment Variables
+
+All options can be set via environment variables using the `MDE_` prefix:
+
+```bash
+export MDE_FILENAME=README.md
+export MDE_SAVE_EXECUTED_SCRIPT=1
+export MDE_DEBUG=1
+
+mde -b deploy
+```
+
+### Configuration Files
+
+Create a `.mde.yml` file in your project directory to set default options:
+
+```yaml
+# .mde.yml
+save_executed_script: true
+user_must_approve: true
+output_script: true
+saved_script_folder: logs
+```
+
+## Related Documentation
+
+- [Execution Control](execution-control.md) - Detailed execution control options (approval, window execution, pausing, debouncing)
+- [Block Execution Modes](block-execution-modes.md) - Batch, interactive, and default execution modes
+- [Block Filtering](block-filtering.md) - Filtering and selecting blocks
+- [Block Naming Patterns](block-naming-patterns.md) - How block names are parsed and matched
+- [Block Scanning Patterns](block-scanning-patterns.md) - How block content is scanned for patterns
+- [Import Options](import-options.md) - @import directive configuration
+- [UX Blocks](ux-blocks.md) - Interactive form blocks
+

data/docs/dev/block-hide.md

@@ -9,7 +9,7 @@ name: name
 @import bats-document-configuration.md
 ```opts :(document_opts)
 # Pattern for blocks to hide from user-selection
-block_name_include_match:
+block_name_hidden_match:
   ^\(.*\)$
 
 # Pattern for the block name in the line defining the block

data/docs/dev/block-type-ux-transform.md

@@ -4,7 +4,7 @@
 exec: basename $(pwd)
 name: Var0
 ```
-$(echo -n "$Var0" | hexdump -v -e '16/1 " %02x"')
+$(print_bytes "$Var0")
 :::
 **With validate and transform, output has no newline.**
 ```ux :[document_ux_transform_1]
@@ -13,7 +13,7 @@ name: Var1
 transform: '%{name}'
 validate: (?<name>.+)
 ```
-$(echo -n "$Var1" | hexdump -v -e '16/1 " %02x"')
+$(print_bytes "$Var1")
 :::
 **With transform `:chomp`, output has no newline.**
 ```ux :[document_ux_transform_2]
@@ -21,7 +21,7 @@ exec: basename $(pwd)
 name: Var2
 transform: :chomp
 ```
-$(echo -n "$Var2" | hexdump -v -e '16/1 " %02x"')
+$(print_bytes "$Var2")
 :::
 **With transform `:upcase`, output is in upper case w/ newline.**
 ```ux :[document_ux_transform_3]
@@ -29,8 +29,9 @@ exec: basename $(pwd)
 name: Var3
 transform: :upcase
 ```
-$(echo -n "$Var3" | hexdump -v -e '16/1 " %02x"')
+$(print_bytes "$Var3")
 @import bats-document-configuration.md
+@import print_bytes.md
 ```opts :(document_opts)
 divider4_collapsible: false
 screen_width: 80

data/docs/dev/print_bytes.md

@@ -0,0 +1,3 @@
+```bash :(document_shell)
+print_bytes () { printf %s "$1" | hexdump -v -e '16/1 " %02x"' ; }
+```

data/docs/dev/shebang.md

@@ -0,0 +1,6 @@
+#!/usr/bin/env -S bin/bmde
+Demonstrate opening the document via Shebang.
+```bash :print-test
+echo Test
+```
+@import bats-document-configuration.md

data/docs/docker-testing.md

@@ -83,6 +83,11 @@ docker run --rm markdown-exec-test bundle exec rake minitest
 - **BATS Location**: `/markdown_exec/test/bats/bin/bats` (symlinked to `/usr/local/bin/bats`)
 - **Test Helpers**: Located in `/markdown_exec/test/test_helper/`
 
+## Related Documentation
+
+- [Option Documentation Process](option-documentation-process.md) - Process for documenting options
+- [Option Documentation Tasks](option-documentation-tasks.md) - Task list for documentation
+
 ## Troubleshooting
 
 ### Submodules not initialized