shell_parser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 89579eb0817c4ec11516c0de8bea58b1dc5cf573bb1a144e61eeb0b249d24a26
+  data.tar.gz: a59972f41df5a30a64ff6cda53a0b9f1fc106f9c91eeac18b9b52b6f593fa9b6
+SHA512:
+  metadata.gz: 2bee7e6ba80006760f78b5de41cc2d289b8f4438ea03aa04e5f6156e9dcf12080e22c319d95a4d832df6684e73f6b41f8bd8ea4fdde2a85006763f97ceda8b48
+  data.tar.gz: 1d4bc6898faffe579f56710563f935c46d259120375b5933aef58fffa849a90976305e586ab03805d4eccde506acaae4be627da54814274db48eee87b48d9fd5

data/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2025-10-19
+
+### Added
+- Initial release
+- POSIX shell command language parser
+- Structured AST with Word, Command, Pipeline, and List nodes
+- Word parts: Literal, Variable, CommandSub with quote_style metadata
+- Support for pipes, redirections, and command lists (&&, ||, ;, &)
+- Proper handling of single/double quotes and expansions
+- Position tracking for syntax highlighting
+- Comprehensive test suite using minitest
+- Examples demonstrating syntax highlighting and execution use cases
+
+[Unreleased]: https://github.com/vidarh/shell-parser/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/vidarh/shell-parser/releases/tag/v0.1.0

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Vidar Hokstad
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,330 @@
+# Shell Parser
+
+A compact Ruby parser for  POSIX  Shell  Command  Language  syntax  that 
+produces a  simple  AST  suitable  for  syntax  highlighting  and  shell 
+execution.
+
+## Goal
+
+Remain compact and minimalist in design, while offering a reasonably
+complete parsing.
+
+## Features
+
+- **Tokenization** - Breaks shell commands into tokens with position tracking
+- **Simple AST** - Clean, easy-to-traverse abstract syntax tree
+- **POSIX Compliance** - Based on POSIX Shell Command Language specification
+- **Quoting Support** - Handles single quotes, double quotes, and backslash escaping
+- **Expansions** - Parses variable expansions (`$VAR`, `${VAR}`) and command substitution (`$(...)`, `` `...` ``)
+- **Command Structures** - Pipelines, lists (&&, ||, ;, &), and redirections
+- **Compact** - ~320 lines of clean, readable Ruby code
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'shell_parser'
+```
+
+And then execute:
+
+```sh
+bundle install
+```
+
+Or install it yourself as:
+
+```sh
+gem install shell_parser
+```
+
+## AST Node Types
+
+### Word
+A word is composed of one or more parts:
+```ruby
+Word = Struct.new(:parts, :pos, :len)
+# parts: array of Literal, Variable, or CommandSub
+# pos: character position in input
+# len: total length
+```
+
+### Word Parts
+
+**Literal** - Plain text (possibly quoted):
+```ruby
+Literal = Struct.new(:value, :pos, :len, :quote_style)
+# value: the text content
+# quote_style: :none, :single, or :double
+```
+
+**Variable** - Variable expansion:
+```ruby
+Variable = Struct.new(:name, :pos, :len, :braced, :quote_style)
+# name: variable name (e.g., "HOME" for $HOME)
+# braced: true if ${VAR} form, false if $VAR
+# quote_style: :none or :double (variables don't expand in single quotes)
+```
+
+**CommandSub** - Command substitution:
+```ruby
+CommandSub = Struct.new(:command, :pos, :len, :style, :quote_style)
+# command: the command text to execute
+# style: :dollar for $(cmd) or :backtick for `cmd`
+# quote_style: :none or :double
+```
+
+### Command
+A simple command with arguments and redirections:
+```ruby
+Command = Struct.new(:words, :redirects)
+# words: array of Word nodes
+# redirects: array of Redirect nodes
+```
+
+### Pipeline
+Commands connected by pipes (`|`):
+```ruby
+Pipeline = Struct.new(:commands, :negated)
+# commands: array of Command nodes
+# negated: boolean (for future ! pipeline support)
+```
+
+### List
+Commands connected by control operators:
+```ruby
+List = Struct.new(:left, :op, :right)
+# left/right: Command, Pipeline, or List
+# op: :and (&&), :or (||), :semi (;), :background (&)
+```
+
+### Redirect
+I/O redirection:
+```ruby
+Redirect = Struct.new(:type, :fd, :target)
+# type: :in (<), :out (>), :append (>>), :heredoc (<<), etc.
+# fd: file descriptor number (optional)
+# target: Word node
+```
+
+## Usage
+
+### Basic Parsing
+
+```ruby
+require_relative 'shell_parser'
+
+# Parse a command
+ast = ShellParser.parse("ls -la /tmp")
+# => Command with 3 words
+
+# Parse a pipeline
+ast = ShellParser.parse("cat file.txt | grep error | wc -l")
+# => Pipeline with 3 commands
+
+# Parse command lists
+ast = ShellParser.parse("make && make test || echo failed")
+# => List with nested lists
+```
+
+### Syntax Highlighting
+
+The parser provides detailed structure perfect for syntax highlighting:
+
+```ruby
+ast = ShellParser.parse("echo $HOME > output.txt")
+
+ast.words.each do |word|
+  word.parts.each do |part|
+    case part
+    when ShellParser::Literal
+      case part.quote_style
+      when :single then highlight_single_quoted(part.value, part.pos, part.len)
+      when :double then highlight_double_quoted(part.value, part.pos, part.len)
+      else highlight_literal(part.value, part.pos, part.len)
+      end
+    when ShellParser::Variable
+      highlight_variable(part.name, part.pos, part.len, part.braced)
+    when ShellParser::CommandSub
+      highlight_command_sub(part.command, part.pos, part.len, part.style)
+    end
+  end
+end
+
+ast.redirects.each do |redir|
+  highlight_redirection(redir.type, redir.target)
+end
+```
+
+The structured representation makes it easy to apply context-aware highlighting:
+```ruby
+# "Hello $USER" is represented as:
+word.parts #=> [
+  Literal("Hello ", quote_style: :double),
+  Variable("USER", quote_style: :double)
+]
+```
+
+### Shell Execution
+
+The AST makes it easy to traverse and execute commands:
+
+```ruby
+ast = ShellParser.parse("echo $HOME > output.txt")
+
+# Expand words by processing their parts
+def expand_word(word)
+  word.parts.map do |part|
+    case part
+    when ShellParser::Literal
+      part.value  # Use as-is
+    when ShellParser::Variable
+      ENV[part.name] || ""  # Look up variable
+    when ShellParser::CommandSub
+      `#{part.command}`.chomp  # Execute command
+    end
+  end.join
+end
+
+# Execute based on AST structure
+case ast
+when ShellParser::Command
+  args = ast.words.map { |w| expand_word(w) }
+  execute_command(args, ast.redirects)
+
+when ShellParser::Pipeline
+  setup_pipe do
+    ast.commands.each do |cmd|
+      args = cmd.words.map { |w| expand_word(w) }
+      execute_in_pipeline(args, cmd.redirects)
+    end
+  end
+
+when ShellParser::List
+  result = execute(ast.left)
+  case ast.op
+  when :and then execute(ast.right) if result == 0
+  when :or then execute(ast.right) if result != 0
+  when :semi then execute(ast.right)
+  when :background then fork { execute(ast.right) }
+  end
+end
+```
+
+The quote_style field tells you how to handle word splitting and glob expansion:
+```ruby
+part.quote_style == :none    # Apply glob expansion and word splitting
+part.quote_style == :single  # Use literal value, no expansion
+part.quote_style == :double  # Expand variables/commands, but no glob/split
+```
+
+## Supported Syntax
+
+### Simple Commands
+```sh
+ls -la /tmp
+echo "hello world"
+```
+
+### Pipelines
+```sh
+cat file.txt | grep pattern | wc -l
+```
+
+### Command Lists
+```sh
+make && make test           # AND - execute if previous succeeds
+make || echo "failed"       # OR - execute if previous fails
+make ; make test            # Sequential - always execute both
+sleep 10 &                  # Background job
+```
+
+### Redirections
+```sh
+command < input.txt         # Input redirection
+command > output.txt        # Output redirection
+command >> output.txt       # Append
+command 2>> error.log       # Redirect stderr
+```
+
+### Quoting
+```sh
+echo 'single quotes preserve everything literally'
+echo "double quotes allow $VAR expansion"
+echo escaped\ space
+```
+
+### Expansions
+```sh
+echo $HOME                  # Variable expansion
+echo ${USER}                # Variable expansion (braced)
+echo $(date)                # Command substitution
+echo `whoami`               # Command substitution (backticks)
+```
+
+## Examples
+
+See `examples.rb` for complete working examples of:
+- Syntax highlighting with token positions
+- Execution plan generation from AST
+- Pretty-printing AST structures
+
+Run examples:
+```sh
+ruby examples.rb
+```
+
+## Design Goals
+
+1. **Simplicity** - Clean, understandable code without excessive abstraction
+2. **Compactness** - Core parser in ~320 lines
+3. **Practicality** - Focus on two main use cases:
+   - Syntax highlighting (needs tokens with positions)
+   - Shell execution (needs command structure)
+4. **POSIX Foundation** - Based on POSIX spec but simplified where practical
+
+## Limitations
+
+This is a simplified parser focused on the core syntax. Not currently supported:
+
+- Compound commands (`if`, `while`, `for`, `case`, `{...}`, `(...)`)
+- Function definitions
+- Arithmetic expansion `$((...))`
+- Parameter expansion modifiers `${var:-default}`
+- Here-documents (parsed but not fully implemented)
+- Pattern matching and globbing
+- Reserved words as special tokens
+
+These can be added incrementally as needed.
+
+## Architecture
+
+**Lexer** (`ShellParser::Lexer`)
+- Scans input character by character
+- Handles quoting, escaping, and special characters
+- Produces token stream with position information
+- Preserves metadata for syntax highlighting
+
+**Parser** (`ShellParser::Parser`)
+- Recursive descent parser
+- Consumes tokens to build AST
+- Handles operator precedence
+- Simple error reporting
+
+**AST** (Struct-based nodes)
+- Lightweight node types using Ruby Structs
+- Easy to pattern match and traverse
+- Minimal memory overhead
+
+## References
+
+- [POSIX Shell Command Language Specification](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html)
+
+## rsh Integration Roadmap
+
+[rsh](https://github.com/vidarh/rsh) is a Ruby shell that currently uses an 80-line tokenizer for command parsing. The integration path with shell_parser:
+
+1. **Add as dependency** — Add `gem 'shell_parser'` to rsh's Gemfile and `require 'shell_parser'` in the main entry point.
+2. **Replace tokenizer** — Replace `tokenize_command` / `parse_shell_command` in `command_parser.rb` with `ShellParser.parse`, gaining proper AST-driven parsing for pipelines, lists, redirects, and quoting.
+3. **AST-driven execution** — Use the structured AST (`Command`, `Pipeline`, `List`) for execution instead of passing raw command strings to `exec`, enabling proper variable expansion, pipeline setup, and redirection handling within the Ruby process.

data/examples/demo_simplified.rb

@@ -0,0 +1,89 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/shell_parser'
+
+def show_word(word)
+  puts "  Word (#{word.parts.length} parts):"
+  word.parts.each do |part|
+    case part
+    when ShellParser::Literal
+      quote_info = part.quote_style == :none ? "" : " [#{part.quote_style}]"
+      puts "    Literal#{quote_info}: #{part.value.inspect}"
+    when ShellParser::Variable
+      quote_info = part.quote_style == :none ? "" : " [#{part.quote_style}]"
+      braced_info = part.braced ? "${...}" : "$..."
+      puts "    Variable#{quote_info}: #{part.name} (#{braced_info})"
+    when ShellParser::CommandSub
+      quote_info = part.quote_style == :none ? "" : " [#{part.quote_style}]"
+      style_info = part.style == :dollar ? "$()" : "`...`"
+      puts "    CommandSub#{quote_info}: #{part.command.inspect} (#{style_info})"
+    end
+  end
+end
+
+def demo(description, input)
+  puts "=" * 70
+  puts description
+  puts "=" * 70
+  puts "Input: #{input.inspect}\n\n"
+
+  ast = ShellParser.parse(input)
+
+  case ast
+  when ShellParser::Command
+    puts "Command:"
+    ast.words.each { |w| show_word(w) }
+  when ShellParser::Pipeline
+    puts "Pipeline:"
+    ast.commands.each_with_index do |cmd, i|
+      puts "  Command #{i + 1}:"
+      cmd.words.each { |w| show_word(w) }
+    end
+  end
+
+  puts
+end
+
+puts "\nSimplified AST Structure Demo"
+puts "=" * 70
+puts
+
+demo(
+  "Command substitution - properly extracted",
+  "echo $(date)"
+)
+
+demo(
+  "Variable expansion - name extracted, braced flag set",
+  "echo $HOME ${USER}"
+)
+
+demo(
+  "Double-quoted with expansion - parts have quote_style metadata",
+  'echo "Hello $USER, today is $(date)"'
+)
+
+demo(
+  "Single-quoted - no expansion, literal preserved",
+  "echo 'Literal $HOME text'"
+)
+
+demo(
+  "Composite word - multiple parts in one word",
+  'log_${DATE}_$(hostname).txt'
+)
+
+demo(
+  "Mixed quoting - each part tracks its own quoting context",
+  'echo "quoted"unquoted"quoted again"'
+)
+
+puts "=" * 70
+puts "Key improvements:"
+puts "  • No nested DoubleQuoted/SingleQuoted wrapper nodes"
+puts "  • Quoting tracked as metadata (quote_style field)"
+puts "  • Variables extracted with name and braced flag"
+puts "  • Command substitutions extracted with command text"
+puts "  • Clean, flat structure easy to traverse"
+puts "=" * 70

data/examples/demo_structure.rb

@@ -0,0 +1,116 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/shell_parser'
+
+def show_word_structure(word, depth = 0)
+  indent = "  " * depth
+
+  word.parts.each do |part|
+    case part
+    when ShellParser::Literal
+      puts "#{indent}Literal: #{part.value.inspect}"
+    when ShellParser::Variable
+      puts "#{indent}Variable: $#{part.braced ? '{' : ''}#{part.name}#{part.braced ? '}' : ''}"
+    when ShellParser::CommandSub
+      puts "#{indent}CommandSub (#{part.style}): #{part.command.inspect}"
+    when ShellParser::SingleQuoted
+      puts "#{indent}SingleQuoted: #{part.value.inspect}"
+    when ShellParser::DoubleQuoted
+      puts "#{indent}DoubleQuoted:"
+      part.parts.each do |inner_part|
+        case inner_part
+        when ShellParser::Literal
+          puts "#{indent}  Literal: #{inner_part.value.inspect}"
+        when ShellParser::Variable
+          puts "#{indent}  Variable: $#{inner_part.braced ? '{' : ''}#{inner_part.name}#{inner_part.braced ? '}' : ''}"
+        when ShellParser::CommandSub
+          puts "#{indent}  CommandSub (#{inner_part.style}): #{inner_part.command.inspect}"
+        end
+      end
+    end
+  end
+end
+
+def demo(description, input)
+  puts "=" * 70
+  puts description
+  puts "=" * 70
+  puts "Input: #{input.inspect}\n\n"
+
+  ast = ShellParser.parse(input)
+
+  case ast
+  when ShellParser::Command
+    puts "Command with #{ast.words.length} word(s):\n\n"
+    ast.words.each_with_index do |word, i|
+      puts "Word #{i + 1}:"
+      show_word_structure(word, 1)
+      puts
+    end
+  when ShellParser::Pipeline
+    puts "Pipeline with #{ast.commands.length} command(s):\n\n"
+    ast.commands.each_with_index do |cmd, i|
+      puts "Command #{i + 1}:"
+      cmd.words.each_with_index do |word, j|
+        puts "  Word #{j + 1}:"
+        show_word_structure(word, 2)
+      end
+      puts
+    end
+  end
+
+  puts
+end
+
+# Demonstrate structured parsing
+
+demo(
+  "Simple command substitution",
+  "echo $(date)"
+)
+
+demo(
+  "Variable expansion (braced and unbraced)",
+  "echo $HOME and ${USER}"
+)
+
+demo(
+  "Command substitution inside double quotes",
+  'echo "Today is $(date)"'
+)
+
+demo(
+  "Multiple expansions in one word",
+  'echo $USER@$HOSTNAME'
+)
+
+demo(
+  "Mixed quoting and expansions",
+  'echo "Hello $USER, your home is $HOME"'
+)
+
+demo(
+  "Nested command substitution",
+  'echo $(echo $(whoami))'
+)
+
+demo(
+  "Complex word with multiple parts",
+  'prefix_${VAR}_$(cmd)_suffix'
+)
+
+demo(
+  "Backtick style command substitution",
+  'echo `date`'
+)
+
+demo(
+  "Single quotes (no expansion)",
+  "echo 'Literal $HOME text'"
+)
+
+demo(
+  "Double quotes with escaped characters",
+  'echo "Quote: \" and dollar: \$"'
+)