parsanol 3.0.0

data/example/documentation/basic.rb.md

@@ -0,0 +1,97 @@
+# README Example - Ruby Implementation
+
+## How to Run
+
+```bash
+cd parsanol-ruby/example/documentation
+ruby basic.rb
+```
+
+## Code Walkthrough
+
+### String Parser Construction
+
+This example demonstrates the minimal parser from the README:
+
+```ruby
+parser =  str('"') >>
+          (
+            str('\\') >> any |
+            str('"').absent? >> any
+          ).repeat.as(:string) >>
+          str('"')
+```
+
+The parser matches quoted strings with escape sequences.
+
+### Escape Sequence Handling
+
+The pattern `str('\\') >> any` handles escaped characters:
+
+```ruby
+# Matches: \" \\ \n etc.
+str('\\') >> any
+```
+
+Any character following a backslash is accepted.
+
+### String Content Matching
+
+The pattern `str('"').absent? >> any` matches non-quote characters:
+
+```ruby
+# Matches any character except "
+str('"').absent? >> any
+```
+
+Combined with escape handling via alternation.
+
+### Named Capture
+
+`.as(:string)` labels the matched content:
+
+```ruby
+tree = parser.parse('"Hello"')
+# => {:string=>"Hello"}
+```
+
+The result is a hash with the captured content.
+
+### Transform Application
+
+Transforms extract and process captured content:
+
+```ruby
+transform = Parsanol::Transform.new do
+  rule(:string => simple(:x)) {
+    puts "String contents: #{x}"
+  }
+end
+transform.apply(tree)
+```
+
+Pattern matching on the tree structure.
+
+## Output Types
+
+```ruby
+# Parse result:
+{:string=>"This is a \"String\" in which you can escape stuff"}
+
+# Transform output:
+String contents: This is a "String" in which you can escape stuff
+```
+
+## Design Decisions
+
+### Why This Example?
+
+This is the canonical "hello world" for Parslet-style parsing. It demonstrates the core concepts in minimal form.
+
+### Why Transform?
+
+Transforms separate parsing from processing. The parser creates structure; transforms interpret it.
+
+### Ruby-Only Feature
+
+This example uses Parslet-compatible API for demonstration purposes.

data/example/email/basic.rb

@@ -0,0 +1,55 @@
+#!/usr/bin/env ruby
+
+# Email address parser with sanitization support.
+# Originally contributed to Parslet, ported to Parsanol as an example.
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+require 'parsanol/parslet'
+require 'parsanol/convenience'
+
+class EmailParser < Parsanol::Parser
+  rule(:space) { match('\s').repeat(1) }
+  rule(:space?) { space.maybe }
+  rule(:dash?) { match['_-'].maybe }
+
+  rule(:at) {
+    str('@') |
+    (dash? >> (str('at') | str('AT')) >> dash?)
+  }
+  rule(:dot) {
+    str('.') |
+    (dash? >> (str('dot') | str('DOT')) >> dash?)
+  }
+
+  rule(:word) { match('[a-z0-9]').repeat(1).as(:word) >> space? }
+  rule(:separator) { dot.as(:dot) >> space? | space }
+  rule(:words) { word >> (separator >> word).repeat }
+
+  rule(:email) {
+    (words.as(:username) >> space? >> at >> space? >> words).as(:email)
+  }
+
+  root(:email)
+end
+
+class EmailSanitizer < Parsanol::Transform
+  rule(:dot => simple(:dot), :word => simple(:word)) { ".#{word}" }
+  rule(:word => simple(:word)) { word }
+
+  rule(:username => sequence(:username)) { username.join + "@" }
+  rule(:username => simple(:username)) { username.to_s + "@" }
+
+  rule(:email => sequence(:email)) { email.join }
+end
+
+parser = EmailParser.new
+sanitizer = EmailSanitizer.new
+
+input = ARGV[0] || begin
+  default = "a.b.c.d@gmail.com"
+  STDERR.puts "usage: #{$0} \"EMAIL_ADDR\""
+  STDOUT.puts "since you haven't specified any EMAIL_ADDR, for testing purposes we're using #{default}"
+  default
+end
+
+p sanitizer.apply(parser.parse_with_debug(input))

data/example/email/basic.rb.md

@@ -0,0 +1,102 @@
+# Email Parser - Ruby Implementation
+
+## How to Run
+
+```bash
+cd parsanol-ruby/example/email
+ruby basic.rb "user@example.com"
+```
+
+## Code Walkthrough
+
+### Flexible At Symbol
+
+The parser handles obfuscated email formats:
+
+```ruby
+rule(:at) {
+  str('@') |
+  (dash? >> (str('at') | str('AT')) >> dash?)
+}
+```
+
+Supports both `@` and `at` (with optional dashes) for spam protection.
+
+### Flexible Dot
+
+Similar flexibility for the dot separator:
+
+```ruby
+rule(:dot) {
+  str('.') |
+  (dash? >> (str('dot') | str('DOT')) >> dash?)
+}
+```
+
+Handles `user@example.com` and `user at example dot com`.
+
+### Word and Words Rules
+
+Email parts are sequences of words:
+
+```ruby
+rule(:word) { match('[a-z0-9]').repeat(1).as(:word) >> space? }
+rule(:separator) { dot.as(:dot) >> space? | space }
+rule(:words) { word >> (separator >> word).repeat }
+```
+
+Words are alphanumeric; separators are dots or spaces.
+
+### Email Structure
+
+The complete email combines username and domain:
+
+```ruby
+rule(:email) {
+  (words.as(:username) >> space? >> at >> space? >> words).as(:email)
+}
+```
+
+Labels distinguish local part from domain.
+
+### Sanitizing Transform
+
+The transform normalizes obfuscated emails:
+
+```ruby
+class EmailSanitizer < Parsanol::Transform
+  rule(:dot => simple(:dot), :word => simple(:word)) { ".#{word}" }
+  rule(:word => simple(:word)) { word }
+
+  rule(:username => sequence(:username)) { username.join + "@" }
+  rule(:username => simple(:username)) { username.to_s + "@" }
+
+  rule(:email => sequence(:email)) { email.join }
+end
+```
+
+Converts "user at example dot com" to "user@example.com".
+
+## Output Types
+
+```ruby
+# Parse tree for "a.b.c.d@gmail.com":
+{:email=>{:username=>[{:word=>"a"}, {:dot=>".", :word=>"b"}, ...], ...}}
+
+# After transform:
+"a.b.c.d@gmail.com"
+```
+
+## Design Decisions
+
+### Why Handle Obfuscated Formats?
+
+Email addresses are often obfuscated to prevent spam harvesting. This parser can extract and normalize them.
+
+### Why Sequence vs Simple in Transform?
+
+`sequence(:x)` handles arrays of values; `simple(:x)` handles single values. Both cases occur depending on email complexity.
+
+### Why Add @ in Username Transform?
+
+The username rule ends with the @ separator, so the transform adds it back during reconstruction.

data/example/email/ruby_transform.rb

@@ -0,0 +1,106 @@
+# Email Parser Example - RubyTransform
+#
+# This example demonstrates parsing email addresses with validation.
+# Shows character classes, repetition, and structured output.
+#
+# Run with: ruby -Ilib example/email_ruby_transform.rb
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require 'parsanol'
+
+# Step 1: Define the email grammar
+class EmailParser < Parsanol::Parser
+  root :email
+
+  rule(:email) {
+    local_part.as(:local) >>
+    str('@') >>
+    domain.as(:domain)
+  }
+
+  rule(:local_part) {
+    (alphanumeric | match('[._%+-]')).repeat(1)
+  }
+
+  rule(:domain) {
+    label >> (str('.') >> label).repeat
+  }
+
+  rule(:label) {
+    alphanumeric.repeat(1)
+  }
+
+  rule(:alphanumeric) { match('[a-zA-Z0-9]') }
+end
+
+# Step 2: Email address class
+class EmailAddress
+  attr_reader :local, :domain
+
+  def initialize(local, domain)
+    @local = local.to_s
+    @domain = domain.to_s
+  end
+
+  def to_s
+    "#{@local}@#{@domain}"
+  end
+
+  def eql?(other)
+    other.is_a?(EmailAddress) && to_s == other.to_s
+  end
+
+  alias == eql?
+
+  def hash
+    to_s.hash
+  end
+end
+
+def parse_email(input)
+  parser = EmailParser.new
+  tree = parser.parse(input)
+
+  puts "Parse tree: #{tree.inspect}"
+
+  # Extract local and domain from tree
+  local = tree[:local].to_s
+  domain = tree[:domain].to_s
+
+  email = EmailAddress.new(local, domain)
+
+  email
+rescue Parsanol::ParseFailed => e
+  puts "Parse failed: #{e.message}"
+  nil
+end
+
+# Example usage
+if __FILE__ == $0
+  puts "=" * 60
+  puts "Email Parser - RubyTransform"
+  puts "=" * 60
+  puts
+
+  test_emails = [
+    "user@example.com",
+    "john.doe@example.org",
+    "test123@subdomain.example.co.uk",
+    "invalid-email",
+    "@missing-local.com",
+    "no-at-sign.com",
+  ]
+
+  test_emails.each do |email_str|
+    puts "-" * 40
+    puts "Input: #{email_str}"
+    email = parse_email(email_str)
+    if email
+      puts "  Email: #{email.to_s}"
+      puts "  Local: #{email.local}"
+      puts "  Domain: #{email.domain}"
+    end
+    puts
+  end
+end

data/example/empty/basic.rb

@@ -0,0 +1,13 @@
+# Basically just demonstrates that you can leave rules empty and get a nice
+# NotImplementedError. A way to quickly spec out your parser rules?
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require 'parsanol/parslet'
+
+class MyParser < Parsanol::Parser
+  rule(:empty) { }
+end
+
+
+MyParser.new.empty.parslet

data/example/empty/basic.rb.md

@@ -0,0 +1,73 @@
+# Empty Rule Handling - Ruby Implementation
+
+## How to Run
+
+```bash
+cd parsanol-ruby/example/empty
+ruby basic.rb
+```
+
+## Code Walkthrough
+
+### Empty Rule Definition
+
+Rules can be defined without bodies:
+
+```ruby
+class MyParser < Parsanol::Parser
+  rule(:empty) { }
+end
+```
+
+This creates a rule placeholder without implementation.
+
+### Error on Use
+
+Calling an empty rule raises `NotImplementedError`:
+
+```ruby
+MyParser.new.empty.parslet
+# => NotImplementedError: rule :empty not implemented
+```
+
+### Use Case: Grammar Sketching
+
+Empty rules help sketch out grammar structure:
+
+```ruby
+class MyParser < Parsanol::Parser
+  rule(:expression) { term >> (operator >> term).repeat }
+  rule(:term) { }        # TODO: implement
+  rule(:operator) { }    # TODO: implement
+end
+```
+
+This lets you plan the structure before filling in details.
+
+## Output Types
+
+```ruby
+# No parse output - raises error
+NotImplementedError: rule :empty not implemented
+```
+
+## Design Decisions
+
+### Why Allow Empty Rules?
+
+Empty rules support incremental grammar development. They provide a way to stub out parts of a grammar.
+
+### Why NotImplementedError?
+
+Using `NotImplementedError` clearly indicates the issue is a missing implementation, not a parse error.
+
+### Ruby-Only Feature
+
+This is a Parslet-specific convenience for Ruby development. In Rust, you would use option types or placeholder patterns.
+
+### When to Use
+
+- Prototyping grammars
+- Planning rule structure
+- Documenting intended grammar layout
+- Teaching parser concepts

data/example/empty/example.json

@@ -0,0 +1,38 @@
+{
+  "id": "empty",
+  "title": "Empty Rule Stub",
+  "description": "Demonstrate the empty rule pattern for grammar prototyping and incremental development.",
+  "category": "conceptual",
+  "tags": ["empty", "stub", "prototyping", "parslet"],
+  "difficulty": "beginner",
+  "concepts": ["empty rule", "prototyping", "incremental development", "stubs"],
+
+  "motivation": {
+    "why": "Empty rules allow prototyping grammars incrementally by stubbing out incomplete parts. This enables testing partial grammars before all rules are implemented.",
+    "useCases": [
+      "Incremental grammar development",
+      "Testing partial grammars",
+      "Prototyping complex languages"
+    ]
+  },
+
+  "inputFormat": {
+    "description": "Any input that matches the partial grammar.",
+    "examples": [
+      { "input": "test", "description": "Input to match against stubbed grammar", "valid": true }
+    ]
+  },
+
+  "outputFormat": {
+    "description": "Partial parse results from incomplete grammar.",
+    "structure": {
+      "partial": { "description": "Partial match from stubbed rules" }
+    }
+  },
+
+  "rubyOnly": true,
+  "parsletCompatible": true,
+  "implementations": {
+    "ruby": { "basic": "basic.rb" }
+  }
+}

data/example/erb/basic.rb

@@ -0,0 +1,47 @@
+# Example that demonstrates how a simple erb-like parser could be constructed. 
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require 'parsanol/parslet'
+
+class ErbParser < Parsanol::Parser
+  rule(:ruby) { (str('%>').absent? >> any).repeat.as(:ruby) }
+  
+  rule(:expression) { (str('=') >> ruby).as(:expression) }
+  rule(:comment) { (str('#') >> ruby).as(:comment) }
+  rule(:code) { ruby.as(:code) }
+  rule(:erb) { expression | comment | code }
+  
+  rule(:erb_with_tags) { str('<%') >> erb >> str('%>') }
+  rule(:text) { (str('<%').absent? >> any).repeat(1) }
+  
+  rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text) }
+  root(:text_with_ruby)
+end
+
+parser = ErbParser.new
+p parser.parse "The value of x is <%= x %>."
+p parser.parse "<% 1 + 2 %>"
+p parser.parse "<%# commented %>"
+ 
+
+evaluator = Parsanol::Transform.new do
+  
+  erb_binding = binding
+  
+  rule(:code => { :ruby => simple(:ruby) }) { eval(ruby, erb_binding); '' }  
+  rule(:expression => { :ruby => simple(:ruby) }) { eval(ruby, erb_binding) }
+  rule(:comment => { :ruby => simple(:ruby) }) { '' }
+  
+  rule(:text => simple(:text)) { text }
+  rule(:text => sequence(:texts)) { texts.join }
+  
+end
+
+puts evaluator.apply(parser.parse(<<-ERB
+The <% a = 2 %>not printed result of "a = 2".
+The <%# a = 1 %>not printed non-evaluated comment "a = 1", see the value of a below.
+The <%= 'nicely' %> printed result.
+The <% b = 3 %>value of a is <%= a %>, and b is <%= b %>.
+ERB
+))

data/example/erb/basic.rb.md

@@ -0,0 +1,103 @@
+# ERB Parser - Ruby Implementation
+
+## How to Run
+
+```bash
+cd parsanol-ruby/example/erb
+ruby basic.rb
+```
+
+## Code Walkthrough
+
+### ERB Tag Content Rule
+
+The ruby content inside ERB tags excludes the closing delimiter:
+
+```ruby
+rule(:ruby) { (str('%>').absent? >> any).repeat.as(:ruby) }
+```
+
+Negative lookahead prevents premature termination when parsing embedded Ruby code.
+
+### ERB Expression Types
+
+Three types of ERB tags are supported:
+
+```ruby
+rule(:expression) { (str('=') >> ruby).as(:expression) }
+rule(:comment) { (str('#') >> ruby).as(:comment) }
+rule(:code) { ruby.as(:code) }
+rule(:erb) { expression | comment | code }
+```
+
+- Expression (`<%= %>`) outputs values
+- Comment (`<%# %>`) is ignored
+- Code (`<% %>`) executes without output
+
+### Complete ERB Tag
+
+Tags combine opening delimiter, content, and closing delimiter:
+
+```ruby
+rule(:erb_with_tags) { str('<%') >> erb >> str('%>') }
+```
+
+The `erb` rule handles the type prefix (=, #, or nothing).
+
+### Text and Template Body
+
+Text content excludes ERB opening tags:
+
+```ruby
+rule(:text) { (str('<%').absent? >> any).repeat(1) }
+rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text) }
+```
+
+Alternation between text and ERB tags allows interleaving.
+
+### Transform for Evaluation
+
+The transform evaluates Ruby code:
+
+```ruby
+evaluator = Parsanol::Transform.new do
+  erb_binding = binding
+
+  rule(:code => { :ruby => simple(:ruby) }) { eval(ruby, erb_binding); '' }
+  rule(:expression => { :ruby => simple(:ruby) }) { eval(ruby, erb_binding) }
+  rule(:comment => { :ruby => simple(:ruby) }) { '' }
+
+  rule(:text => simple(:text)) { text }
+  rule(:text => sequence(:texts)) { texts.join }
+end
+```
+
+Code blocks execute silently; expressions return values; comments produce empty strings.
+
+## Output Types
+
+```ruby
+# Parse tree:
+{:text=>[
+  {:text=>"The value of x is "@s},
+  {:expression=>{:ruby=>" x "@s}},
+  {:text=>"."@s}
+]}
+
+# After transform (evaluated):
+"The value of x is 42."
+```
+
+## Design Decisions
+
+### Why Separate Expression and Code Tags?
+
+`<%= %>` outputs the result, `<% %>` executes for side effects. This distinction is fundamental to template evaluation.
+
+### Why Use Binding in Transform?
+
+A shared binding allows code in one tag to affect later expressions (e.g., setting a variable).
+
+### Why Comment as Separate Type?
+
+Comments should be completely ignored during evaluation, not parsed as Ruby code.

data/example/erb/optimized.rb

@@ -0,0 +1,42 @@
+# Please also look at the more naive 'erb.rb'. This shows how to optimize an
+# ERB like parser using parslet. 
+
+$:.unshift File.join(File.dirname(__FILE__), "/../lib")
+
+require 'parsanol/parslet'
+require './qed/applique/gobbleup'
+require 'parsanol/accelerator'
+
+class ErbParser < Parsanol::Parser
+  rule(:ruby) { (str('%>').absent? >> any).repeat.as(:ruby) }
+  
+  rule(:expression) { (str('=') >> ruby).as(:expression) }
+  rule(:comment) { (str('#') >> ruby).as(:comment) }
+  rule(:code) { ruby.as(:code) }
+  rule(:erb) { expression | comment | code }
+  
+  rule(:erb_with_tags) { str('<%') >> erb >> str('%>') }
+  rule(:text) { (str('<%').absent? >> any).repeat(1) }
+  
+  rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text) }
+  root(:text_with_ruby)
+end
+
+parser = ErbParser.new
+
+A = Parsanol::Accelerator
+optimized = A.apply(parser, 
+  A.rule((A.str(:x).absent? >> A.any).repeat(1)) { GobbleUp.new(x, 1) }, 
+  A.rule((A.str(:x).absent? >> A.any).repeat(0)) { GobbleUp.new(x, 0) })
+
+input = File.read(File.dirname(__FILE__) + "/big.erb")
+
+# Remove the comment marks here to see what difference the optimisation makes.
+# Commented out for the acceptance tests to run. 
+#
+# require 'benchmark'
+# Benchmark.bm(7) do |bm|
+#   bm.report('original') { parser.parse(input) }
+#   bm.report('gobble') { optimized.parse(input) }
+# end
+p optimized.parse(input)