babel_bridge 0.5.1 → 0.5.3

data/Rakefile

@@ -1,6 +1,12 @@
 require "bundler/gem_tasks"
 require "rspec/core/rake_task"
 
-RSpec::Core::RakeTask.new(:spec)
-
 task :default => :spec
+
+desc "Run specs"
+RSpec::Core::RakeTask.new do |task|
+    task.pattern = "**/spec/*_spec.rb"
+    task.rspec_opts = Dir.glob("[0-9][0-9][0-9]_*").collect { |x| "-I#{x}" }.sort
+    task.rspec_opts << '--color'
+    task.rspec_opts << '-f documentation'
+end

data/TODO

@@ -0,0 +1,100 @@
+TODO: merge date entries into new TOPIC organized section
+
+By-Topic
+--------
+
+Parser Feedback (updated Jan 2013
+
+  If parsing failed and there was a negative (dont) match that prevented further parsing at the failure index, report it in a separate list: "Possibly could have continued if DIDN'T match:"
+
+  --
+
+  One confusing parser failure is when the greedy nature of PEG causes one rule to prevent another from matching.
+
+  Is it possible to a) detect this situation and b) to we provide it as a suggestion to fix (re-order rules) or should the parser automatically try alternaties when parsing fails? Likely the latter changes the Big-O behavior of the algorithm.
+
+Convert ignore_whitespace engine to "delimiters" (Dec 2012)
+
+  Replace engine under the hood for ignore_whitespace:
+
+  * Add global inter-token delimiter pattern with optional, per-rule overrides.
+  * ignore_whitespace will still be supported - just sets the global delimiter to /\s*/
+  * rewind_whitespace will be removed - override the containing rule's delimiter to ""
+
+  possible syntax:
+
+  rule :rule_name, pattern_a, pattern_b, :delimiter => "" # disable global delimiter; pattern_b must come immediately after pattern_a
+  rule :rule_name, pattern_a, pattern_b, :delimiter => "..." # must match "..." between pattern_a and pattern_b
+
+  Discussion:
+
+  ignore-whitespace should be considered a "token delimiter" much like the optional delimiter of the "many" node. We could then eliminate the need for "rewind_whitespace" if we allowed you to change the "token delimiter" for a given rule. Often when you need rewind_whitespace, you need it more than once in the same rule. It seems silly to first match the whitespace, and then unmatch it. Let's just allow you to change the delimiter to whatever you want - including the empty string - as well as have a global default (which ignore_whitespace sets to /\s*/).
+
+  The one question is should we provide some way to access what this "token delimiter" matches? Note this gets a little strange with "many" and it's delimiter since it will be matching: match_pattern, token_delimiter, match_delimiter, token_delimiter, match_pattern.
+
+"Python-like" Support (Dec 2012)
+
+  Add clean, easy support for indention based languages like python or coffeescript.
+
+Nodes use ruby ranges instead of "offset" and "length" (Nov 2012)
+
+  Would like to convert all Node member variables "offset" and "length" to "range" -- use Ruby ranges.
+
+Left-Recursion (Nov 2012)
+
+  http://en.wikipedia.org/wiki/Parsing_expression_grammar
+
+  Left-Recursion: Given what wikipedia says about left-recursion, I don't want to "support it" at the expense of losing linear-time parsing. I think the right answer is to "handle it nicely"
+
+  Detection:
+    * detect when we attempt to match a rule-variant that is already being matched at the same character position further up the stack.
+    * When we detect such a situation, immediately fail to match - as attempting to match leads to infinite recursion.
+
+  Idea 1: detect and throw error
+
+  Idea 2: This changes parser behavior in that it will not cause an error. Instead, the parser will proceed to attempt to match other variants.
+
+  Idea 3: An alternative behavior for left-recursion could be "greedy". The first time it happens, we proceed to any alternate rules. If there are none, then we fail. If we succeed, then attempt to match with just one recursive loop. If that succeeds, we advance to two recursive loops. This is kind-of like tail recursion optimization except it is more like head-recursion in this sense :).
+
+Arbitrary Nested Patterns (Dec 2010)
+
+  rule :my_rule, dont.match("hi", "there", /mom|dad/)
+
+  This just streamlines some rules into one-liners.
+
+Add "or(a,b)" Pattern Matcher (Dec 2010)
+
+  Including an or(a,b) pattern matcher.
+
+Pluralize Many-Rules (?) (Aug 2011)
+
+  When matching "many(:stick)", it would be nice to be able to refer to all the matches as "sticks" not "stick". Need "pluralize".
+
+  Counter: our namespace is already pretty overloaded. This may confuse things.
+
+Add Shared Methods for all Nodes for a specific Parser
+
+  I want the ability to add methods to the base-class for all Nodes on a per-parser basis.
+
+  This means that each parser needs to define a new node base-class, derived from BabelBridge::Node, from which all the Rule Nodes derive.
+
+Better handling of EmptyNode? (Dec 2010)
+
+  Right now an Optional node that is not matched returns an instance of EmptyNode. However, in the parsed result, ideally that match-slot would have the value "nil". How can we accomplish that simply?
+
+2010-11-*
+---------
+TODO-FEATURE: The "expecting" feature is so good I wonder if we should add the ability to make and apply suggestions to "repair the parse".
+  This would need:
+    a) default values for regex terminals
+      . string terminals are their own default values
+      . default values for regex should be verified to match the regex
+    b) an interactive prompter if there is more than one option
+
+TODO-IMPROVEMENT: "Expecting" should show line numbers instead of char numbers, but it should only be calculated on demand. This means we need a smarter formatter for our possible-error-logging.
+
+IDEA: could use the "-" prefix operator to mean "dont":
+  -"this"
+  -:that
+  -match(:foo)
+  -many(:foo)

data/babel_bridge.gemspec

@@ -1,4 +1,6 @@
-require File.join(File.dirname(__FILE__),"lib/babel_bridge.rb")
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'babel_bridge/version'
 
 $gemspec = Gem::Specification.new do |gem|
   gem.name = "babel_bridge"
@@ -15,9 +17,15 @@ Babel Bridge is an object oriented parser generator for parsing expression gramm
 Generate memoizing packrat parsers 100% in Ruby code with a simple embedded DSL.
 DESCRIPTION
 
-  gem.files = ["LICENSE", "README", "Rakefile", "babel_bridge.gemspec", "{test,spec,lib,doc,examples}/**/*"].map{|p| Dir[p]}.flatten
-  gem.has_rdoc = false
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
 
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'rspec'
+  gem.add_development_dependency 'simplecov'
+  gem.add_development_dependency 'guard-rspec'
+  gem.add_development_dependency 'guard-test'
+  gem.add_development_dependency 'rb-fsevent'
 end

data/examples/json/json_parser.rb

@@ -0,0 +1,23 @@
+# basic parser that accepts only legal JSON
+require "babel_bridge"
+
+class JsonParser < BabelBridge::Parser
+  ignore_whitespace
+
+  rule :document, any(:object, :array)
+
+  rule :array,  '[', many?(:value, ','), ']'
+  rule :object, '{', many?(:pair,  ','), '}'
+
+  rule :pair, :string, ':', :value
+
+  rule :value, any(:object, :array, :number, :string, :true, :false, :null)
+
+  rule :string, /"(?:[^"\\]|\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4}))*"/
+  rule :number, /-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?/
+  rule :true,   "true"
+  rule :false,  "false"
+  rule :null,   "null"
+end
+
+BabelBridge::Shell.new(JsonParser.new).start

data/examples/json/json_parser2.rb

@@ -0,0 +1,37 @@
+# basic parser that accepts only legal JSON
+# parse-tree-nodes support "#evaluate" which returns the ruby-equivalent data-structure
+require "babel_bridge"
+
+class JsonParser < BabelBridge::Parser
+  ignore_whitespace
+
+  rule :document, any(:object, :array)
+
+  rule :array, '[', many?(:value, ','), ']' do
+    def evaluate; value.collect {|v| v.evaluate} end
+  end
+
+  rule :object, '{', many?(:pair,  ','), '}' do
+    def evaluate; Hash[ pair.collect {|p| p.evaluate } ] end
+  end
+
+  rule :pair, :string, ':', :value do
+    def evaluate; [ eval(string.to_s), value.evaluate ] end
+  end
+
+  rule :value, any(:object, :array, :ruby_compatible_literal, :null)
+
+  rule :ruby_compatible_literal, any(:number, :string, :true, :false) do
+    def evaluate; eval(to_s); end
+  end
+
+  rule :string, /"(?:[^"\\]|\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4}))*"/
+  rule :number, /-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?/
+  rule :true,   "true"
+  rule :false,  "false"
+  rule :null,   "null" do
+    def evaluate; nil end
+  end
+end
+
+BabelBridge::Shell.new(JsonParser.new).start

data/lib/babel_bridge.rb

@@ -9,11 +9,12 @@ http://babel-bridge.rubyforge.org/
   string
   version
   nodes
+  pattern_element_hash
   pattern_element
   shell
   rule_variant
   rule
   parser
 }.each do |file|
-  require File.join(File.dirname(__FILE__),file)  
-end
+  require File.join(File.dirname(__FILE__),"babel_bridge",file)
+end

data/lib/{nodes → babel_bridge/nodes}/node.rb

@@ -88,7 +88,7 @@ class Node
   # info methods
   #********************
   alias :next :offset_after_match
-  def text; src[match_range] end  # the substring in src matched
+  def text; match_length == 0 ? "" : src[match_range] end  # the substring in src matched
 
   # length returns the number of sub-nodes
   def length

data/lib/{nodes → babel_bridge/nodes}/non_terminal_node.rb

@@ -36,13 +36,5 @@ class NonTerminalNode < Node
       update_match_length
     end
   end
-
-  def [](i)
-    matches[i]
-  end
-
-  def each(&block)
-    matches.each(&block)
-  end
 end
 end

data/lib/{parser.rb → babel_bridge/parser.rb}

@@ -145,14 +145,17 @@ class Parser
   #
   #*********************************************
   class <<self
-    def many(m,delimiter=nil) PatternElementHash.new.match.many(m).delimiter(delimiter) end
+    def many(m,delimiter=nil)  PatternElementHash.new.match.many(m).delimiter(delimiter) end
     def many?(m,delimiter=nil) PatternElementHash.new.optionally.match.many(m).delimiter(delimiter) end
-    def many!(m,delimiter=nil) PatternElementHash.new.dont.match.many(m).delimiter(delimiter) end
 
+    def match(*args)  PatternElementHash.new.match(*args) end
     def match?(*args) PatternElementHash.new.optionally.match(*args) end
-    def match(*args) PatternElementHash.new.match(*args) end
     def match!(*args) PatternElementHash.new.dont.match(*args) end
 
+    def any(*args) PatternElementHash.new.any(args) end
+    def any?(*args) PatternElementHash.new.optionally.any(args) end
+    def any!(*args) PatternElementHash.new.dont.any(args) end
+
     def dont; PatternElementHash.new.dont end
     def optionally; PatternElementHash.new.optionally end
     def could; PatternElementHash.new.could end
@@ -234,11 +237,9 @@ class Parser
   # If nil is returned, parsing failed. Call .parser_failure_info after failure for a human-readable description of the failure.
   # src: the string to parse
   # options:
-  #   offset: where to start in the string for parsing
   #   rule: lets you specify the root rule for matching
   #   partial_match: allow partial matching
   def parse(src, options={})
-    offset = options[:offset] || 0
     rule = options[:rule] || self.class.root_rule
     reset_parser_tracking
     @start_time = Time.now
@@ -266,14 +267,6 @@ class Parser
     @end_time-@start_time
   end
 
-  def parse_and_puts_errors(src,out=$stdout)
-    ret=parse(src)
-    unless ret
-      out.puts parser_failure_info
-    end
-    ret
-  end
-
   # options[:verbose] => false
   def node_list_string(node_list,common_root=[],options={})
     return unless node_list
@@ -323,7 +316,7 @@ Parse path at failure:
 Expecting#{expecting_list.length>1 ? ' one of' : ''}:
 #{Tools.uniform_tabs(Tools.indent(expecting_list.values.collect do |a|
     list=node_list_string(nodes_interesting_parse_path(a[:node]),common_root,options)
-    "#{a[:pattern].inspect}\t #{list}"
+    "#{a[:pattern]}\t #{list}"
   end.sort.join("\n"),"  "))}
 ENDTXT
   end

data/lib/{pattern_element.rb → babel_bridge/pattern_element.rb}

@@ -5,24 +5,6 @@ http://babel-bridge.rubyforge.org/
 =end
 
 module BabelBridge
-# hash which can be used declaratively
-class PatternElementHash
-  attr_accessor :hash
-
-  def initialize
-    @hash = {}
-  end
-
-  def [](key) @hash[key] end
-  def []=(key,value) @hash[key]=value end
-
-  def method_missing(method_name, *args)  #method_name is a symbol
-    return self if args.length==1 && !args[0] # if nil is provided, don't set anything
-    raise "More than one argument is not supported. #{self.class}##{method_name} args=#{args.inspect}" if args.length > 1
-    @hash[method_name] = args[0] || true # on the other hand, if no args are provided, assume true
-    self
-  end
-end
 
 # PatternElement provides optimized parsing for each Element of a pattern
 # PatternElement provides all the logic for parsing:
@@ -90,7 +72,7 @@ class PatternElement
 
     if !match && (terminal || negative)
       # log failures on Terminal patterns for debug output if overall parse fails
-      parent_node.parser.log_parsing_failure parent_node.next, :pattern => self.match, :node => parent_node
+      parent_node.parser.log_parsing_failure parent_node.next, :pattern => self.to_s, :node => parent_node
     end
 
     match.delimiter = delimiter if match
@@ -106,20 +88,15 @@ class PatternElement
     self.match = match
     match = match[0] if match.kind_of?(Array) && match.length == 1
     case match
-    when TrueClass then init_true
     when String then    init_string match
     when Regexp then    init_regex match
     when Symbol then    init_rule match
     when PatternElementHash then      init_hash match
+    when Array then     init_array match
     else                raise "invalid pattern type: #{match.inspect}"
     end
   end
 
-  # "true" parser always matches the empty string
-  def init_true
-    self.parser=lambda {|parent_node| EmptyNode.new(parent_node)}
-  end
-
   # initialize PatternElement as a parser that matches exactly the string specified
   def init_string(string)
     init_regex Regexp.escape(string)
@@ -155,6 +132,29 @@ class PatternElement
     end
   end
 
+  def init_any(patterns)
+    patterns = patterns.collect {|p| PatternElement.new(p,@init_options)}
+    self.parser = lambda do |parent_node|
+      patterns.each do |p|
+        node = p.parse(parent_node)
+        return node if node
+      end
+      nil
+    end
+  end
+
+  def init_array(patterns)
+    patterns = patterns.collect {|p| PatternElement.new(p,@init_options)}
+    self.parser = lambda do |parent_node|
+      parent_node.attempt_match do
+        patterns.each do |p|
+          return nil unless parent_node.match p
+        end
+      end
+      parent_node
+    end
+  end
+
   # initialize the PatternElement from hashed parameters
   def init_hash(hash)
     if hash[:parser]
@@ -163,6 +163,8 @@ class PatternElement
       init_many hash
     elsif hash[:match]
       init hash[:match]
+    elsif hash[:any]
+      init_any hash[:any]
     else
       raise "extended-options patterns (specified by a hash) must have either :parser=> or a :match=> set"
     end

data/lib/babel_bridge/pattern_element_hash.rb

@@ -0,0 +1,22 @@
+module BabelBridge
+# hash which can be used declaratively
+class PatternElementHash
+  attr_accessor :hash
+
+  def initialize
+    @hash = {}
+  end
+
+  def inspect; hash.inspect; end
+
+  def [](key) @hash[key] end
+  def []=(key,value) @hash[key]=value end
+
+  def method_missing(method_name, *args)  #method_name is a symbol
+    return self if args.length==1 && !args[0] # if nil is provided, don't set anything
+    raise "More than one argument is not supported. #{self.class}##{method_name} args=#{args.inspect}" if args.length > 1
+    @hash[method_name] = args[0] || true # on the other hand, if no args are provided, assume true
+    self
+  end
+end
+end

data/lib/{rule_variant.rb → babel_bridge/rule_variant.rb}

@@ -37,10 +37,6 @@ class RuleVariant
     @pattern_elements||=pattern.collect { |match| [PatternElement.new(match, :rule_variant => self, :pattern_element => true), delimiter_pattern] }.flatten[0..-2]
   end
 
-  def parse_element(element_parser, node)
-    node.add_match element_parser.parse(node), element_parser.name
-  end
-
   # returns a Node object if it matches, nil otherwise
   def parse(parent_node)
     #return parse_nongreedy_optional(src,offset,parent_node) # nongreedy optionals break standard PEG