citrus 2.2.2 → 2.3.0

data/doc/syntax.markdown

@@ -27,6 +27,9 @@ match in a case-insensitive manner.
 
     `abc`         # match "abc" in any case
 
+Besides case sensitivity, case-insensitive strings have the same behavior as
+double quoted strings.
+
 See [Terminal](api/classes/Citrus/Terminal.html) and
 [StringTerminal](api/classes/Citrus/StringTerminal.html) for more information.
 
@@ -69,6 +72,9 @@ that does not match a given expression.
     ~'a'          # match all characters until an "a"
     ~/xyz/        # match all characters until /xyz/ matches
 
+When using this operator (the tilde), at least one character must be consumed
+for the rule to succeed.
+
 See [AndPredicate](api/classes/Citrus/AndPredicate.html),
 [NotPredicate](api/classes/Citrus/NotPredicate.html), and
 [ButPredicate](api/classes/Citrus/ButPredicate.html) for more information.
@@ -98,25 +104,25 @@ levels of precedence is below.
 
 See [Choice](api/classes/Citrus/Choice.html) for more information.
 
+## Grouping
+
+As is common in many programming languages, parentheses may be used to override
+the normal binding order of operators. In the following example parentheses are
+used to make the vertical bar between `'b'` and `'c'` bind tighter than the
+space between `'a'` and `'b'`.
+
+    'a' ('b' | 'c')   # match "a", then "b" or "c"
+
 ## Labels
 
 Match objects may be referred to by a different name than the rule that
-originally generated them. Labels are created by placing the label and a colon
+originally generated them. Labels are added by placing the label and a colon
 immediately preceding any expression.
 
     chars:/[a-z]+/  # the characters matched by the regular expression
                     # may be referred to as "chars" in an extension
                     # method
 
-See [Label](api/classes/Citrus/Label.html) for more information.
-
-## Grouping
-
-As is common in many programming languages, parentheses may be used to override
-the normal binding order of operators.
-
-    'a' ('b' | 'c')   # match "a", then "b" or "c"
-
 ## Extensions
 
 Extensions may be specified using either "module" or "block" syntax. When using
@@ -128,17 +134,16 @@ in between less than and greater than symbols.
                               # times and extend the match with the
                               # CouponCode module
 
-Additionally, extensions may be specified inline using curly braces. Inside the
-curly braces you may embed method definitions that will be used to extend match
-objects.
+Additionally, extensions may be specified inline using curly braces. When using
+this method, the code inside the curly braces may be invoked by calling the
+`value` method on the match object.
 
-    # match any digit and return its integer value when calling the
-    # #value method on the match object
-    [0-9] {
-      def value
-        to_i
-      end
-    }
+    [0-9] { to_i }        # match any digit and return its integer value when
+                          # calling the #value method on the match object
+
+Note that when using the inline block method you may also specify arguments in
+between vertical bars immediately following the opening curly brace, just like
+in Ruby blocks.
 
 ## Super
 
@@ -146,6 +151,24 @@ When including a grammar inside another, all rules in the child that have the
 same name as a rule in the parent also have access to the `super` keyword to
 invoke the parent rule.
 
+    grammar Number
+      def number
+        [0-9]+
+      end
+    end
+    
+    grammar FloatingPoint
+      include Number
+      
+      rule number
+        super ('.' super)?
+      end
+    end
+
+In the example above, the `FloatingPoint` grammar includes `Number`. Both have a
+rule named `number`, so `FloatingPoint#number` has access to `Number#number` by
+means of using `super`.
+
 See [Super](api/classes/Citrus/Super.html) for more information.
 
 ## Precedence
@@ -155,21 +178,21 @@ their precedence. A higher precedence indicates tighter binding.
 
 Operator                  | Name                      | Precedence
 ------------------------- | ------------------------- | ----------
-`''`                      | String (single quoted)    | 6
-`""`                      | String (double quoted)    | 6
-<code>``</code>           | String (case insensitive) | 6
-`[]`                      | Character class           | 6
-`.`                       | Dot (any character)       | 6
-`//`                      | Regular expression        | 6
-`()`                      | Grouping                  | 6
-`*`                       | Repetition (arbitrary)    | 5
-`+`                       | Repetition (one or more)  | 5
-`?`                       | Repetition (zero or one)  | 5
-`&`                       | And predicate             | 4
-`!`                       | Not predicate             | 4
-`~`                       | But predicate             | 4
-`:`                       | Label                     | 4
-`<>`                      | Extension (module name)   | 3
-`{}`                      | Extension (literal)       | 3
+`''`                      | String (single quoted)    | 7
+`""`                      | String (double quoted)    | 7
+<code>``</code>           | String (case insensitive) | 7
+`[]`                      | Character class           | 7
+`.`                       | Dot (any character)       | 7
+`//`                      | Regular expression        | 7
+`()`                      | Grouping                  | 7
+`*`                       | Repetition (arbitrary)    | 6
+`+`                       | Repetition (one or more)  | 6
+`?`                       | Repetition (zero or one)  | 6
+`&`                       | And predicate             | 5
+`!`                       | Not predicate             | 5
+`~`                       | But predicate             | 5
+`<>`                      | Extension (module name)   | 4
+`{}`                      | Extension (literal)       | 4
+`:`                       | Label                     | 3
 `e1 e2`                   | Sequence                  | 2
 <code>e1 &#124; e2</code> | Ordered choice            | 1

data/examples/calc.citrus

@@ -52,7 +52,9 @@ grammar Calc
   end
 
   rule group
-    (lparen term rparen) { term.value }
+    (lparen term rparen) {
+      term.value
+    }
   end
 
   ## Lexical syntax
@@ -62,11 +64,15 @@ grammar Calc
   end
 
   rule float
-    (digits '.' digits space*) { strip.to_f }
+    (digits '.' digits space*) {
+      strip.to_f
+    }
   end
 
   rule integer
-    (digits space*) { strip.to_i }
+    (digits space*) {
+      strip.to_i
+    }
   end
 
   rule digits

data/examples/calc.rb

@@ -55,7 +55,9 @@ grammar :Calc do
   end
 
   rule :group do
-    all(:lparen, :term, :rparen) { term.value }
+    all(:lparen, :term, :rparen) {
+      term.value
+    }
   end
 
   ## Lexical syntax
@@ -65,11 +67,15 @@ grammar :Calc do
   end
 
   rule :float do
-    all(:digits, '.', :digits, zero_or_more(:space)) { strip.to_f }
+    all(:digits, '.', :digits, zero_or_more(:space)) {
+      strip.to_f
+    }
   end
 
   rule :integer do
-    all(:digits, zero_or_more(:space)) { strip.to_i }
+    all(:digits, zero_or_more(:space)) {
+      strip.to_i
+    }
   end
 
   rule :digits do

data/examples/ip.rb

@@ -1,3 +1,5 @@
+$LOAD_PATH.unshift(File.expand_path('../../lib', __FILE__))
+
 require 'citrus'
 
 # This file contains a small suite of tests for the grammars found in ip.citrus.

data/lib/citrus.rb

@@ -8,7 +8,7 @@ require 'strscan'
 module Citrus
   autoload :File, 'citrus/file'
 
-  VERSION = [2, 2, 2]
+  VERSION = [2, 3, 0]
 
   # Returns the current version of Citrus as a string.
   def self.version
@@ -20,37 +20,47 @@ module Citrus
 
   Infinity = 1.0 / 0
 
-  F = ::File
-
   CLOSE = -1
 
-  # Loads the grammar from the given +file+ into the global scope using #eval.
-  def self.load(file)
-    file << '.citrus' unless F.file?(file)
-    raise "Cannot find file #{file}" unless F.file?(file)
-    raise "Cannot read file #{file}" unless F.readable?(file)
-    eval(F.read(file))
+  # Parses the given Citrus +code+ using +options+.
+  def self.parse(code, options={})
+    File.parse(code, options)
   end
 
   # Evaluates the given Citrus parsing expression grammar +code+ in the global
-  # scope. Returns an array of any grammar modules that are created. Implicitly
-  # raises +SyntaxError+ on a failed parse.
+  # scope. Returns an array of any grammar modules that are created.
+  #
+  #     Citrus.eval(<<CITRUS)
+  #     grammar MyGrammar
+  #       rule abc
+  #         "abc"
+  #       end
+  #     end
+  #     CITRUS
+  #
   def self.eval(code)
-    parse(code, :consume => true).value
+    parse(code).value
   end
 
-  # Parses the given Citrus +code+ using the given +options+. Returns the
-  # generated match tree. Raises a +SyntaxError+ if the parse fails.
-  def self.parse(code, options={})
-    File.parse(code, options)
+  # Evaluates the given expression and creates a new Rule object from it.
+  #
+  #     Citrus.rule('"a" | "b"')
+  #
+  def self.rule(expr)
+    parse(expr, :root => :rule_body).value
+  end
+
+  # Loads the grammar from the given +file+ into the global scope using #eval.
+  def self.load(file)
+    file << '.citrus' unless ::File.file?(file)
+    raise "Cannot find file #{file}" unless ::File.file?(file)
+    raise "Cannot read file #{file}" unless ::File.readable?(file)
+    eval(::File.read(file))
   end
 
   # A standard error class that all Citrus errors extend.
   class Error < RuntimeError; end
 
-  # Raised when a match cannot be found.
-  class NoMatchError < Error; end
-
   # Raised when a parse fails.
   class ParseError < Error
     # The +input+ given here is an instance of Citrus::Input.
@@ -59,9 +69,7 @@ module Citrus
       @line_offset = input.line_offset(offset)
       @line_number = input.line_number(offset)
       @line = input.line(offset)
-      msg = "Failed to parse input on line %d at offset %d\n%s" %
-        [line_number, line_offset, detail]
-      super(msg)
+      super("Failed to parse input on line #{line_number} at offset #{line_offset}\n#{detail}")
     end
 
     # The 0-based offset at which the error occurred in the input, i.e. the
@@ -82,12 +90,12 @@ module Citrus
     # Returns a string that, when printed, gives a visual representation of
     # exactly where the error occurred on its line in the input.
     def detail
-      "%s\n%s^" % [line, ' ' * line_offset]
+      "#{line}\n#{' ' * line_offset}^"
     end
   end
 
-  # This class represents the core of the parsing algorithm. It wraps the input
-  # string and serves matches to all nonterminals.
+  # An Input is a scanner that is responsible for executing rules at different
+  # positions in the input string and persisting event streams.
   class Input < StringScanner
     def initialize(string)
       super(string)
@@ -97,40 +105,25 @@ module Citrus
     # The maximum offset in the input that was successfully parsed.
     attr_reader :max_offset
 
-    # A nested hash of rule id's to offsets and their respective matches. Only
-    # present if memoing is enabled.
-    attr_reader :cache
-
-    # The number of times the cache was hit. Only present if memoing is enabled.
-    attr_reader :cache_hits
-
-    # Resets all internal variables so that this object may be used in another
-    # parse.
     def reset # :nodoc:
       @max_offset = 0
       super
     end
 
-    # Returns the length of this input.
-    def length
-      string.length
-    end
-
     # Returns an array containing the lines of text in the input.
     def lines
-      string.send(string.respond_to?(:lines) ? :lines : :to_s).to_a
-    end
-
-    # Iterates over the lines of text in the input using the given +block+.
-    def each_line(&block)
-      string.each_line(&block)
+      if string.respond_to?(:lines)
+        string.lines.to_a
+      else
+        string.to_a
+      end
     end
 
     # Returns the 0-based offset of the given +pos+ in the input on the line
     # on which it is found. +pos+ defaults to the current pointer position.
     def line_offset(pos=pos)
       p = 0
-      each_line do |line|
+      string.each_line do |line|
         len = line.length
         return (pos - p) if p + len >= pos
         p += len
@@ -142,7 +135,7 @@ module Citrus
     # given +pos+. +pos+ defaults to the current pointer position.
     def line_index(pos=pos)
       p = n = 0
-      each_line do |line|
+      string.each_line do |line|
         p += line.length
         return n if p >= pos
         n += 1
@@ -156,7 +149,7 @@ module Citrus
       line_index(pos) + 1
     end
 
-    alias lineno line_number
+    alias_method :lineno, :line_number
 
     # Returns the text of the line that contains the character at the given
     # +pos+. +pos+ defaults to the current pointer position.
@@ -165,17 +158,16 @@ module Citrus
     end
 
     # Returns an array of events for the given +rule+ at the current pointer
-    # position. Objects in this array may be one of three types: a rule id,
-    # Citrus::CLOSE, or a length.
+    # position. Objects in this array may be one of three types: a Rule,
+    # Citrus::CLOSE, or a length (integer).
     def exec(rule, events=[])
       start = pos
       index = events.size
 
-      rule.exec(self, events)
-
-      if index < events.size
-        self.pos = start + events[-1]
+      if rule.exec(self, events).size > index
+        pos = start + events[-1]
         @max_offset = pos if pos > @max_offset
+        self.pos = pos
       else
         self.pos = start
       end
@@ -186,47 +178,59 @@ module Citrus
     # Returns the length of a match for the given +rule+ at the current pointer
     # position, +nil+ if none can be made.
     def test(rule)
-      rule.exec(self)[-1]
+      start = pos
+      events = rule.exec(self)
+      self.pos = start
+      events[-1]
     end
 
     # Returns +true+ when using memoization to cache match results.
     def memoized?
-      !! @cache
+      false
     end
+  end
 
-    # Modifies this object to cache match results during a parse. This technique
-    # (also known as "Packrat" parsing) guarantees parsers will operate in
-    # linear time but costs significantly more in terms of time and memory
-    # required to perform a parse. For more information, please read the paper
-    # on Packrat parsing at http://pdos.csail.mit.edu/~baford/packrat/icfp02/.
-    def memoize!
-      return if memoized?
-
+  # A MemoizingInput is an Input that caches segments of the event stream for
+  # particular rules in a parse. This technique (also known as "Packrat"
+  # parsing) guarantees parsers will operate in linear time but costs
+  # significantly more in terms of time and memory required to perform a parse.
+  # For more information, please read the paper on Packrat parsing at
+  # http://pdos.csail.mit.edu/~baford/packrat/icfp02/.
+  class MemoizingInput < Input
+    def initialize(string)
+      super(string)
       @cache = {}
       @cache_hits = 0
+    end
 
-      # Using +instance_eval+ here preserves access to +super+ within the
-      # methods we define inside the block.
-      instance_eval do
-        def exec(rule, events=[]) # :nodoc:
-          c = @cache[rule.id] ||= {}
+    # A nested hash of rules to offsets and their respective matches.
+    attr_reader :cache
 
-          e = if c[pos]
-            @cache_hits += 1
-            c[pos]
-          else
-            c[pos] = super(rule)
-          end
+    # The number of times the cache was hit.
+    attr_reader :cache_hits
 
-          events.concat(e)
-        end
+    def reset # :nodoc:
+      @cache.clear
+      @cache_hits = 0
+      super
+    end
 
-        def reset # :nodoc:
-          @cache.clear
-          @cache_hits = 0
-          super
-        end
+    def exec(rule, events=[]) # :nodoc:
+      c = @cache[rule] ||= {}
+
+      e = if c[pos]
+        @cache_hits += 1
+        c[pos]
+      else
+        c[pos] = super(rule)
       end
+
+      events.concat(e)
+    end
+
+    # Returns +true+ when using memoization to cache match results.
+    def memoized?
+      true
     end
   end
 
@@ -327,7 +331,7 @@ module Citrus
     end
 
     # Gets/sets the rule with the given +name+. If +obj+ is given the rule
-    # will be set to the value of +obj+ passed through Rule#new. If a block is
+    # will be set to the value of +obj+ passed through Rule.for. If a block is
     # given, its return value will be used for the value of +obj+.
     #
     # It is important to note that this method will also check any included
@@ -340,7 +344,7 @@ module Citrus
       if obj
         rule_names << sym unless has_rule?(sym)
 
-        rule = Rule.new(obj)
+        rule = Rule.for(obj)
         rule.name = name
         setup_super(rule, name)
         rule.grammar = self
@@ -350,7 +354,9 @@ module Citrus
 
       rules[sym] || super_rule(sym)
     rescue => e
-      raise 'Cannot create rule "%s": %s' % [name, e.message]
+      # This preserves the backtrace.
+      e.message.replace("Cannot create rule \"#{name}\": #{e.message}")
+      raise e
     end
 
     # Gets/sets the +name+ of the root rule of this grammar. If no root rule is
@@ -364,7 +370,7 @@ module Citrus
     # Creates a new rule that will match any single character. A block may be
     # provided to specify semantic behavior (via #ext).
     def dot(&block)
-      ext(Rule.new(DOT), block)
+      ext(Rule.for(DOT), block)
     end
 
     # Creates a new Super for the rule currently being defined in the grammar. A
@@ -387,18 +393,10 @@ module Citrus
 
     # Creates a new ButPredicate using the given +rule+. A block may be provided
     # to specify semantic behavior (via #ext).
-    def but(rule, &block)
+    def butp(rule, &block)
       ext(ButPredicate.new(rule), block)
     end
 
-    alias butp but # For consistency with #andp and #notp.
-
-    # Creates a new Label using the given +rule+ and +label+. A block may be
-    # provided to specify semantic behavior (via #ext).
-    def label(rule, label, &block)
-      ext(Label.new(rule, label), block)
-    end
-
     # Creates a new Repeat using the given +rule+. +min+ and +max+ specify the
     # minimum and maximum number of times the rule must match. A block may be
     # provided to specify semantic behavior (via #ext).
@@ -433,30 +431,37 @@ module Citrus
       ext(Choice.new(args), block)
     end
 
+    # Adds +label+ to the given +rule+.A block may be provided to specify
+    # semantic behavior (via #ext).
+    def label(rule, label, &block)
+      rule = ext(rule, block)
+      rule.label = label
+      rule
+    end
+
     # Specifies a Module that will be used to extend all matches created with
     # the given +rule+. A block may also be given that will be used to create
-    # an anonymous module. See Rule#ext=.
+    # an anonymous module. See Rule#extension=.
     def ext(rule, mod=nil, &block)
-      rule = Rule.new(rule)
+      rule = Rule.for(rule)
       mod = block if block
       rule.extension = mod if mod
       rule
     end
+
+    # Creates a new Module from the given +block+ and sets it to be the
+    # extension of the given +rule+. See Rule#extension=.
+    def mod(rule, &block)
+      rule.extension = Module.new(&block)
+      rule
+    end
   end
 
-  # A Rule is an object that is used by a grammar to create matches on the
+  # A Rule is an object that is used by a grammar to create matches on an
   # Input during parsing.
   module Rule
-    # Evaluates the given expression and creates a new rule object from it.
-    #
-    #     Citrus::Rule.eval('"a" | "b"')
-    #
-    def self.eval(expr)
-      Citrus.parse(expr, :root => :rule_body, :consume => true).value
-    end
-
     # Returns a new Rule object depending on the type of object given.
-    def self.new(obj)
+    def self.for(obj)
       case obj
       when Rule     then obj
       when Symbol   then Alias.new(obj)
@@ -466,33 +471,10 @@ module Citrus
       when Range    then Choice.new(obj.to_a)
       when Numeric  then StringTerminal.new(obj.to_s)
       else
-        raise ArgumentError, "Invalid rule object: %s" % obj.inspect
+        raise ArgumentError, "Invalid rule object: #{obj.inspect}"
       end
     end
 
-    @unique_id = 0
-
-    # A global registry for Rule objects. Keyed by rule id.
-    @rules = {}
-
-    # Adds the given +rule+ to the global registry and gives it an id.
-    def self.<<(rule) # :nodoc:
-      rule.id = (@unique_id += 1)
-      @rules[rule.id] = rule
-    end
-
-    # Returns the Rule object with the given +id+.
-    def self.[](id)
-      @rules[id]
-    end
-
-    def initialize(*args) # :nodoc:
-      Rule << self
-    end
-
-    # An integer id that is unique to this rule.
-    attr_accessor :id
-
     # The grammar this rule belongs to.
     attr_accessor :grammar
 
@@ -501,28 +483,25 @@ module Citrus
       @name = name.to_sym
     end
 
-    # Returns the name of this rule.
-    def name
-      @name || '<anonymous>'
-    end
+    # The name of this rule.
+    attr_reader :name
 
-    # Returns +true+ if this rule has a name, +false+ otherwise.
-    def named?
-      !! @name
+    # Sets the label of this rule.
+    def label=(label)
+      @label = label.to_sym
     end
 
+    # A label for this rule. If a rule has a label, all matches that it creates
+    # will be accessible as named captures from the scope of their parent match
+    # using that label.
+    attr_reader :label
+
     # Specifies a module that will be used to extend all Match objects that
     # result from this rule. If +mod+ is a Proc, it is used to create an
-    # anonymous module.
+    # anonymous module with a +value+ method.
     def extension=(mod)
       if Proc === mod
-        begin
-          tmp = Module.new(&mod)
-          raise ArgumentError if tmp.instance_methods.empty?
-          mod = tmp
-        rescue NoMethodError, ArgumentError, NameError
-          mod = Module.new { define_method(:value, &mod) }
-        end
+        mod = Module.new { define_method(:value, &mod) }
       end
 
       raise ArgumentError unless Module === mod
@@ -543,18 +522,22 @@ module Citrus
     #             +false+.
     # consume::   If this is +true+ a ParseError will be raised during a parse
     #             unless the entire input string is consumed. Defaults to
-    #             +false+.
+    #             +true+.
     def parse(string, options={})
       opts = default_parse_options.merge(options)
 
-      input = Input.new(string)
-      input.memoize! if opts[:memoize]
+      input = if opts[:memoize]
+        MemoizingInput.new(string)
+      else
+        Input.new(string)
+      end
+
       input.pos = opts[:offset] if opts[:offset] > 0
 
       events = input.exec(self)
       length = events[-1]
 
-      if !length || (opts[:consume] && length < (input.length - opts[:offset]))
+      if !length || (opts[:consume] && length < (string.length - opts[:offset]))
         raise ParseError.new(input)
       end
 
@@ -565,135 +548,71 @@ module Citrus
     def default_parse_options # :nodoc:
       { :offset   => 0,
         :memoize  => false,
-        :consume  => false
+        :consume  => true
       }
     end
 
     # Tests whether or not this rule matches on the given +string+. Returns the
     # length of the match if any can be made, +nil+ otherwise.
     def test(string)
-      input = Input.new(string)
-      input.test(self)
+      Input.new(string).test(self)
     end
 
     # Returns +true+ if this rule is a Terminal.
     def terminal?
-      is_a?(Terminal)
-    end
-
-    # Returns +true+ if this rule is able to propagate extensions from child
-    # rules to the scope of the parent, +false+ otherwise. In general, this will
-    # return +false+ for any rule whose match value is derived from an arbitrary
-    # number of child rules, such as a Repeat or a Sequence. Note that this is
-    # not true for Choice objects because they rely on exactly 1 rule to match,
-    # as do Proxy objects.
-    def propagates_extensions?
-      case self
-      when AndPredicate, NotPredicate, ButPredicate, Repeat, Sequence
-        false
-      else
-        true
-      end
-    end
-
-    # Returns +true+ if this rule needs to be surrounded by parentheses when
-    # using #embed.
-    def paren?
       false
     end
 
-    # Returns a string version of this rule that is suitable to be used in the
-    # string representation of another rule.
-    def embed
-      named? ? name.to_s : (paren? ? '(%s)' % to_s : to_s)
-    end
-
-    def inspect # :nodoc:
-      to_s
+    # Returns +true+ if this rule should extend a match but should not appear in
+    # its event stream.
+    def elide?
+      false
     end
 
-    def extend_match(match) # :nodoc:
-      match.names << name if named?
-      match.extend(extension) if extension
+    # Returns +true+ if this rule needs to be surrounded by parentheses when
+    # using #to_embedded_s.
+    def needs_paren? # :nodoc:
+      is_a?(Nonterminal) && rules.length > 1
     end
-  end
-
-  # A Terminal is a Rule that matches directly on the input stream and may not
-  # contain any other rule. Terminals are essentially wrappers for regular
-  # expressions. As such, the Citrus notation is identical to Ruby's regular
-  # expression notation, e.g.:
-  #
-  #     /expr/
-  #
-  # Character classes and the dot symbol may also be used in Citrus notation for
-  # compatibility with other parsing expression implementations, e.g.:
-  #
-  #     [a-zA-Z]
-  #     .
-  #
-  class Terminal
-    include Rule
 
-    def initialize(rule=/^/)
-      super
-      @rule = rule
+    # Returns the Citrus notation of this rule as a string.
+    def to_s
+      if label
+        "#{label}:" + (needs_paren? ? "(#{to_citrus})" : to_citrus)
+      else
+        to_citrus
+      end
     end
 
-    # The actual Regexp object this rule uses to match.
-    attr_reader :rule
+    alias_method :to_str, :to_s
 
-    # Returns an array of events for this rule on the given +input+.
-    def exec(input, events=[])
-      length = input.scan_full(rule, false, false)
-      if length
-        events << id
-        events << CLOSE
-        events << length
+    # Returns the Citrus notation of this rule as a string that is suitable to
+    # be embedded in the string representation of another rule.
+    def to_embedded_s # :nodoc:
+      if name
+        name.to_s
+      else
+        needs_paren? && label.nil? ? "(#{to_s})" : to_s
       end
-      events
     end
 
-    # Returns +true+ if this rule is case sensitive.
-    def case_sensitive?
-      !rule.casefold?
+    def ==(other)
+      case other
+      when Rule
+        to_s == other.to_s
+      else
+        super
+      end
     end
 
-    # Returns the Citrus notation of this rule as a string.
-    def to_s
-      rule.inspect
-    end
-  end
+    alias_method :eql?, :==
 
-  # A StringTerminal is a Terminal that may be instantiated from a String
-  # object. The Citrus notation is any sequence of characters enclosed in either
-  # single or double quotes, e.g.:
-  #
-  #     'expr'
-  #     "expr"
-  #
-  # This notation works the same as it does in Ruby; i.e. strings in double
-  # quotes may contain escape sequences while strings in single quotes may not.
-  # In order to specify that a string should ignore case when matching, enclose
-  # it in backticks instead of single or double quotes, e.g.:
-  #
-  #     `expr`
-  #
-  # Besides case sensitivity, case-insensitive strings have the same semantics
-  # as double-quoted strings.
-  class StringTerminal < Terminal
-    # The +flags+ will be passed directly to Regexp#new.
-    def initialize(rule='', flags=0)
-      super(Regexp.new(Regexp.escape(rule), flags))
-      @string = rule
+    def inspect # :nodoc:
+      to_s
     end
 
-    # Returns the Citrus notation of this rule as a string.
-    def to_s
-      if case_sensitive?
-        @string.inspect
-      else
-        @string.inspect.gsub(/^"|"$/, '`')
-      end
+    def extend_match(match) # :nodoc:
+      match.extend(extension) if extension
     end
   end
 
@@ -705,7 +624,6 @@ module Citrus
     include Rule
 
     def initialize(rule_name='<proxy>')
-      super
       self.rule_name = rule_name
     end
 
@@ -724,19 +642,29 @@ module Citrus
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
-      events << id
-
       index = events.size
-      start = index - 1
+
       if input.exec(rule, events).size > index
-        events << CLOSE
-        events << events[-2]
-      else
-        events.slice!(start, events.size)
+        # Proxy objects insert themselves into the event stream in place of the
+        # rule they are proxy for.
+        events[index] = self
       end
 
       events
     end
+
+    # Returns +true+ if this rule should extend a match but should not appear in
+    # its event stream.
+    def elide? # :nodoc:
+      rule.elide?
+    end
+
+    def extend_match(match) # :nodoc:
+      # Proxy objects preserve the extension of the rule they are proxy for, and
+      # may also use their own extension.
+      rule.extend_match(match)
+      super
+    end
   end
 
   # An Alias is a Proxy for a rule in the same grammar. It is used in rule
@@ -749,7 +677,7 @@ module Citrus
     include Proxy
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
+    def to_citrus # :nodoc:
       rule_name.to_s
     end
 
@@ -758,8 +686,14 @@ module Citrus
     # Searches this proxy's grammar and any included grammars for a rule with
     # this proxy's #rule_name. Raises an error if one cannot be found.
     def resolve!
-      grammar.rule(rule_name) or raise RuntimeError,
-        'No rule named "%s" in grammar %s' % [rule_name, grammar.name]
+      rule = grammar.rule(rule_name)
+
+      unless rule
+        raise RuntimeError,
+          "No rule named \"#{rule_name}\" in grammar #{grammar.name}"
+      end
+
+      rule
     end
   end
 
@@ -774,7 +708,7 @@ module Citrus
     include Proxy
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
+    def to_citrus # :nodoc:
       'super'
     end
 
@@ -783,8 +717,119 @@ module Citrus
     # Searches this proxy's included grammars for a rule with this proxy's
     # #rule_name. Raises an error if one cannot be found.
     def resolve!
-      grammar.super_rule(rule_name) or raise RuntimeError,
-        'No rule named "%s" in hierarchy of grammar %s' % [rule_name, grammar.name]
+      rule = grammar.super_rule(rule_name)
+
+      unless rule
+        raise RuntimeError,
+          "No rule named \"#{rule_name}\" in hierarchy of grammar #{grammar.name}"
+      end
+
+      rule
+    end
+  end
+
+  # A Terminal is a Rule that matches directly on the input stream and may not
+  # contain any other rule. Terminals are essentially wrappers for regular
+  # expressions. As such, the Citrus notation is identical to Ruby's regular
+  # expression notation, e.g.:
+  #
+  #     /expr/
+  #
+  # Character classes and the dot symbol may also be used in Citrus notation for
+  # compatibility with other parsing expression implementations, e.g.:
+  #
+  #     [a-zA-Z]
+  #     .
+  #
+  # Character classes have the same semantics as character classes inside Ruby
+  # regular expressions. The dot matches any character, including newlines.
+  class Terminal
+    include Rule
+
+    def initialize(regexp=/^/)
+      @regexp = regexp
+    end
+
+    # The actual Regexp object this rule uses to match.
+    attr_reader :regexp
+
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      length = input.scan_full(@regexp, false, false)
+
+      if length
+        events << self
+        events << CLOSE
+        events << length
+      end
+
+      events
+    end
+
+    # Returns +true+ if this rule is case sensitive.
+    def case_sensitive?
+      !@regexp.casefold?
+    end
+
+    def ==(other)
+      case other
+      when Regexp
+        @regexp == other
+      else
+        super
+      end
+    end
+
+    # Returns +true+ if this rule is a Terminal.
+    def terminal? # :nodoc:
+      true
+    end
+
+    # Returns the Citrus notation of this rule as a string.
+    def to_citrus # :nodoc:
+      @regexp.inspect
+    end
+  end
+
+  # A StringTerminal is a Terminal that may be instantiated from a String
+  # object. The Citrus notation is any sequence of characters enclosed in either
+  # single or double quotes, e.g.:
+  #
+  #     'expr'
+  #     "expr"
+  #
+  # This notation works the same as it does in Ruby; i.e. strings in double
+  # quotes may contain escape sequences while strings in single quotes may not.
+  # In order to specify that a string should ignore case when matching, enclose
+  # it in backticks instead of single or double quotes, e.g.:
+  #
+  #     `expr`
+  #
+  # Besides case sensitivity, case-insensitive strings have the same semantics
+  # as double-quoted strings.
+  class StringTerminal < Terminal
+    # The +flags+ will be passed directly to Regexp#new.
+    def initialize(rule='', flags=0)
+      super(Regexp.new(Regexp.escape(rule), flags))
+      @string = rule
+    end
+
+    def ==(other)
+      case other
+      when String
+        @string == other
+      else
+        super
+      end
+    end
+
+    # Returns the Citrus notation of this rule as a string.
+    def to_citrus # :nodoc:
+      if case_sensitive?
+        @string.inspect
+      else
+        @string.inspect.gsub(/^"|"$/, '`')
+      end
     end
   end
 
@@ -796,8 +841,7 @@ module Citrus
     include Rule
 
     def initialize(rules=[])
-      super
-      @rules = rules.map {|r| Rule.new(r) }
+      @rules = rules.map {|r| Rule.for(r) }
     end
 
     # An array of the actual Rule objects this rule uses to match.
@@ -809,8 +853,13 @@ module Citrus
     end
   end
 
-  # A Predicate is a Nonterminal that contains one other rule.
-  module Predicate
+  # An AndPredicate is a Nonterminal that contains a rule that must match. Upon
+  # success an empty match is returned and no input is consumed. The Citrus
+  # notation is any expression preceded by an ampersand, e.g.:
+  #
+  #     &expr
+  #
+  class AndPredicate
     include Nonterminal
 
     def initialize(rule='')
@@ -821,145 +870,108 @@ module Citrus
     def rule
       rules[0]
     end
-  end
-
-  # An AndPredicate is a Predicate that contains a rule that must match. Upon
-  # success an empty match is returned and no input is consumed. The Citrus
-  # notation is any expression preceded by an ampersand, e.g.:
-  #
-  #     &expr
-  #
-  class AndPredicate
-    include Predicate
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
       if input.test(rule)
-        events << id
+        events << self
         events << CLOSE
         events << 0
       end
+
       events
     end
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      '&' + rule.embed
+    def to_citrus # :nodoc:
+      '&' + rule.to_embedded_s
     end
   end
 
-  # A NotPredicate is a Predicate that contains a rule that must not match. Upon
-  # success an empty match is returned and no input is consumed. The Citrus
+  # A NotPredicate is a Nonterminal that contains a rule that must not match.
+  # Upon success an empty match is returned and no input is consumed. The Citrus
   # notation is any expression preceded by an exclamation mark, e.g.:
   #
   #     !expr
   #
   class NotPredicate
-    include Predicate
+    include Nonterminal
+
+    def initialize(rule='')
+      super([rule])
+    end
+
+    # Returns the Rule object this rule uses to match.
+    def rule
+      rules[0]
+    end
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
       unless input.test(rule)
-        events << id
+        events << self
         events << CLOSE
         events << 0
       end
+
       events
     end
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      '!' + rule.embed
+    def to_citrus # :nodoc:
+      '!' + rule.to_embedded_s
     end
   end
 
-  # A ButPredicate is a Predicate that consumes all characters until its rule
+  # A ButPredicate is a Nonterminal that consumes all characters until its rule
   # matches. It must match at least one character in order to succeed. The
   # Citrus notation is any expression preceded by a tilde, e.g.:
   #
   #     ~expr
   #
   class ButPredicate
-    include Predicate
+    include Nonterminal
 
-    DOT_RULE = Rule.new(DOT)
+    DOT_RULE = Rule.for(DOT)
+
+    def initialize(rule='')
+      super([rule])
+    end
+
+    # Returns the Rule object this rule uses to match.
+    def rule
+      rules[0]
+    end
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
       length = 0
+
       until input.test(rule)
         len = input.exec(DOT_RULE)[-1]
         break unless len
         length += len
       end
+
       if length > 0
-        events << id
+        events << self
         events << CLOSE
         events << length
       end
-      events
-    end
-
-    # Returns the Citrus notation of this rule as a string.
-    def to_s
-      '~' + rule.embed
-    end
-  end
-
-  # A Label is a Predicate that applies a new name to any matches made by its
-  # rule. The Citrus notation is any sequence of word characters (i.e.
-  # <tt>[a-zA-Z0-9_]</tt>) followed by a colon, followed by any other
-  # expression, e.g.:
-  #
-  #     label:expr
-  #
-  class Label
-    include Predicate
-
-    def initialize(rule='', label='<label>')
-      super(rule)
-      self.label = label
-    end
-
-    # Sets the name of this label.
-    def label=(label)
-      @label = label.to_sym
-    end
-
-    # The label this rule adds to all its matches.
-    attr_reader :label
-
-    # Returns an array of events for this rule on the given +input+.
-    def exec(input, events=[])
-      events << id
-
-      index = events.size
-      start = index - 1
-      if input.exec(rule, events).size > index
-        events << CLOSE
-        events << events[-2]
-      else
-        events.slice!(start, events.size)
-      end
 
       events
     end
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      label.to_s + ':' + rule.embed
-    end
-
-    def extend_match(match) # :nodoc:
-      match.names << label
-      super
+    def to_citrus # :nodoc:
+      '~' + rule.to_embedded_s
     end
   end
 
-  # A Repeat is a Predicate that specifies a minimum and maximum number of times
-  # its rule must match. The Citrus notation is an integer, +N+, followed by an
-  # asterisk, followed by another integer, +M+, all of which follow any other
-  # expression, e.g.:
+  # A Repeat is a Nonterminal that specifies a minimum and maximum number of
+  # times its rule must match. The Citrus notation is an integer, +N+, followed
+  # by an asterisk, followed by another integer, +M+, all of which follow any
+  # other expression, e.g.:
   #
   #     expr N*M
   #
@@ -976,22 +988,29 @@ module Citrus
   #     expr?
   #
   class Repeat
-    include Predicate
+    include Nonterminal
 
     def initialize(rule='', min=1, max=Infinity)
       raise ArgumentError, "Min cannot be greater than max" if min > max
-      super(rule)
+      super([rule])
       @range = Range.new(min, max)
     end
 
+    # Returns the Rule object this rule uses to match.
+    def rule
+      rules[0]
+    end
+
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
-      events << id
+      events << self
 
       index = events.size
       start = index - 1
       length = n = 0
-      while n < max && input.exec(rule, events).size > index
+      m = max
+
+      while n < m && input.exec(rule, events).size > index
         index = events.size
         length += events[-1]
         n += 1
@@ -1030,44 +1049,37 @@ module Citrus
     end
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      rule.embed + operator
-    end
-  end
-
-  # A List is a Nonterminal that contains any number of other rules and tests
-  # them for matches in sequential order.
-  module List
-    include Nonterminal
-
-    # See Rule#paren?.
-    def paren?
-      rules.length > 1
+    def to_citrus # :nodoc:
+      rule.to_embedded_s + operator
     end
   end
 
-  # A Choice is a List where only one rule must match. The Citrus notation is
-  # two or more expressions separated by a vertical bar, e.g.:
+  # A Sequence is a Nonterminal where all rules must match. The Citrus notation
+  # is two or more expressions separated by a space, e.g.:
   #
-  #     expr | expr
+  #     expr expr
   #
-  class Choice
-    include List
+  class Sequence
+    include Nonterminal
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
-      events << id
+      events << self
 
       index = events.size
       start = index - 1
-      n = 0
-      while n < rules.length && input.exec(rules[n], events).size == index
+      length = n = 0
+      m = rules.length
+
+      while n < m && input.exec(rules[n], events).size > index
+        index = events.size
+        length += events[-1]
         n += 1
       end
 
-      if index < events.size
+      if n == rules.length
         events << CLOSE
-        events << events[-2]
+        events << length
       else
         events.slice!(start, events.size)
       end
@@ -1076,181 +1088,272 @@ module Citrus
     end
 
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      rules.map {|r| r.embed }.join(' | ')
+    def to_citrus # :nodoc:
+      rules.map {|r| r.to_embedded_s }.join(' ')
     end
   end
 
-  # A Sequence is a List where all rules must match. The Citrus notation is two
-  # or more expressions separated by a space, e.g.:
+  # A Choice is a Nonterminal where only one rule must match. The Citrus
+  # notation is two or more expressions separated by a vertical bar, e.g.:
   #
-  #     expr expr
+  #     expr | expr
   #
-  class Sequence
-    include List
+  class Choice
+    include Nonterminal
 
     # Returns an array of events for this rule on the given +input+.
     def exec(input, events=[])
-      events << id
+      events << self
 
       index = events.size
-      start = index - 1
-      length = n = 0
-      while n < rules.length && input.exec(rules[n], events).size > index
-        index = events.size
-        length += events[-1]
+      n = 0
+      m = rules.length
+
+      while n < m && input.exec(rules[n], events).size == index
         n += 1
       end
 
-      if n == rules.length
+      if index < events.size
         events << CLOSE
-        events << length
+        events << events[-2]
       else
-        events.slice!(start, events.size)
+        events.pop
       end
 
       events
     end
 
+    # Returns +true+ if this rule should extend a match but should not appear in
+    # its event stream.
+    def elide? # :nodoc:
+      true
+    end
+
     # Returns the Citrus notation of this rule as a string.
-    def to_s
-      rules.map {|r| r.embed }.join(' ')
+    def to_citrus # :nodoc:
+      rules.map {|r| r.to_embedded_s }.join(' | ')
     end
   end
 
   # The base class for all matches. Matches are organized into a tree where any
-  # match may contain any number of other matches. This class provides several
-  # convenient tree traversal methods that help when examining parse results.
-  class Match < String
+  # match may contain any number of other matches. Nodes of the tree are lazily
+  # instantiated as needed. This class provides several convenient tree
+  # traversal methods that help when examining and interpreting parse results.
+  class Match
     def initialize(string, events=[])
-      raise ArgumentError, "Invalid events for match length %d" %
-        string.length if events[-1] && string.length != events[-1]
+      @string = string
 
-      super(string)
-      @events = events
+      if events.length > 0
+        if events[-1] != string.length
+          raise ArgumentError, "Invalid events for length #{string.length}"
+        end
 
-      extend!
-    end
+        elisions = []
 
-    # The array of events that was passed to the constructor.
-    attr_reader :events
+        while events[0].elide?
+          elisions.unshift(events.shift)
+          events = events.slice(0, events.length - 2)
+        end
 
-    # An array of all names of this match. A name is added to a match object
-    # for each rule that returns that object when matching. These names can then
-    # be used to determine which rules were satisfied by a given match.
-    def names
-      @names ||= []
-    end
+        events[0].extend_match(self)
 
-    # The name of the lowest level rule that originally created this match.
-    def name
-      names.first
-    end
+        elisions.each do |rule|
+          rule.extend_match(self)
+        end
+      end
 
-    # Returns +true+ if this match has the given +name+.
-    def has_name?(name)
-      names.include?(name.to_sym)
+      @events = events
     end
 
-    # Returns an array of all Rule objects that extend this match.
-    def extenders
-      @extenders ||= begin
-        extenders = []
-        @events.each do |event|
-          break if event == CLOSE
-          rule = Rule[event]
-          extenders.unshift(rule)
-          break unless rule.propagates_extensions?
-        end
-        extenders
-      end
+    # The array of events for this match.
+    attr_reader :events
+
+    # Returns the length of this match.
+    def length
+      @string.length
     end
 
-    # Returns an array of Match objects that are submatches of this match in the
-    # order they appeared in the input.
-    def matches
-      @matches ||= begin
-        matches = []
+    # Returns a hash of capture names to arrays of matches with that name,
+    # in the order they appeared in the input.
+    def captures
+      @captures ||= begin
+        captures = {}
         stack = []
         offset = 0
         close = false
         index = 0
+        last_length = nil
+        in_proxy = false
+        count = 0
 
         while index < @events.size
           event = @events[index]
+
           if close
             start = stack.pop
-            if stack.size == extenders.size
-              matches << Match.new(slice(offset, event), @events[start..index])
-              offset += event
+
+            if Rule === start
+              rule = start
+              os = stack.pop
+              start = stack.pop
+
+              match = Match.new(@string.slice(os, event), @events[start..index])
+
+              # We can lookup immediate submatches by their index.
+              if stack.size == 1
+                captures[count] = match
+                count += 1
+              end
+
+              # We can lookup matches that were created by proxy by the name of
+              # the rule they are proxy for.
+              if Proxy === rule
+                if captures[rule.rule_name]
+                  captures[rule.rule_name] << match
+                else
+                  captures[rule.rule_name] = [match]
+                end
+              end
+
+              # We can lookup matches that were created by rules with labels by
+              # that label.
+              if rule.label
+                if captures[rule.label]
+                  captures[rule.label] << match
+                else
+                  captures[rule.label] = [match]
+                end
+              end
+
+              in_proxy = false
             end
+
+            unless last_length
+              last_length = event
+            end
+
             close = false
           elsif event == CLOSE
             close = true
           else
             stack << index
+
+            # We can calculate the offset of this rule event by adding back the
+            # last match length.
+            if last_length
+              offset += last_length
+              last_length = nil
+            end
+
+            # We should not create captures when traversing the portion of the
+            # event stream that is masked by a proxy in the original rule
+            # definition.
+            unless in_proxy || stack.size == 1
+              stack << offset
+              stack << event
+              in_proxy = true if Proxy === event
+            end
           end
+
           index += 1
         end
 
-        matches
+        captures
       end
     end
 
-    # Returns an array of all sub-matches with the given +name+. If +deep+ is
-    # +false+, returns only sub-matches that are immediate descendants of this
-    # match.
-    def find(name, deep=true)
-      ms = matches.select {|m| m.has_name?(name) }
-      matches.each {|m| ms.concat(m.find(name, deep)) } if deep
-      ms
+    # Returns an array of all immediate submatches of this match.
+    def matches
+      @matches ||= (0...captures.size).map {|n| captures[n] }.compact
     end
 
-    # A shortcut for retrieving the first immediate sub-match of this match. If
-    # +name+ is given, attempts to retrieve the first immediate sub-match named
-    # +name+.
-    def first(name=nil)
-      name ? find(name, false).first : matches.first
+    # A shortcut for retrieving the first immediate submatch of this match.
+    def first
+      captures[0]
     end
 
     # The default value for a match is its string value. This method is
     # overridden in most cases to be more meaningful according to the desired
     # interpretation.
-    alias value to_s
-
-    # Allows sub-matches of this match to be retrieved by name as instance
-    # methods.
-    def method_missing(sym, *args)
-      if sym == :to_ary
-        # This is a workaround for a bug in Ruby 1.9 with classes that
-        # extend String.
-        super
+    alias_method :value, :to_s
+
+    # Allows methods of this match's string to be called directly and provides
+    # a convenient interface for retrieving the first match with a given name.
+    def method_missing(sym, *args, &block)
+      if @string.respond_to?(sym)
+        @string.__send__(sym, *args, &block)
       else
-        first(sym) or raise NoMatchError, 'No match named "%s" in %s (%s)' %
-          [sym, self, name]
+        captures[sym].first if captures[sym]
       end
     end
 
-    # Returns a string representation of this match that displays the entire
-    # match tree for easy viewing in the console.
-    def dump
-      dump_lines.join("\n")
+    def to_s
+      @string
     end
 
-    def dump_lines(indent='  ') # :nodoc:
-      line = to_s.inspect
-      line << ' (%s)' % names.join(',') unless names.empty?
-      matches.inject([line]) do |lines, m|
-        lines.concat(m.dump_lines(indent).map {|line| indent + line })
+    alias_method :to_str, :to_s
+
+    def ==(other)
+      case other
+      when String
+        @string == other
+      when Match
+        @string == other.to_s
+      else
+        super
       end
     end
 
-  private
+    alias_method :eql?, :==
 
-    def extend! # :nodoc:
-      extenders.each do |rule|
-        rule.extend_match(self)
+    def inspect
+      @string.inspect
+    end
+
+    # Prints the entire subtree of this match using the given +indent+ to
+    # indicate nested match levels. Useful for debugging.
+    def dump(indent=' ')
+      lines = []
+      stack = []
+      offset = 0
+      close = false
+      index = 0
+      last_length = nil
+
+      while index < @events.size
+        event = @events[index]
+
+        if close
+          os = stack.pop
+          start = stack.pop
+          rule = stack.pop
+
+          space = indent * (stack.size / 3)
+          string = @string.slice(os, event)
+          lines[start] = "#{space}#{string.inspect} rule=#{rule}, offset=#{os}, length=#{event}"
+
+          unless last_length
+            last_length = event
+          end
+
+          close = false
+        elsif event == CLOSE
+          close = true
+        else
+          if last_length
+            offset += last_length
+            last_length = nil
+          end
+
+          stack << event
+          stack << index
+          stack << offset
+        end
+
+        index += 1
       end
+
+      puts lines.compact.join("\n")
     end
   end
 end