citrus 2.1.2 → 2.2.0

data/citrus.gemspec

@@ -25,7 +25,6 @@ Gem::Specification.new do |s|
 
   s.test_files = s.files.select {|path| path =~ /^test\/.*_test.rb/ }
 
-  s.add_dependency('builder')
   s.add_development_dependency('rake')
 
   s.has_rdoc = true

data/doc/extras.markdown

@@ -0,0 +1,16 @@
+# Extras
+
+
+Several files are included in the Citrus repository that make it easier to work
+with grammar files in various editors.
+
+## TextMate
+
+To install the Citrus [TextMate](http://macromates.com/) bundle, simply
+double-click on the `Citrus.tmbundle` file in the `extras` directory.
+
+## Vim
+
+To install the [Vim](http://www.vim.org/) scripts, copy the files in
+`extras/vim` to a directory in Vim's
+[runtimepath](http://vimdoc.sourceforge.net/htmldoc/options.html#\'runtimepath\').

data/doc/syntax.markdown

@@ -10,46 +10,57 @@ already be familiar to Ruby programmers.
 Terminals may be represented by a string or a regular expression. Both follow
 the same rules as Ruby string and regular expression literals.
 
-    'abc'
-    "abc\n"
-    /\xFF/
+    'abc'         # match "abc"
+    "abc\n"       # match "abc\n"
+    /abc/i        # match "abc" in any case
+    /\xFF/        # match "\xFF"
 
 Character classes and the dot (match anything) symbol are supported as well for
 compatibility with other parsing expression implementations.
 
     [a-z0-9]      # match any lowercase letter or digit
     [\x00-\xFF]   # match any octet
-    .             # match anything, even new lines
+    .             # match any single character, including new lines
 
-See [Terminal](api/classes/Citrus/Terminal.html) for more information.
+Also, strings may use backticks instead of quotes to indicate that they should
+match in a case-insensitive manner.
+
+    `abc`         # match "abc" in any case
+
+See [Terminal](api/classes/Citrus/Terminal.html) and
+[StringTerminal](api/classes/Citrus/StringTerminal.html) for more information.
 
 ## Repetition
 
 Quantifiers may be used after any expression to specify a number of times it
-must match. The universal form of a quantifier is N*M where N is the minimum and
-M is the maximum number of times the expression may match.
+must match. The universal form of a quantifier is `N*M` where `N` is the minimum
+and `M` is the maximum number of times the expression may match.
 
-    'abc'1*2      # match "abc" a minimum of one, maximum
-                  # of two times
+    'abc'1*2      # match "abc" a minimum of one, maximum of two times
     'abc'1*       # match "abc" at least once
     'abc'*2       # match "abc" a maximum of twice
 
-The + and ? operators are supported as well for the common cases of 1* and *1
-respectively.
+Additionally, the minimum and maximum may be omitted entirely to specify that an
+expression may match zero or more times.
+
+    'abc'*        # match "abc" zero or more times
+
+The `+` and `?` operators are supported as well for the common cases of `1*` and
+`*1` respectively.
 
-    'abc'+        # match "abc" at least once
-    'abc'?        # match "abc" a maximum of once
+    'abc'+        # match "abc" one or more times
+    'abc'?        # match "abc" zero or one time
 
 See [Repeat](api/classes/Citrus/Repeat.html) for more information.
 
 ## Lookahead
 
-Both positive and negative lookahead are supported in Citrus. Use the & and !
-operators to indicate that an expression either should or should not match. In
-neither case is any input consumed.
+Both positive and negative lookahead are supported in Citrus. Use the `&` and
+`!` operators to indicate that an expression either should or should not match.
+In neither case is any input consumed.
 
     &'a' 'b'      # match a "b" preceded by an "a"
-    !'a' 'b'      # match a "b" that is not preceded by an "a"
+    'a' !'b'      # match an "a" that is not followed by a "b"
     !'a' .        # match any character except for "a"
 
 A special form of lookahead is also supported which will match any character
@@ -75,20 +86,17 @@ See [Sequence](api/classes/Citrus/Sequence.html) for more information.
 ## Choices
 
 Ordered choice is indicated by a vertical bar that separates two expressions.
-Note that any operator binds more tightly than the bar.
+When using choice, each expression is tried in order. When one matches, the
+rule returns the match immediately without trying the remaining rules.
 
     'a' | 'b'       # match "a" or "b"
     'a' 'b' | 'c'   # match "a" then "b" (in sequence), or "c"
 
-See [Choice](api/classes/Citrus/Choice.html) for more information.
-
-## Super
-
-When including a grammar inside another, all rules in the child that have the
-same name as a rule in the parent also have access to the "super" keyword to
-invoke the parent rule.
+It is important to note when using ordered choice that any operator binds more
+tightly than the vertical bar. A full chart of operators and their respective
+levels of precedence is below.
 
-See [Super](api/classes/Citrus/Super.html) for more information.
+See [Choice](api/classes/Citrus/Choice.html) for more information.
 
 ## Labels
 
@@ -96,12 +104,50 @@ Match objects may be referred to by a different name than the rule that
 originally generated them. Labels are created by placing the label and a colon
 immediately preceding any expression.
 
-    chars:/[a-z]+/  # the characters matched by the regular
-                    # expression may be referred to as "chars"
-                    # in a block method
+    chars:/[a-z]+/  # the characters matched by the regular expression
+                    # may be referred to as "chars" in an extension
+                    # method
 
 See [Label](api/classes/Citrus/Label.html) for more information.
 
+## Grouping
+
+As is common in many programming languages, parentheses may be used to override
+the normal binding order of operators.
+
+    'a' ('b' | 'c')   # match "a", then "b" or "c"
+
+## Extensions
+
+Extensions may be specified using either "module" or "block" syntax. When using
+module syntax, specify the name of a module that is used to extend match objects
+in between less than and greater than symbols.
+
+    [a-z0-9]5*9 <CouponCode>  # match a string that consists of any lower
+                              # cased letter or digit between 5 and 9
+                              # times and extend the match with the
+                              # CouponCode module
+
+Additionally, extensions may be specified inline using curly braces. Inside the
+curly braces you may embed method definitions that will be used to extend match
+objects.
+
+    # match any digit and return its integer value when calling the
+    # #value method on the match object
+    [0-9] {
+      def value
+        to_i
+      end
+    }
+
+## Super
+
+When including a grammar inside another, all rules in the child that have the
+same name as a rule in the parent also have access to the `super` keyword to
+invoke the parent rule.
+
+See [Super](api/classes/Citrus/Super.html) for more information.
+
 ## Precedence
 
 The following table contains a list of all Citrus symbols and operators and
@@ -111,6 +157,7 @@ Operator                  | Name                      | Precedence
 ------------------------- | ------------------------- | ----------
 `''`                      | String (single quoted)    | 6
 `""`                      | String (double quoted)    | 6
+<code>``</code>           | String (case insensitive) | 6
 `[]`                      | Character class           | 6
 `.`                       | Dot (any character)       | 6
 `//`                      | Regular expression        | 6

data/doc/testing.markdown

@@ -22,12 +22,11 @@ case that could be used to test that our grammar works properly.
       end
     end
 
-The key here is using the `root`
-[option](api/classes/Citrus/GrammarMethods.html#M000031) when performing the
-parse to specify the name of the rule at which the parse should start. In
-`test_number`, since `:number` was given the parse will start at that rule as if
-it were the root rule of the entire grammar. The ability to change the root rule
-on the fly like this enables easy unit testing of the entire grammar.
+The key here is using the `:root` option when performing the parse to specify
+the name of the rule at which the parse should start. In `test_number`, since
+`:number` was given the parse will start at that rule as if it were the root
+rule of the entire grammar. The ability to change the root rule on the fly like
+this enables easy unit testing of the entire grammar.
 
 Also note that because match objects are themselves strings, assertions may be
 made to test equality of match objects with string values.
@@ -36,9 +35,9 @@ made to test equality of match objects with string values.
 
 When a parse fails, a [ParseError](api/classes/Citrus/ParseError.html) object is
 generated which provides a wealth of information about exactly where the parse
-failed. Using this object, you could possibly provide some useful feedback to
-the user about why the input was bad. The following code demonstrates one way
-to do this.
+failed including the offset, line number, line text, and line offset. Using this
+object, you could possibly provide some useful feedback to the user about why
+the input was bad. The following code demonstrates one way to do this.
 
     def parse_some_stuff(stuff)
       match = StuffGrammar.parse(stuff)
@@ -47,14 +46,7 @@ to do this.
         [e.line_number, e.line_offset]
     end
 
-In addition to useful error objects, Citrus also includes a special file that
-should help grammar authors when debugging grammars. To get this extra
-functionality, simply `require 'citrus/debug'` instead of `require 'citrus'`
-when running your code.
-
-When debugging is enabled, you can visualize parse trees in the console as XML
-documents. This can help when determining which rules are generating which
-matches and how they are organized in the output. Also when debugging, each
-match object automatically records its offset in the original input, which can
-also be very helpful in keeping track of which offsets in the input generated
-which matches.
+In addition to useful error objects, Citrus also includes a means of visualizing
+match trees in the console via `Match#dump`. This can help when determining
+which rules are generating which matches and how they are organized in the
+match tree.

data/examples/calc.citrus

@@ -5,7 +5,7 @@
 # An identical grammar that is written using pure Ruby can be found in calc.rb.
 grammar Calc
 
-  ## Hierarchy
+  ## Hierarchical syntax
 
   rule term
     additive | factor
@@ -55,50 +55,51 @@ grammar Calc
     (lparen term rparen) { term.value }
   end
 
-  ## Syntax
+  ## Lexical syntax
 
   rule number
     float | integer
   end
 
   rule float
-    (digits '.' digits space) { strip.to_f }
+    (digits '.' digits space*) { strip.to_f }
   end
 
   rule integer
-    (digits space) { strip.to_i }
+    (digits space*) { strip.to_i }
   end
 
   rule digits
+    # Numbers may contain underscores in Ruby.
     [0-9]+ ('_' [0-9]+)*
   end
 
   rule additive_operator
-    (('+' | '-') space) { |a, b|
+    (('+' | '-') space*) { |a, b|
       a.send(strip, b)
     }
   end
 
   rule multiplicative_operator
-    (('*' | '/' | '%') space) { |a, b|
+    (('*' | '/' | '%') space*) { |a, b|
       a.send(strip, b)
     }
   end
 
   rule exponential_operator
-    ('**' space) { |a, b|
+    ('**' space*) { |a, b|
       a ** b
     }
   end
 
   rule unary_operator
-    (('~' | '+' | '-') space) { |n|
+    (('~' | '+' | '-') space*) { |n|
       # Unary + and - require an @.
       n.send(strip == '~' ? strip : '%s@' % strip)
     }
   end
 
-  rule lparen '(' space  end
-  rule rparen ')' space  end
-  rule space  [ \t\n\r]* end
+  rule lparen '(' space* end
+  rule rparen ')' space* end
+  rule space  [ \t\n\r]  end
 end

data/examples/calc.rb

@@ -8,7 +8,7 @@ require 'citrus'
 # found in calc.citrus.
 grammar :Calc do
 
-  ## Hierarchy
+  ## Hierarchical syntax
 
   rule :term do
     any(:additive, :factor)
@@ -58,50 +58,51 @@ grammar :Calc do
     all(:lparen, :term, :rparen) { term.value }
   end
 
-  ## Syntax
+  ## Lexical syntax
 
   rule :number do
     any(:float, :integer)
   end
 
   rule :float do
-    all(:digits, '.', :digits, :space) { strip.to_f }
+    all(:digits, '.', :digits, zero_or_more(:space)) { strip.to_f }
   end
 
   rule :integer do
-    all(:digits, :space) { strip.to_i }
+    all(:digits, zero_or_more(:space)) { strip.to_i }
   end
 
   rule :digits do
+    # Numbers may contain underscores in Ruby.
     /[0-9]+(?:_[0-9]+)*/
   end
 
   rule :additive_operator do
-    all(any('+', '-'), :space) { |a, b|
+    all(any('+', '-'), zero_or_more(:space)) { |a, b|
       a.send(strip, b)
     }
   end
 
   rule :multiplicative_operator do
-    all(any('*', '/', '%'), :space) { |a, b|
+    all(any('*', '/', '%'), zero_or_more(:space)) { |a, b|
       a.send(strip, b)
     }
   end
 
   rule :exponential_operator do
-    all('**', :space) { |a, b|
+    all('**', zero_or_more(:space)) { |a, b|
       a ** b
     }
   end
 
   rule :unary_operator do
-    all(any('~', '+', '-'), :space) { |n|
+    all(any('~', '+', '-'), zero_or_more(:space)) { |n|
       # Unary + and - require an @.
       n.send(strip == '~' ? strip : '%s@' % strip)
     }
   end
 
-  rule :lparen, ['(', :space]
-  rule :rparen, [')', :space]
-  rule :space,  /[ \t\n\r]*/
+  rule :lparen, ['(', zero_or_more(:space)]
+  rule :rparen, [')', zero_or_more(:space)]
+  rule :space,  /[ \t\n\r]/
 end

data/lib/citrus.rb

@@ -8,7 +8,7 @@ require 'strscan'
 module Citrus
   autoload :File, 'citrus/file'
 
-  VERSION = [2, 1, 2]
+  VERSION = [2, 2, 0]
 
   # Returns the current version of Citrus as a string.
   def self.version
@@ -22,6 +22,8 @@ module Citrus
 
   F = ::File
 
+  CLOSE = -1
+
   # Loads the grammar from the given +file+ into the global scope using #eval.
   def self.load(file)
     file << '.citrus' unless F.file?(file)
@@ -40,26 +42,12 @@ module Citrus
   # Parses the given Citrus +code+ using the given +options+. Returns the
   # generated match tree. Raises a +SyntaxError+ if the parse fails.
   def self.parse(code, options={})
-    begin
-      File.parse(code, options)
-    rescue ParseError => e
-      raise SyntaxError.new(e)
-    end
+    File.parse(code, options)
   end
 
   # A standard error class that all Citrus errors extend.
   class Error < RuntimeError; end
 
-  # Raised when there is an error parsing Citrus code.
-  class SyntaxError < Error
-    # The +error+ given here should be a +ParseError+ object.
-    def initialize(error)
-      msg = "Syntax error on line %d at offset %d\n%s" %
-        [error.line_number, error.line_offset, error.detail]
-      super(msg)
-    end
-  end
-
   # Raised when a match cannot be found.
   class NoMatchError < Error; end
 
@@ -71,8 +59,8 @@ module Citrus
       @line_offset = input.line_offset(offset)
       @line_number = input.line_number(offset)
       @line = input.line(offset)
-      msg = "Failed to parse input at offset %d\n" % offset
-      msg << detail
+      msg = "Failed to parse input on line %d at offset %d\n%s" %
+        [line_number, line_offset, detail]
       super(msg)
     end
 
@@ -106,7 +94,7 @@ module Citrus
       @max_offset = 0
     end
 
-    # The maximum offset that has been achieved during a parse.
+    # The maximum offset in the input that was successfully parsed.
     attr_reader :max_offset
 
     # A nested hash of rule id's to offsets and their respective matches. Only
@@ -116,11 +104,11 @@ module Citrus
     # The number of times the cache was hit. Only present if memoing is enabled.
     attr_reader :cache_hits
 
-    # Resets all internal variables so that this object may be used in
-    # another parse.
-    def reset
-      super
+    # Resets all internal variables so that this object may be used in another
+    # parse.
+    def reset # :nodoc:
       @max_offset = 0
+      super
     end
 
     # Returns the length of this input.
@@ -153,7 +141,7 @@ module Citrus
     # Returns the 0-based number of the line that contains the character at the
     # given +pos+. +pos+ defaults to the current pointer position.
     def line_index(pos=pos)
-      p, n = 0, 0
+      p = n = 0
       each_line do |line|
         p += line.length
         return n if p >= pos
@@ -176,20 +164,29 @@ module Citrus
       lines[line_index(pos)]
     end
 
-    # Returns the match for the given +rule+ at the current pointer position,
-    # which is +nil+ if no match can be made.
-    def match(rule)
-      offset = pos
-      match = rule.match(self)
+    # Returns an array of events for the given +rule+ at the current pointer
+    # position. Objects in this array may be one of three types: a rule id,
+    # Citrus::CLOSE, or a length.
+    def exec(rule, events=[])
+      start = pos
+      index = events.size
 
-      if match
+      rule.exec(self, events)
+
+      if index < events.size
+        self.pos = start + events[-1]
         @max_offset = pos if pos > @max_offset
       else
-        # Reset the position for the next attempt at a match.
-        self.pos = offset unless match
+        self.pos = start
       end
 
-      match
+      events
+    end
+
+    # Returns the length of a match for the given +rule+ at the current pointer
+    # position, +nil+ if none can be made.
+    def test(rule)
+      rule.exec(self)[-1]
     end
 
     # Returns +true+ when using memoization to cache match results.
@@ -205,29 +202,31 @@ module Citrus
     def memoize!
       return if memoized?
 
+      @cache = {}
+      @cache_hits = 0
+
       # Using +instance_eval+ here preserves access to +super+ within the
       # methods we define inside the block.
       instance_eval do
-        def match(rule) # :nodoc:
+        def exec(rule, events=[]) # :nodoc:
           c = @cache[rule.id] ||= {}
 
-          if c.key?(pos)
+          e = if c[pos]
             @cache_hits += 1
             c[pos]
           else
-            c[pos] = super
+            c[pos] = super(rule)
           end
+
+          events.concat(e)
         end
 
         def reset # :nodoc:
-          super
-          @cache = {}
+          @cache.clear
           @cache_hits = 0
+          super
         end
       end
-
-      @cache = {}
-      @cache_hits = 0
     end
   end
 
@@ -266,6 +265,16 @@ module Citrus
       super
     end
 
+    # Parses the given +string+ using this grammar's root rule. Optionally, the
+    # name of a different rule may be given here as the value of the +:root+
+    # option. Otherwise, all options are the same as in Rule#parse.
+    def parse(string, options={})
+      rule_name = options.delete(:root) || root
+      rule = rule(rule_name)
+      raise 'No rule named "%s"' % rule_name unless rule
+      rule.parse(string, options)
+    end
+
     # Returns the name of this grammar as a string.
     def name
       super.to_s
@@ -310,9 +319,9 @@ module Citrus
     # and returns it on success. Returns +nil+ on failure.
     def super_rule(name)
       sym = name.to_sym
-      included_grammars.each do |g|
-        r = g.rule(sym)
-        return r if r
+      included_grammars.each do |grammar|
+        rule = grammar.rule(sym)
+        return rule if rule
       end
       nil
     end
@@ -433,48 +442,6 @@ module Citrus
       rule.extension = mod if mod
       rule
     end
-
-    # Parses the given input +string+ using the given +options+. If no match can
-    # be made, a ParseError is raised. See #default_parse_options for a detailed
-    # description of available parse options.
-    def parse(string, options={})
-      opts = default_parse_options.merge(options)
-      raise 'No root rule specified' unless opts[:root]
-
-      root_rule = rule(opts[:root])
-      raise 'No rule named "%s"' % root unless root_rule
-
-      input = Input.new(string)
-      input.memoize! if opts[:memoize]
-      input.pos = opts[:offset] if opts[:offset] > 0
-
-      match = input.match(root_rule)
-      if match.nil? || (opts[:consume] && input.length != match.length)
-        raise ParseError.new(input)
-      end
-
-      match
-    end
-
-    # The default set of options that is used in #parse. The options hash may
-    # have any of the following keys:
-    #
-    # offset::    The offset at which the parse should start. Defaults to 0.
-    # root::      The name of the root rule to use for the parse. Defaults
-    #             to the name supplied by calling #root.
-    # memoize::   If this is +true+ the matches generated during a parse are
-    #             memoized. See Input#memoize! for more information. Defaults to
-    #             +false+.
-    # consume::   If this is +true+ a ParseError will be raised during a parse
-    #             unless the entire input string is consumed. Defaults to
-    #             +false+.
-    def default_parse_options
-      { :offset   => 0,
-        :root     => root,
-        :memoize  => false,
-        :consume  => false
-      }
-    end
   end
 
   # A Rule is an object that is used by a grammar to create matches on the
@@ -491,12 +458,13 @@ module Citrus
     # Returns a new Rule object depending on the type of object given.
     def self.new(obj)
       case obj
-      when Rule           then obj
-      when Symbol         then Alias.new(obj)
-      when String, Regexp then Terminal.new(obj)
-      when Array          then Sequence.new(obj)
-      when Range          then Choice.new(obj.to_a)
-      when Numeric        then Terminal.new(obj.to_s)
+      when Rule     then obj
+      when Symbol   then Alias.new(obj)
+      when String   then StringTerminal.new(obj)
+      when Regexp   then Terminal.new(obj)
+      when Array    then Sequence.new(obj)
+      when Range    then Choice.new(obj.to_a)
+      when Numeric  then StringTerminal.new(obj.to_s)
       else
         raise ArgumentError, "Invalid rule object: %s" % obj.inspect
       end
@@ -504,26 +472,44 @@ module Citrus
 
     @unique_id = 0
 
-    # Generates a new rule id.
-    def self.new_id
-      @unique_id += 1
+    # A global registry for Rule objects. Keyed by rule id.
+    @rules = {}
+
+    # Adds the given +rule+ to the global registry and gives it an id.
+    def self.<<(rule) # :nodoc:
+      rule.id = (@unique_id += 1)
+      @rules[rule.id] = rule
     end
 
-    # The grammar this rule belongs to.
-    attr_accessor :grammar
+    # Returns the Rule object with the given +id+.
+    def self.[](id)
+      @rules[id]
+    end
 
-    # An integer id that is unique to this rule.
-    def id
-      @id ||= Rule.new_id
+    def initialize(*args) # :nodoc:
+      Rule << self
     end
 
+    # An integer id that is unique to this rule.
+    attr_accessor :id
+
+    # The grammar this rule belongs to.
+    attr_accessor :grammar
+
     # Sets the name of this rule.
     def name=(name)
       @name = name.to_sym
     end
 
-    # The name of this rule.
-    attr_reader :name
+    # Returns the name of this rule.
+    def name
+      @name || '<anonymous>'
+    end
+
+    # Returns +true+ if this rule has a name, +false+ otherwise.
+    def named?
+      !! @name
+    end
 
     # Specifies a module that will be used to extend all Match objects that
     # result from this rule. If +mod+ is a Proc, it is used to create an
@@ -532,9 +518,9 @@ module Citrus
       if Proc === mod
         begin
           tmp = Module.new(&mod)
-          raise ArgumentError unless tmp.instance_methods.any?
+          raise ArgumentError if tmp.instance_methods.empty?
           mod = tmp
-        rescue ArgumentError, NameError, NoMethodError
+        rescue NoMethodError, ArgumentError, NameError
           mod = Module.new { define_method(:value, &mod) }
         end
       end
@@ -547,11 +533,70 @@ module Citrus
     # The module this rule uses to extend new matches.
     attr_reader :extension
 
+    # Attempts to parse the given +string+ and return a Match if any can be
+    # made. The +options+ may contain any of the following keys:
+    #
+    # offset::    The offset in +string+ at which to start the parse. Defaults
+    #             to 0.
+    # memoize::   If this is +true+ the matches generated during a parse are
+    #             memoized. See Input#memoize! for more information. Defaults to
+    #             +false+.
+    # consume::   If this is +true+ a ParseError will be raised during a parse
+    #             unless the entire input string is consumed. Defaults to
+    #             +false+.
+    def parse(string, options={})
+      opts = default_parse_options.merge(options)
+
+      input = Input.new(string)
+      input.memoize! if opts[:memoize]
+      input.pos = opts[:offset] if opts[:offset] > 0
+
+      start = input.pos
+      events = input.exec(self)
+      length = events[-1]
+
+      if !length || (opts[:consume] && length < (input.length - opts[:offset]))
+        raise ParseError.new(input)
+      end
+
+      Match.new(string.slice(start, length), events)
+    end
+
+    # The default set of options to use when parsing.
+    def default_parse_options # :nodoc:
+      { :offset   => 0,
+        :memoize  => false,
+        :consume  => false
+      }
+    end
+
+    # Tests whether or not this rule matches on the given +string+. Returns the
+    # length of the match if any can be made, +nil+ otherwise.
+    def test(string)
+      input = Input.new(string)
+      input.test(self)
+    end
+
     # Returns +true+ if this rule is a Terminal.
     def terminal?
       is_a?(Terminal)
     end
 
+    # Returns +true+ if this rule is able to propagate extensions from child
+    # rules to the scope of the parent, +false+ otherwise. In general, this will
+    # return +false+ for any rule whose match value is derived from an arbitrary
+    # number of child rules, such as a Repeat or a Sequence. Note that this is
+    # not true for Choice objects because they rely on exactly 1 rule to match,
+    # as do Proxy objects.
+    def propagates_extensions?
+      case self
+      when AndPredicate, NotPredicate, ButPredicate, Repeat, Sequence
+        false
+      else
+        true
+      end
+    end
+
     # Returns +true+ if this rule needs to be surrounded by parentheses when
     # using #embed.
     def paren?
@@ -561,23 +606,90 @@ module Citrus
     # Returns a string version of this rule that is suitable to be used in the
     # string representation of another rule.
     def embed
-      name ? name.to_s : (paren? ? '(%s)' % to_s : to_s)
+      named? ? name.to_s : (paren? ? '(%s)' % to_s : to_s)
     end
 
     def inspect # :nodoc:
       to_s
     end
+  end
 
-  private
+  # A Terminal is a Rule that matches directly on the input stream and may not
+  # contain any other rule. Terminals are essentially wrappers for regular
+  # expressions. As such, the Citrus notation is identical to Ruby's regular
+  # expression notation, e.g.:
+  #
+  #     /expr/
+  #
+  # Character classes and the dot symbol may also be used in Citrus notation for
+  # compatibility with other parsing expression implementations, e.g.:
+  #
+  #     [a-zA-Z]
+  #     .
+  #
+  class Terminal
+    include Rule
+
+    def initialize(rule=/^/)
+      super
+      @rule = rule
+    end
+
+    # The actual Regexp object this rule uses to match.
+    attr_reader :rule
+
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      length = input.scan_full(rule, false, false)
+      if length
+        events << id
+        events << CLOSE
+        events << length
+      end
+      events
+    end
 
-    def extend_match(match, name)
-      match.extend(extension) if extension
-      match.names << name if name
-      match
+    # Returns +true+ if this rule is case sensitive.
+    def case_sensitive?
+      !rule.casefold?
     end
 
-    def create_match(data)
-      extend_match(Match.new(data), name)
+    # Returns the Citrus notation of this rule as a string.
+    def to_s
+      rule.inspect
+    end
+  end
+
+  # A StringTerminal is a Terminal that may be instantiated from a String
+  # object. The Citrus notation is any sequence of characters enclosed in either
+  # single or double quotes, e.g.:
+  #
+  #     'expr'
+  #     "expr"
+  #
+  # This notation works the same as it does in Ruby; i.e. strings in double
+  # quotes may contain escape sequences while strings in single quotes may not.
+  # In order to specify that a string should ignore case when matching, enclose
+  # it in backticks instead of single or double quotes, e.g.:
+  #
+  #     `expr`
+  #
+  # Besides case sensitivity, case-insensitive strings have the same semantics
+  # as double-quoted strings.
+  class StringTerminal < Terminal
+    # The +flags+ will be passed directly to Regexp#new.
+    def initialize(rule='', flags=0)
+      super(Regexp.new(Regexp.escape(rule), flags))
+      @string = rule
+    end
+
+    # Returns the Citrus notation of this rule as a string.
+    def to_s
+      if case_sensitive?
+        @string.inspect
+      else
+        @string.inspect.gsub(/^"|"$/, '`')
+      end
     end
   end
 
@@ -589,6 +701,7 @@ module Citrus
     include Rule
 
     def initialize(rule_name='<proxy>')
+      super
       self.rule_name = rule_name
     end
 
@@ -605,10 +718,9 @@ module Citrus
       @rule ||= resolve!
     end
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      m = input.match(rule)
-      extend_match(m, name) if m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      input.exec(rule, events)
     end
   end
 
@@ -631,10 +743,8 @@ module Citrus
     # Searches this proxy's grammar and any included grammars for a rule with
     # this proxy's #rule_name. Raises an error if one cannot be found.
     def resolve!
-      rule = grammar.rule(rule_name)
-      raise RuntimeError, 'No rule named "%s" in grammar %s' %
-        [rule_name, grammar.name] unless rule
-      rule
+      grammar.rule(rule_name) or raise RuntimeError,
+        'No rule named "%s" in grammar %s' % [rule_name, grammar.name]
     end
   end
 
@@ -658,60 +768,8 @@ module Citrus
     # Searches this proxy's included grammars for a rule with this proxy's
     # #rule_name. Raises an error if one cannot be found.
     def resolve!
-      rule = grammar.super_rule(rule_name)
-      raise RuntimeError, 'No rule named "%s" in hierarchy of grammar %s' %
-        [rule_name, grammar.name] unless rule
-      rule
-    end
-  end
-
-  # A Terminal is a Rule that matches directly on the input stream and may not
-  # contain any other rule. Terminals may be created from either a String or a
-  # Regexp object. When created from strings, the Citrus notation is any
-  # sequence of characters enclosed in either single or double quotes, e.g.:
-  #
-  #     'expr'
-  #     "expr"
-  #
-  # When created from a regular expression, the Citrus notation is identical to
-  # Ruby's regular expression notation, e.g.:
-  #
-  #     /expr/
-  #
-  # Character classes and the dot symbol may also be used in Citrus notation for
-  # compatibility with other parsing expression implementations, e.g.:
-  #
-  #     [a-zA-Z]
-  #     .
-  #
-  class Terminal
-    include Rule
-
-    def initialize(rule='')
-      case rule
-      when String
-        @string = rule
-        @rule = Regexp.new(Regexp.escape(rule))
-      when Regexp
-        @rule = rule
-      else
-        raise ArgumentError, "Cannot create terminal from object: %s" % 
-          rule.inspect
-      end
-    end
-
-    # The actual Regexp object this rule uses to match.
-    attr_reader :rule
-
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      m = input.scan(rule)
-      create_match(m) if m
-    end
-
-    # Returns the Citrus notation of this rule as a string.
-    def to_s
-      (@string || @rule).inspect
+      grammar.super_rule(rule_name) or raise RuntimeError,
+        'No rule named "%s" in hierarchy of grammar %s' % [rule_name, grammar.name]
     end
   end
 
@@ -723,15 +781,16 @@ module Citrus
     include Rule
 
     def initialize(rules=[])
+      super
       @rules = rules.map {|r| Rule.new(r) }
     end
 
     # An array of the actual Rule objects this rule uses to match.
     attr_reader :rules
 
-    def grammar=(grammar)
-      @rules.each {|r| r.grammar = grammar }
+    def grammar=(grammar) # :nodoc:
       super
+      @rules.each {|r| r.grammar = grammar }
     end
   end
 
@@ -758,9 +817,14 @@ module Citrus
   class AndPredicate
     include Predicate
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      create_match('') if input.match(rule)
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      if input.test(rule)
+        events << id
+        events << CLOSE
+        events << 0
+      end
+      events
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -778,9 +842,14 @@ module Citrus
   class NotPredicate
     include Predicate
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      create_match('') unless input.match(rule)
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      unless input.test(rule)
+        events << id
+        events << CLOSE
+        events << 0
+      end
+      events
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -800,16 +869,20 @@ module Citrus
 
     DOT_RULE = Rule.new(DOT)
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      matches = []
-      while input.match(rule).nil?
-        m = input.match(DOT_RULE)
-        break unless m
-        matches << m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      length = 0
+      until input.test(rule)
+        len = input.exec(DOT_RULE)[-1]
+        break unless len
+        length += len
+      end
+      if length > 0
+        events << id
+        events << CLOSE
+        events << length
       end
-      # Create a single match from the aggregate text value of all submatches.
-      create_match(matches.join) if matches.any?
+      events
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -841,12 +914,9 @@ module Citrus
     # The label this rule adds to all its matches.
     attr_reader :label
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    # When a Label makes a match, it re-names the match to the value of its
-    # #label.
-    def match(input)
-      m = input.match(rule)
-      extend_match(m, label) if m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      input.exec(rule, events)
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -878,20 +948,32 @@ module Citrus
     include Predicate
 
     def initialize(rule='', min=1, max=Infinity)
-      super(rule)
       raise ArgumentError, "Min cannot be greater than max" if min > max
+      super(rule)
       @range = Range.new(min, max)
     end
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      matches = []
-      while matches.length < @range.end
-        m = input.match(rule)
-        break unless m
-        matches << m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      events << id
+
+      index = events.size
+      start = index - 1
+      length = n = 0
+      while n < max && input.exec(rule, events).size > index
+        index = events.size
+        length += events[-1]
+        n += 1
       end
-      create_match(matches) if @range.include?(matches.length)
+
+      if n >= min
+        events << CLOSE
+        events << length
+      else
+        events.slice!(start, events.size)
+      end
+
+      events
     end
 
     # The minimum number of times this rule must match.
@@ -941,13 +1023,25 @@ module Citrus
   class Choice
     include List
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      rules.each do |rule|
-        m = input.match(rule)
-        return extend_match(m, name) if m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      events << id
+
+      index = events.size
+      start = index - 1
+      n = 0
+      while n < rules.length && input.exec(rules[n], events).size == index
+        n += 1
       end
-      nil
+
+      if index < events.size
+        events << CLOSE
+        events << events[-2]
+      else
+        events.slice!(start, events.size)
+      end
+
+      events
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -964,15 +1058,27 @@ module Citrus
   class Sequence
     include List
 
-    # Returns the Match for this rule on +input+, +nil+ if no match can be made.
-    def match(input)
-      matches = []
-      rules.each do |rule|
-        m = input.match(rule)
-        break unless m
-        matches << m
+    # Returns an array of events for this rule on the given +input+.
+    def exec(input, events=[])
+      events << id
+
+      index = events.size
+      start = index - 1
+      length = n = 0
+      while n < rules.length && input.exec(rules[n], events).size > index
+        index = events.size
+        length += events[-1]
+        n += 1
+      end
+
+      if n == rules.length
+        events << CLOSE
+        events << length
+      else
+        events.slice!(start, events.size)
       end
-      create_match(matches) if matches.length == rules.length
+
+      events
     end
 
     # Returns the Citrus notation of this rule as a string.
@@ -985,19 +1091,19 @@ module Citrus
   # match may contain any number of other matches. This class provides several
   # convenient tree traversal methods that help when examining parse results.
   class Match < String
-    def initialize(data)
-      case data
-      when String
-        super(data)
-      when Array
-        super(data.join)
-        @matches = data
-      else
-        raise ArgumentError, "Cannot create match from object: %s" % 
-          data.inspect
-      end
+    def initialize(string, events=[])
+      raise ArgumentError, "Invalid events for match length %d" %
+        string.length if events[-1] && string.length != events[-1]
+
+      super(string)
+      @events = events
+
+      extend!
     end
 
+    # The array of events that was passed to the constructor.
+    attr_reader :events
+
     # An array of all names of this match. A name is added to a match object
     # for each rule that returns that object when matching. These names can then
     # be used to determine which rules were satisfied by a given match.
@@ -1012,20 +1118,64 @@ module Citrus
 
     # Returns +true+ if this match has the given +name+.
     def has_name?(name)
-      names.include?(name)
+      names.include?(name.to_sym)
+    end
+
+    # Returns an array of all Rule objects that extend this match.
+    def extenders
+      @extenders ||= begin
+        extenders = []
+        @events.each do |event|
+          break if event == CLOSE
+          rule = Rule[event]
+          extenders.unshift(rule)
+          break unless rule.propagates_extensions?
+        end
+        extenders
+      end
+    end
+
+    # Returns a reference to the Rule object that first created this match.
+    def creator
+      extenders.first
     end
 
-    # An array of all sub-matches of this match.
+    # Returns an array of Match objects that are submatches of this match in the
+    # order they appeared in the input.
     def matches
-      @matches ||= []
+      @matches ||= begin
+        matches = []
+        stack = []
+        offset = 0
+        close = false
+        index = 0
+
+        while index < @events.size
+          event = @events[index]
+          if close
+            start = stack.pop
+            if stack.size == extenders.size
+              matches << Match.new(slice(offset, event), @events[start..index])
+              offset += event
+            end
+            close = false
+          elsif event == CLOSE
+            close = true
+          else
+            stack << index
+          end
+          index += 1
+        end
+
+        matches
+      end
     end
 
     # Returns an array of all sub-matches with the given +name+. If +deep+ is
     # +false+, returns only sub-matches that are immediate descendants of this
     # match.
     def find(name, deep=true)
-      sym = name.to_sym
-      ms = matches.select {|m| m.has_name?(sym) }
+      ms = matches.select {|m| m.has_name?(name) }
       matches.each {|m| ms.concat(m.find(name, deep)) } if deep
       ms
     end
@@ -1034,31 +1184,44 @@ module Citrus
     # +name+ is given, attempts to retrieve the first immediate sub-match named
     # +name+.
     def first(name=nil)
-      name.nil? ? matches.first : find(name, false).first
+      name ? find(name, false).first : matches.first
     end
 
-    # Returns +true+ if this match has no descendants (was created from a
-    # Terminal).
-    def terminal?
-      matches.length == 0
+    # Allows sub-matches of this match to be retrieved by name as instance
+    # methods.
+    def method_missing(sym, *args)
+      if sym == :to_ary
+        # This is a workaround for a bug in Ruby 1.9 with classes that
+        # extend String.
+        super
+      else
+        first(sym) or raise NoMatchError, 'No match named "%s" in %s (%s)' %
+          [sym, self, name]
+      end
     end
 
-    # Creates a new String object from the contents of this match.
-    def to_s
-      String.new(self)
+    # Returns a string representation of this match that displays the entire
+    # match tree for easy viewing in the console.
+    def dump
+      dump_lines.join("\n")
     end
 
-    # Allows sub-matches of this match to be retrieved by name as instance
-    # methods.
-    def method_missing(sym, *args)
-      m = first(sym)
-      return m if m
-      raise NoMatchError, 'No match named "%s" in %s (%s)' %
-        [sym, self, name || '<anonymous>']
+    def dump_lines(indent='  ') # :nodoc:
+      line = to_s.inspect
+      line << ' (%s)' % names.join(',') unless names.empty?
+      matches.inject([line]) do |lines, m|
+        lines.concat(m.dump_lines(indent).map {|line| indent + line })
+      end
     end
 
-    def to_ary
-      # This method intentionally left blank to work around a bug in Ruby 1.9.
+  private
+
+    # Extends this match with the extensions provided by its #rules.
+    def extend! # :nodoc:
+      extenders.each do |rule|
+        self.names << rule.name if rule.named?
+        extend(rule.extension) if rule.extension
+      end
     end
   end
 end