citrus 2.0.1 → 2.1.1

data/README

@@ -5,8 +5,8 @@
                           Parsing Expressions for Ruby
 
 
-Citrus is a compact and powerful parsing library for 
-[Ruby](http://ruby-lang.org/) that combines the elegance and expressiveness of 
+Citrus is a compact and powerful parsing library for
+[Ruby](http://ruby-lang.org/) that combines the elegance and expressiveness of
 the language with the simplicity and power of
 [parsing expressions](http://en.wikipedia.org/wiki/Parsing_expression_grammar).
 
@@ -16,13 +16,13 @@ the language with the simplicity and power of
 
 Via [RubyGems](http://rubygems.org/):
 
-    $ sudo gem install citrus
+    $ gem install citrus
 
 From a local copy:
 
     $ git clone git://github.com/mjijackson/citrus.git
     $ cd citrus
-    $ rake package && sudo rake install
+    $ rake package install
 
 
 # Background
@@ -77,23 +77,23 @@ thereof.
 A Citrus grammar is really just a souped-up Ruby
 [module](http://ruby-doc.org/core/classes/Module.html). These modules may be
 included in other grammar modules in the same way that Ruby modules are normally
-used. This property allows you to divide a complex grammar into more manageable, 
-reusable pieces that may be combined at runtime. Any grammar rule with the same 
-name as a rule in an included grammar may access that rule with a mechanism 
+used. This property allows you to divide a complex grammar into more manageable,
+reusable pieces that may be combined at runtime. Any grammar rule with the same
+name as a rule in an included grammar may access that rule with a mechanism
 similar to Ruby's super keyword.
 
 ## Matches
 
-Matches are created by rule objects when they match on the input. A 
-[Match](api/classes/Citrus/Match.html) is actually a 
-[String](http://ruby-doc.org/core/classes/String.html) object with some extra 
+Matches are created by rule objects when they match on the input. A
+[Match](api/classes/Citrus/Match.html) is actually a
+[String](http://ruby-doc.org/core/classes/String.html) object with some extra
 information attached such as the name(s) of the rule(s) from which it was
 generated and any submatches it may contain.
 
 During a parse, matches are arranged in a tree structure where any match may
 contain any number of other matches. This structure is determined by the way in
-which the rule that generated each match is used in the grammar. For example, a 
-match that is created from a non-terminal rule that contains several other 
+which the rule that generated each match is used in the grammar. For example, a
+match that is created from a non-terminal rule that contains several other
 terminals will likewise contain several matches, one for each terminal.
 
 Match objects may be extended with semantic information in the form of methods.
@@ -207,28 +207,28 @@ See [Label](api/classes/Citrus/Label.html) for more information.
 
 ## Precedence
 
-The following table contains a list of all Citrus operators and their 
-precedence. A higher precedence indicates tighter binding.
-
-Operator    | Name                      | Precedence
------------ | ------------------------- | ----------
-''          | String (single quoted)    | 6
-""          | String (double quoted)    | 6
-[]          | Character class           | 6
-.           | Dot (any character)       | 6
-//          | Regular expression        | 6
-()          | Grouping                  | 6
-*           | Repetition (arbitrary)    | 5
-+           | Repetition (one or more)  | 5
-?           | Repetition (zero or one)  | 5
-&           | And predicate             | 4
-!           | Not predicate             | 4
-~           | But predicate             | 4
-:           | Label                     | 4
-<>          | Extension (module name)   | 3
-{}          | Extension (literal)       | 3
-e1 e2       | Sequence                  | 2
-e1 | e2     | Ordered choice            | 1
+The following table contains a list of all Citrus symbols and operators and
+their precedence. A higher precedence indicates tighter binding.
+
+Operator  | Name                      | Precedence
+--------- | ------------------------- | ----------
+''        | String (single quoted)    | 6
+""        | String (double quoted)    | 6
+[]        | Character class           | 6
+.         | Dot (any character)       | 6
+//        | Regular expression        | 6
+()        | Grouping                  | 6
+*         | Repetition (arbitrary)    | 5
++         | Repetition (one or more)  | 5
+?         | Repetition (zero or one)  | 5
+&         | And predicate             | 4
+!         | Not predicate             | 4
+~         | But predicate             | 4
+:         | Label                     | 4
+<>        | Extension (module name)   | 3
+{}        | Extension (literal)       | 3
+e1 e2     | Sequence                  | 2
+e1 | e2   | Ordered choice            | 1
 
 
 # Example
@@ -272,13 +272,12 @@ and "1 + 2+3", but it does not have enough semantic information to be able to
 actually interpret these expressions.
 
 At this point, when the grammar parses a string it generates a tree of
-[Match](api/classes/Citrus/Match.html) objects. Each match is created by a rule.
-A match knows what text it contains, its offset in the original input, and what
-submatches it contains.
+[Match](api/classes/Citrus/Match.html) objects. Each match is created by a rule
+and may itself be comprised of any number of submatches.
 
 Submatches are created whenever a rule contains another rule. For example, in
-the grammar above the number rule matches a string of digits followed by white
-space. Thus, a match generated by the number rule will contain two submatches.
+the grammar above `number` matches a string of digits followed by white space.
+Thus, a match generated by this rule will contain two submatches.
 
 We can define methods inside a set of curly braces that will be used to extend
 matches when they are created. This works in similar fashion to using Ruby's
@@ -352,14 +351,14 @@ Congratulations! You just ran your first piece of Citrus code.
 
 One interesting thing to notice about the above sequence of commands is the
 return value of [Citrus#load](api/classes/Citrus.html#M000003). When you use
-`Citrus.load` to
-load a grammar file (and likewise [Citrus#eval](api/classes/Citrus.html#M000004) to evaluate
-a raw string of grammar code), the return value is an array of all the grammars
-present in that file.
+`Citrus.load` to load a grammar file (and likewise
+[Citrus#eval](api/classes/Citrus.html#M000004) to evaluate a raw string of
+grammar code), the return value is an array of all the grammars present in that
+file.
 
-Take a look at 
+Take a look at
 [examples/calc.citrus](http://github.com/mjijackson/citrus/blob/master/examples/calc.citrus)
-for an example of a calculator that is able to parse and evaluate more complex 
+for an example of a calculator that is able to parse and evaluate more complex
 mathematical expressions.
 
 ## Implicit Value
@@ -383,23 +382,86 @@ as:
       }
     end
 
-Since no method name is explicitly specified in the semantic blocks, they may be 
+Since no method name is explicitly specified in the semantic blocks, they may be
 called using the `value` method.
 
 
+# Testing
+
+
+Citrus was designed to facilitate simple and powerful testing of grammars. To
+demonstrate how this is to be done, we'll use the `Addition` grammar from our
+previous [example](example.html). The following code demonstrates a simple test
+case that could be used to test that our grammar works properly.
+
+    class AdditionTest < Test::Unit::TestCase
+      def test_additive
+        match = Addition.parse('23 + 12', :root => :additive)
+        assert(match)
+        assert_equal('23 + 12', match)
+        assert_equal(35, match.value)
+      end
+
+      def test_number
+        match = Addition.parse('23', :root => :number)
+        assert(match)
+        assert_equal('23', match)
+        assert_equal(23, match.value)
+      end
+    end
+
+The key here is using the `root`
+[option](api/classes/Citrus/GrammarMethods.html#M000031) when performing the
+parse to specify the name of the rule at which the parse should start. In
+`test_number`, since `:number` was given the parse will start at that rule as if
+it were the root rule of the entire grammar. The ability to change the root rule
+on the fly like this enables easy unit testing of the entire grammar.
+
+Also note that because match objects are themselves strings, assertions may be
+made to test equality of match objects with string values.
+
+## Debugging
+
+When a parse fails, a [ParseError](api/classes/Citrus/ParseError.html) object is
+generated which provides a wealth of information about exactly where the parse
+failed. Using this object, you could possibly provide some useful feedback to
+the user about why the input was bad. The following code demonstrates one way
+to do this.
+
+    def parse_some_stuff(stuff)
+      match = StuffGrammar.parse(stuff)
+    rescue Citrus::ParseError => e
+      raise ArgumentError, "Invalid stuff on line %d, offset %d!" %
+        [e.line_number, e.line_offset]
+    end
+
+In addition to useful error objects, Citrus also includes a special file that
+should help grammar authors when debugging grammars. To get this extra
+functionality, simply `require 'citrus/debug'` instead of `require 'citrus'`
+when running your code.
+
+When debugging is enabled, you can visualize parse trees in the console as XML
+documents. This can help when determining which rules are generating which
+matches and how they are organized in the output. Also when debugging, each
+match object automatically records its offset in the original input, which can
+also be very helpful in keeping track of which offsets in the input generated
+which matches.
+
+
 # Links
 
 
 The primary resource for all things to do with parsing expressions can be found
-on the original [Packrat and Parsing Expression Grammars page](http://pdos.csail.mit.edu/~baford/packrat) at MIT.
+on the original [Packrat and Parsing Expression Grammars page](http://pdos.csail.mit.edu/~baford/packrat)
+at MIT.
 
-Also, a useful summary of parsing expression grammars can be found on 
+Also, a useful summary of parsing expression grammars can be found on
 [Wikipedia](http://en.wikipedia.org/wiki/Parsing_expression_grammar).
 
 Citrus draws inspiration from another Ruby library for writing parsing
 expression grammars, Treetop. While Citrus' syntax is similar to that of
-[Treetop](http://treetop.rubyforge.org), it's not identical. The link is 
-included here for those who may wish toexplore an alternative implementation.
+[Treetop](http://treetop.rubyforge.org), it's not identical. The link is
+included here for those who may wish to explore an alternative implementation.
 
 
 # License

data/doc/background.markdown

@@ -50,23 +50,23 @@ thereof.
 A Citrus grammar is really just a souped-up Ruby
 [module](http://ruby-doc.org/core/classes/Module.html). These modules may be
 included in other grammar modules in the same way that Ruby modules are normally
-used. This property allows you to divide a complex grammar into more manageable, 
-reusable pieces that may be combined at runtime. Any grammar rule with the same 
-name as a rule in an included grammar may access that rule with a mechanism 
+used. This property allows you to divide a complex grammar into more manageable,
+reusable pieces that may be combined at runtime. Any grammar rule with the same
+name as a rule in an included grammar may access that rule with a mechanism
 similar to Ruby's super keyword.
 
 ## Matches
 
-Matches are created by rule objects when they match on the input. A 
-[Match](api/classes/Citrus/Match.html) is actually a 
-[String](http://ruby-doc.org/core/classes/String.html) object with some extra 
+Matches are created by rule objects when they match on the input. A
+[Match](api/classes/Citrus/Match.html) is actually a
+[String](http://ruby-doc.org/core/classes/String.html) object with some extra
 information attached such as the name(s) of the rule(s) from which it was
 generated and any submatches it may contain.
 
 During a parse, matches are arranged in a tree structure where any match may
 contain any number of other matches. This structure is determined by the way in
-which the rule that generated each match is used in the grammar. For example, a 
-match that is created from a non-terminal rule that contains several other 
+which the rule that generated each match is used in the grammar. For example, a
+match that is created from a non-terminal rule that contains several other
 terminals will likewise contain several matches, one for each terminal.
 
 Match objects may be extended with semantic information in the form of methods.

data/doc/example.markdown

@@ -39,13 +39,12 @@ and "1 + 2+3", but it does not have enough semantic information to be able to
 actually interpret these expressions.
 
 At this point, when the grammar parses a string it generates a tree of
-[Match](api/classes/Citrus/Match.html) objects. Each match is created by a rule.
-A match knows what text it contains, its offset in the original input, and what
-submatches it contains.
+[Match](api/classes/Citrus/Match.html) objects. Each match is created by a rule
+and may itself be comprised of any number of submatches.
 
 Submatches are created whenever a rule contains another rule. For example, in
-the grammar above the number rule matches a string of digits followed by white
-space. Thus, a match generated by the number rule will contain two submatches.
+the grammar above `number` matches a string of digits followed by white space.
+Thus, a match generated by this rule will contain two submatches.
 
 We can define methods inside a set of curly braces that will be used to extend
 matches when they are created. This works in similar fashion to using Ruby's
@@ -119,14 +118,14 @@ Congratulations! You just ran your first piece of Citrus code.
 
 One interesting thing to notice about the above sequence of commands is the
 return value of [Citrus#load](api/classes/Citrus.html#M000003). When you use
-`Citrus.load` to
-load a grammar file (and likewise [Citrus#eval](api/classes/Citrus.html#M000004) to evaluate
-a raw string of grammar code), the return value is an array of all the grammars
-present in that file.
+`Citrus.load` to load a grammar file (and likewise
+[Citrus#eval](api/classes/Citrus.html#M000004) to evaluate a raw string of
+grammar code), the return value is an array of all the grammars present in that
+file.
 
-Take a look at 
+Take a look at
 [examples/calc.citrus](http://github.com/mjijackson/citrus/blob/master/examples/calc.citrus)
-for an example of a calculator that is able to parse and evaluate more complex 
+for an example of a calculator that is able to parse and evaluate more complex
 mathematical expressions.
 
 ## Implicit Value
@@ -150,5 +149,5 @@ as:
       }
     end
 
-Since no method name is explicitly specified in the semantic blocks, they may be 
+Since no method name is explicitly specified in the semantic blocks, they may be
 called using the `value` method.

data/doc/index.markdown

@@ -1,5 +1,5 @@
-Citrus is a compact and powerful parsing library for 
-[Ruby](http://ruby-lang.org/) that combines the elegance and expressiveness of 
+Citrus is a compact and powerful parsing library for
+[Ruby](http://ruby-lang.org/) that combines the elegance and expressiveness of
 the language with the simplicity and power of
 [parsing expressions](http://en.wikipedia.org/wiki/Parsing_expression_grammar).
 
@@ -9,10 +9,10 @@ the language with the simplicity and power of
 
 Via [RubyGems](http://rubygems.org/):
 
-    $ sudo gem install citrus
+    $ gem install citrus
 
 From a local copy:
 
     $ git clone git://github.com/mjijackson/citrus.git
     $ cd citrus
-    $ rake package && sudo rake install
+    $ rake package install

data/doc/links.markdown

@@ -2,12 +2,13 @@
 
 
 The primary resource for all things to do with parsing expressions can be found
-on the original [Packrat and Parsing Expression Grammars page](http://pdos.csail.mit.edu/~baford/packrat) at MIT.
+on the original [Packrat and Parsing Expression Grammars page](http://pdos.csail.mit.edu/~baford/packrat)
+at MIT.
 
-Also, a useful summary of parsing expression grammars can be found on 
+Also, a useful summary of parsing expression grammars can be found on
 [Wikipedia](http://en.wikipedia.org/wiki/Parsing_expression_grammar).
 
 Citrus draws inspiration from another Ruby library for writing parsing
 expression grammars, Treetop. While Citrus' syntax is similar to that of
-[Treetop](http://treetop.rubyforge.org), it's not identical. The link is 
-included here for those who may wish toexplore an alternative implementation.
+[Treetop](http://treetop.rubyforge.org), it's not identical. The link is
+included here for those who may wish to explore an alternative implementation.

data/doc/syntax.markdown

@@ -104,8 +104,8 @@ See [Label](api/classes/Citrus/Label.html) for more information.
 
 ## Precedence
 
-The following table contains a list of all Citrus operators and their 
-precedence. A higher precedence indicates tighter binding.
+The following table contains a list of all Citrus symbols and operators and
+their precedence. A higher precedence indicates tighter binding.
 
 Operator                  | Name                      | Precedence
 ------------------------- | ------------------------- | ----------

data/doc/testing.markdown

@@ -0,0 +1,60 @@
+# Testing
+
+
+Citrus was designed to facilitate simple and powerful testing of grammars. To
+demonstrate how this is to be done, we'll use the `Addition` grammar from our
+previous [example](example.html). The following code demonstrates a simple test
+case that could be used to test that our grammar works properly.
+
+    class AdditionTest < Test::Unit::TestCase
+      def test_additive
+        match = Addition.parse('23 + 12', :root => :additive)
+        assert(match)
+        assert_equal('23 + 12', match)
+        assert_equal(35, match.value)
+      end
+
+      def test_number
+        match = Addition.parse('23', :root => :number)
+        assert(match)
+        assert_equal('23', match)
+        assert_equal(23, match.value)
+      end
+    end
+
+The key here is using the `root`
+[option](api/classes/Citrus/GrammarMethods.html#M000031) when performing the
+parse to specify the name of the rule at which the parse should start. In
+`test_number`, since `:number` was given the parse will start at that rule as if
+it were the root rule of the entire grammar. The ability to change the root rule
+on the fly like this enables easy unit testing of the entire grammar.
+
+Also note that because match objects are themselves strings, assertions may be
+made to test equality of match objects with string values.
+
+## Debugging
+
+When a parse fails, a [ParseError](api/classes/Citrus/ParseError.html) object is
+generated which provides a wealth of information about exactly where the parse
+failed. Using this object, you could possibly provide some useful feedback to
+the user about why the input was bad. The following code demonstrates one way
+to do this.
+
+    def parse_some_stuff(stuff)
+      match = StuffGrammar.parse(stuff)
+    rescue Citrus::ParseError => e
+      raise ArgumentError, "Invalid stuff on line %d, offset %d!" %
+        [e.line_number, e.line_offset]
+    end
+
+In addition to useful error objects, Citrus also includes a special file that
+should help grammar authors when debugging grammars. To get this extra
+functionality, simply `require 'citrus/debug'` instead of `require 'citrus'`
+when running your code.
+
+When debugging is enabled, you can visualize parse trees in the console as XML
+documents. This can help when determining which rules are generating which
+matches and how they are organized in the output. Also when debugging, each
+match object automatically records its offset in the original input, which can
+also be very helpful in keeping track of which offsets in the input generated
+which matches.

data/lib/citrus.rb

@@ -8,7 +8,7 @@ require 'strscan'
 module Citrus
   autoload :File, 'citrus/file'
 
-  VERSION = [2, 0, 1]
+  VERSION = [2, 1, 1]
 
   # Returns the current version of Citrus as a string.
   def self.version
@@ -27,92 +27,204 @@ module Citrus
     file << '.citrus' unless F.file?(file)
     raise "Cannot find file #{file}" unless F.file?(file)
     raise "Cannot read file #{file}" unless F.readable?(file)
-    self.eval(F.read(file))
+    eval(F.read(file))
   end
 
   # Evaluates the given Citrus parsing expression grammar +code+ in the global
-  # scope. The +code+ may contain the definition of any number of modules.
-  # Returns an array of any grammar modules that are created.
+  # scope. Returns an array of any grammar modules that are created. Implicitly
+  # raises +SyntaxError+ on a failed parse.
   def self.eval(code)
-    File.parse(code).value
+    parse(code, :consume => true).value
   end
 
-  # This error is raised whenever a parse fails.
-  class ParseError < Exception
-    def initialize(input)
-      @input = input
-      msg = "Failed to parse input at offset %d\n" % offset
-      msg << detail
-      super(msg)
+  # Parses the given Citrus +code+ using the given +options+. Returns the
+  # generated match tree. Raises a +SyntaxError+ if the parse fails.
+  def self.parse(code, options={})
+    begin
+      File.parse(code, options)
+    rescue ParseError => e
+      raise SyntaxError.new(e)
     end
+  end
 
-    # The Input object that was used for the parse.
-    attr_reader :input
+  # A standard error class that all Citrus errors extend.
+  class Error < RuntimeError; end
 
-    # Returns the 0-based offset at which the error occurred in the input, i.e.
-    # the maximum offset in the input that was successfully parsed before the
-    # error occurred.
-    def offset
-      input.max_offset
+  # Raised when there is an error parsing Citrus code.
+  class SyntaxError < Error
+    # The +error+ given here should be a +ParseError+ object.
+    def initialize(error)
+      msg = "Syntax error on line %d at offset %d\n%s" %
+        [error.line_number, error.line_offset, error.detail]
+      super(msg)
     end
+  end
+
+  # Raised when a match cannot be found.
+  class NoMatchError < Error; end
 
-    # Returns the text of the line on which the error occurred.
-    def line
-      lines[line_index]
+  # Raised when a parse fails.
+  class ParseError < Error
+    # The +input+ given here is an instance of Citrus::Input.
+    def initialize(input)
+      @offset = input.max_offset
+      @line_offset = input.line_offset(offset)
+      @line_number = input.line_number(offset)
+      @line = input.line(offset)
+      msg = "Failed to parse input at offset %d\n" % offset
+      msg << detail
+      super(msg)
     end
 
-    # Returns the 1-based number of the line in the input where the error
+    # The 0-based offset at which the error occurred in the input, i.e. the
+    # maximum offset in the input that was successfully parsed before the error
     # occurred.
-    def line_number
-      line_index + 1
-    end
+    attr_reader :offset
 
-    alias lineno line_number
+    # The 0-based offset at which the error occurred on the line on which it
+    # occurred in the input.
+    attr_reader :line_offset
 
-    # Returns the 0-based offset at which the error occurred on the line on
-    # which it occurred.
-    def line_offset
-      pos = 0
-      each_line do |line|
-        len = line.length
-        return (offset - pos) if pos + len >= offset
-        pos += len
-      end
-      0
-    end
+    # The 1-based number of the line in the input where the error occurred.
+    attr_reader :line_number
+
+    # The text of the line in the input where the error occurred.
+    attr_reader :line
 
     # Returns a string that, when printed, gives a visual representation of
     # exactly where the error occurred on its line in the input.
     def detail
       "%s\n%s^" % [line, ' ' * line_offset]
     end
+  end
 
-  private
+  # This class represents the core of the parsing algorithm. It wraps the input
+  # string and serves matches to all nonterminals.
+  class Input < StringScanner
+    def initialize(string)
+      super(string)
+      @max_offset = 0
+    end
+
+    # The maximum offset that has been achieved during a parse.
+    attr_reader :max_offset
+
+    # A nested hash of rule id's to offsets and their respective matches. Only
+    # present if memoing is enabled.
+    attr_reader :cache
+
+    # The number of times the cache was hit. Only present if memoing is enabled.
+    attr_reader :cache_hits
 
-    def string
-      input.string
+    # Returns the length of this input.
+    def length
+      string.length
     end
 
+    # Returns an array containing the lines of text in the input.
     def lines
       string.send(string.respond_to?(:lines) ? :lines : :to_s).to_a
     end
 
+    # Iterates over the lines of text in the input using the given +block+.
     def each_line(&block)
       string.each_line(&block)
     end
 
-    # Returns the 0-based number of the line in the input where the error
-    # occurred.
-    def line_index
-      pos = 0
-      idx = 0
+    # Returns the 0-based offset of the given +pos+ in the input on the line
+    # on which it is found. +pos+ defaults to the current pointer position.
+    def line_offset(pos=pos)
+      p = 0
+      each_line do |line|
+        len = line.length
+        return (pos - p) if p + len >= pos
+        p += len
+      end
+      0
+    end
+
+    # Returns the 0-based number of the line that contains the character at the
+    # given +pos+. +pos+ defaults to the current pointer position.
+    def line_index(pos=pos)
+      p, n = 0, 0
       each_line do |line|
-        pos += line.length
-        return idx if pos >= offset
-        idx += 1
+        p += line.length
+        return n if p >= pos
+        n += 1
       end
       0
     end
+
+    # Returns the 1-based number of the line that contains the character at the
+    # given +pos+. +pos+ defaults to the current pointer position.
+    def line_number(pos=pos)
+      line_index(pos) + 1
+    end
+
+    alias lineno line_number
+
+    # Returns the text of the line that contains the character at the given
+    # +pos+. +pos+ defaults to the current pointer position.
+    def line(pos=pos)
+      lines[line_index(pos)]
+    end
+
+    # Returns the match for the given +rule+ at the current pointer position,
+    # which is +nil+ if no match can be made.
+    def match(rule)
+      offset = pos
+      match = rule.match(self)
+
+      if match
+        @max_offset = pos if pos > @max_offset
+      else
+        # Reset the position for the next attempt at a match.
+        self.pos = offset unless match
+      end
+
+      match
+    end
+
+    # Returns +true+ when using memoization to cache match results.
+    def memoized?
+      !! @cache
+    end
+
+    # Modifies this object to cache match results during a parse. This technique
+    # (also known as "Packrat" parsing) guarantees parsers will operate in
+    # linear time but costs significantly more in terms of time and memory
+    # required to perform a parse. For more information, please read the paper
+    # on Packrat parsing at http://pdos.csail.mit.edu/~baford/packrat/icfp02/.
+    def memoize!
+      return if memoized?
+
+      # Using +instance_eval+ here preserves access to +super+ within the
+      # methods we define inside the block.
+      instance_eval do
+        def match(rule) # :nodoc:
+          c = @cache[rule.id] ||= {}
+
+          if c.key?(pos)
+            @cache_hits += 1
+            c[pos]
+          else
+            c[pos] = super
+          end
+        end
+
+        # Resets all internal variables so that this object may be used in
+        # another parse.
+        def reset
+          super
+          @max_offset = 0
+          @cache = {}
+          @cache_hits = 0
+        end
+      end
+
+      @cache = {}
+      @cache_hits = 0
+    end
   end
 
   # Inclusion of this module into another extends the receiver with the grammar
@@ -361,85 +473,6 @@ module Citrus
     end
   end
 
-  # This class represents the core of the parsing algorithm. It wraps the input
-  # string and serves matches to all nonterminals.
-  class Input < StringScanner
-    def initialize(string)
-      super(string)
-      @max_offset = 0
-    end
-
-    # The maximum offset that has been achieved during a parse.
-    attr_reader :max_offset
-
-    # A nested hash of rule id's to offsets and their respective matches. Only
-    # present if memoing is enabled.
-    attr_reader :cache
-
-    # The number of times the cache was hit. Only present if memoing is enabled.
-    attr_reader :cache_hits
-
-    # Returns the length of this input.
-    def length
-      string.length
-    end
-
-    # Returns the match for a given +rule+ at the current position in the input.
-    def match(rule)
-      offset = pos
-      match = rule.match(self)
-
-      if match
-        @max_offset = pos if pos > @max_offset
-      else
-        # Reset the position for the next attempt at a match.
-        self.pos = offset
-      end
-
-      match
-    end
-
-    # Returns true if this input uses memoization to cache match results. See
-    # #memoize!.
-    def memoized?
-      !! @cache
-    end
-
-    # Modifies this object to cache match results during a parse. This technique
-    # (also known as "Packrat" parsing) guarantees parsers will operate in
-    # linear time but costs significantly more in terms of time and memory
-    # required to perform a parse. For more information, please read the paper
-    # on Packrat parsing at http://pdos.csail.mit.edu/~baford/packrat/icfp02/.
-    def memoize!
-      return if memoized?
-
-      # Using +instance_eval+ here preserves access to +super+ within the
-      # methods we define inside the block.
-      instance_eval do
-        def match(rule)
-          c = @cache[rule.id] ||= {}
-
-          if c.key?(pos)
-            @cache_hits += 1
-            c[pos]
-          else
-            c[pos] = super
-          end
-        end
-
-        def reset
-          super
-          @max_offset = 0
-          @cache = {}
-          @cache_hits = 0
-        end
-      end
-
-      @cache = {}
-      @cache_hits = 0
-    end
-  end
-
   # A Rule is an object that is used by a grammar to create matches on the
   # Input during parsing.
   module Rule
@@ -448,7 +481,7 @@ module Citrus
     #     Citrus::Rule.eval('"a" | "b"')
     #
     def self.eval(expr)
-      File.parse(expr, :root => :rule_body).value
+      Citrus.parse(expr, :root => :rule_body, :consume => true).value
     end
 
     # Returns a new Rule object depending on the type of object given.
@@ -668,7 +701,7 @@ module Citrus
 
     # Returns the Match for this rule on +input+, +nil+ if no match can be made.
     def match(input)
-      m = input.scan(@rule)
+      m = input.scan(rule)
       create_match(m) if m
     end
 
@@ -1016,7 +1049,8 @@ module Citrus
     def method_missing(sym, *args)
       m = first(sym)
       return m if m
-      raise 'No match named "%s" in %s (%s)' % [sym, self, name]
+      raise NoMatchError, 'No match named "%s" in %s (%s)' %
+        [sym, self, name || '<anonymous>']
     end
 
     def to_ary
@@ -1037,8 +1071,8 @@ class Object
   #     end
   #
   def grammar(name, &block)
-    obj = respond_to?(:const_set) ? self : Object
-    obj.const_set(name, Citrus::Grammar.new(&block))
+    namespace = respond_to?(:const_set) ? self : Object
+    namespace.const_set(name, Citrus::Grammar.new(&block))
   rescue NameError
     raise ArgumentError, 'Invalid grammar name: %s' % name
   end

data/lib/citrus/debug.rb

@@ -58,7 +58,7 @@ module Citrus
   ].each do |rule_class|
     rule_class.class_eval do
       alias original_match match
-  
+
       def match(input)
         m = original_match(input)
         m.offset = input.pos - m.length if m

data/lib/citrus/file.rb

@@ -1,41 +1,67 @@
 require 'citrus'
 
 module Citrus
+  # Some helper methods for rules that alias +module_name+ and don't want to
+  # use +Kernel#eval+ to retrieve Module objects.
+  module ModuleHelpers #:nodoc:
+    def module_segments
+      @module_segments ||= module_name.value.split('::')
+    end
+
+    def module_namespace
+      module_segments[0..-2].inject(Object) do |namespace, constant|
+        constant.empty? ? namespace : namespace.const_get(constant)
+      end
+    end
+
+    def module_basename
+      module_segments.last
+    end
+  end
+
   # A grammar for Citrus grammar files. This grammar is used in Citrus#eval to
   # parse and evaluate Citrus grammars and serves as a prime example of how to
   # create a complex grammar complete with semantic interpretation in pure Ruby.
-  File = Grammar.new do
+  File = Grammar.new do #:nodoc:
 
     ## Hierarchical syntax
 
     rule :file do
       all(:space, zero_or_more(any(:require, :grammar))) {
-        find(:require).each { |r| require r.value }
-        find(:grammar).map { |g| g.value }
+        find(:require).each {|r| require r.value }
+        find(:grammar).map {|g| g.value }
       }
     end
 
     rule :grammar do
       all(:grammar_keyword, :module_name, :grammar_body, :end_keyword) {
-        code = '%s = Citrus::Grammar.new' % module_name.value
-        grammar = eval(code, TOPLEVEL_BINDING)
+        include ModuleHelpers
+
+        def value
+          module_namespace.const_set(module_basename, grammar_body.value)
+        end
+      }
+    end
+
+    rule :grammar_body do
+      zero_or_more(any(:include, :root, :rule)) {
+        grammar = Grammar.new
 
-        modules = find(:include).map { |inc| eval(inc.value, TOPLEVEL_BINDING) }
-        modules.each { |mod| grammar.include(mod) }
+        find(:include).map do |inc|
+          grammar.include(inc.value)
+        end
 
         root = find(:root).last
         grammar.root(root.value) if root
 
-        find(:rule).each { |r| grammar.rule(r.rule_name.value, r.value) }
+        find(:rule).each do |r|
+          grammar.rule(r.rule_name.value, r.value)
+        end
 
         grammar
       }
     end
 
-    rule :grammar_body do
-      zero_or_more(any(:include, :root, :rule))
-    end
-
     rule :rule do
       all(:rule_keyword, :rule_name, :rule_body, :end_keyword) {
         rule_body.value
@@ -43,27 +69,37 @@ module Citrus
     end
 
     rule :rule_body do
-      all(:sequence, :choice) {
-        @choices ||= [ sequence ] + choice.value
-        values = @choices.map { |c| c.value }
-        values.length > 1 ? Choice.new(values) : values[0]
+      zero_or_one(:choice) {
+        # An empty rule definition matches the empty string.
+        matches.length > 0 ? choice.value : Rule.new('')
       }
     end
 
     rule :choice do
-      zero_or_more([ :bar, :sequence ]) {
-        matches.map { |m| m.matches[1] }
+      all(:sequence, zero_or_more([ :bar, :sequence ])) {
+        def rules
+          @rules ||= [ sequence.value ] + matches[1].matches.map {|m| m.matches[1].value }
+        end
+
+        def value
+          rules.length > 1 ? Choice.new(rules) : rules.first
+        end
       }
     end
 
     rule :sequence do
-      zero_or_more(:appendix) {
-        values = matches.map { |m| m.value }
-        values.length > 1 ? Sequence.new(values) : values[0]
+      one_or_more(:expression) {
+        def rules
+          @rules ||= matches.map {|m| m.value }
+        end
+
+        def value
+          rules.length > 1 ? Sequence.new(rules) : rules.first
+        end
       }
     end
 
-    rule :appendix do
+    rule :expression do
       all(:prefix, zero_or_one(:extension)) {
         rule = prefix.value
         extension = matches[1].first
@@ -105,7 +141,13 @@ module Citrus
     end
 
     rule :include do
-      all(:include_keyword, :module_name) { module_name.value }
+      all(:include_keyword, :module_name) {
+        include ModuleHelpers
+
+        def value
+          module_namespace.const_get(module_basename)
+        end
+      }
     end
 
     rule :root do
@@ -142,13 +184,13 @@ module Citrus
 
     rule :quoted_string do
       all(/(["'])(?:\\?.)*?\1/, :space) {
-        eval(first.to_s)
+        eval(first)
       }
     end
 
     rule :character_class do
       all(/\[(?:\\?.)*?\]/, :space) {
-        Regexp.new('\A' + first.to_s, nil, 'n')
+        Regexp.new('\A' + first, nil, 'n')
       }
     end
 
@@ -160,7 +202,7 @@ module Citrus
 
     rule :regular_expression do
       all(/\/(?:\\?.)*?\/[imxouesn]*/, :space) {
-        eval(first.to_s)
+        eval(first)
       }
     end
 
@@ -198,12 +240,16 @@ module Citrus
 
     rule :tag do
       all(:lt, :module_name, :gt) {
-        eval(module_name.value, TOPLEVEL_BINDING)
+        include ModuleHelpers
+
+        def value
+          module_namespace.const_get(module_basename)
+        end
       }
     end
 
     rule :block do
-      all(:lcurly, zero_or_more(any(:block, /[^}]+/)), :rcurly) {
+      all(:lcurly, zero_or_more(any(:block, /[^{}]+/)), :rcurly) {
         eval('Proc.new ' + to_s)
       }
     end

data/test/alias_test.rb

@@ -32,7 +32,7 @@ class AliasTest < Test::Unit::TestCase
     assert(match)
     assert('ab', match.value)
 
-    assert_raise RuntimeError do
+    assert_raise NoMatchError do
       match.b
     end
   end

data/test/debug_test.rb

@@ -0,0 +1,23 @@
+require File.expand_path('../helper', __FILE__)
+
+# This file tests functionality that is only present when debugging is enabled.
+
+class DebugTest < Test::Unit::TestCase
+
+  def test_offset
+    match = Words.parse('one two')
+    assert(match)
+    assert_equal(0, match.offset)
+
+    words = match.find(:word)
+    assert(match)
+    assert_equal(2, words.length)
+
+    assert_equal('one', words[0])
+    assert_equal(0, words[0].offset)
+
+    assert_equal('two', words[1])
+    assert_equal(4, words[1].offset)
+  end
+
+end

data/test/file_test.rb

@@ -210,20 +210,60 @@ class CitrusFileTest < Test::Unit::TestCase
     assert_instance_of(AndPredicate, match.value)
   end
 
+  def test_empty
+    grammar = file(:rule_body)
+
+    match = grammar.parse('')
+    assert(match)
+  end
+
+  def test_choice
+    grammar = file(:choice)
+
+    match = grammar.parse('"a" | "b"')
+    assert(match)
+    assert_equal(2, match.rules.length)
+    assert_instance_of(Choice, match.value)
+
+    match = grammar.parse('"a" | ("b" "c")')
+    assert(match)
+    assert_equal(2, match.rules.length)
+    assert_instance_of(Choice, match.value)
+  end
+
   def test_sequence
     grammar = file(:sequence)
 
     match = grammar.parse('"" ""')
     assert(match)
-    assert_kind_of(Rule, match.value)
+    assert_equal(2, match.rules.length)
     assert_instance_of(Sequence, match.value)
 
     match = grammar.parse('"a" "b" "c"')
     assert(match)
-    assert_kind_of(Rule, match.value)
+    assert_equal(3, match.rules.length)
     assert_instance_of(Sequence, match.value)
   end
 
+  def test_expression
+    grammar = file(:expression)
+
+    match = grammar.parse('"" <Module>')
+    assert(match)
+    assert_kind_of(Rule, match.value)
+    assert_kind_of(Module, match.value.extension)
+
+    match = grammar.parse('"" {}')
+    assert(match)
+    assert_kind_of(Rule, match.value)
+    assert_kind_of(Module, match.value.extension)
+
+    match = grammar.parse('"" {} ')
+    assert(match)
+    assert_kind_of(Rule, match.value)
+    assert_kind_of(Module, match.value.extension)
+  end
+
   def test_prefix
     grammar = file(:prefix)
 
@@ -248,25 +288,6 @@ class CitrusFileTest < Test::Unit::TestCase
     assert_instance_of(Label, match.value)
   end
 
-  def test_appendix
-    grammar = file(:appendix)
-
-    match = grammar.parse('"" <Module>')
-    assert(match)
-    assert_kind_of(Rule, match.value)
-    assert_kind_of(Module, match.value.extension)
-
-    match = grammar.parse('"" {}')
-    assert(match)
-    assert_kind_of(Rule, match.value)
-    assert_kind_of(Module, match.value.extension)
-
-    match = grammar.parse('"" {} ')
-    assert(match)
-    assert_kind_of(Rule, match.value)
-    assert_kind_of(Module, match.value.extension)
-  end
-
   def test_suffix
     grammar = file(:suffix)
 
@@ -325,11 +346,11 @@ class CitrusFileTest < Test::Unit::TestCase
 
     match = grammar.parse('include Module')
     assert(match)
-    assert_equal('Module', match.value)
+    assert_equal(Module, match.value)
 
     match = grammar.parse('include ::Module')
     assert(match)
-    assert_equal('::Module', match.value)
+    assert_equal(Module, match.value)
   end
 
   def test_root
@@ -577,6 +598,15 @@ class CitrusFileTest < Test::Unit::TestCase
     match = grammar.parse("{\n  def value\n    'a'\n  end\n} ")
     assert(match)
     assert(match.value)
+
+  end
+
+  def test_block_with_interpolation
+    grammar = file(:block)
+
+    match = grammar.parse('{ "#{number}" }')
+    assert(match)
+    assert(match.value)
   end
 
   def test_repeat

data/test/input_test.rb

@@ -0,0 +1,22 @@
+require File.expand_path('../helper', __FILE__)
+
+class InputTest < Test::Unit::TestCase
+
+  def test_new_input
+    input = Input.new("abc\ndef\nghi")
+    assert_equal(0, input.line_offset)
+    assert_equal(0, input.line_index)
+    assert_equal(1, input.line_number)
+    assert_equal("abc\n", input.line)
+  end
+
+  def test_advanced_input
+    input = Input.new("abc\ndef\nghi")
+    input.pos = 6
+    assert_equal(2, input.line_offset)
+    assert_equal(1, input.line_index)
+    assert_equal(2, input.line_number)
+    assert_equal("def\n", input.line)
+  end
+
+end

data/test/match_test.rb

@@ -53,20 +53,4 @@ class MatchTest < Test::Unit::TestCase
     assert_equal(15, match.find(:alpha).length)
   end
 
-  def test_offset
-    match = Words.parse('one two')
-    assert(match)
-    assert_equal(0, match.offset)
-
-    words = match.find(:word)
-    assert(match)
-    assert_equal(2, words.length)
-
-    assert_equal('one', words[0])
-    assert_equal(0, words[0].offset)
-
-    assert_equal('two', words[1])
-    assert_equal(4, words[1].offset)
-  end
-
 end

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 2
-  - 0
   - 1
-  version: 2.0.1
+  - 1
+  version: 2.1.1
 platform: ruby
 authors: 
 - Michael Jackson
@@ -61,6 +61,7 @@ files:
 - doc/license.markdown
 - doc/links.markdown
 - doc/syntax.markdown
+- doc/testing.markdown
 - examples/calc.citrus
 - examples/calc.rb
 - examples/ip.citrus
@@ -83,9 +84,11 @@ files:
 - test/calc_file_test.rb
 - test/calc_test.rb
 - test/choice_test.rb
+- test/debug_test.rb
 - test/file_test.rb
 - test/grammar_test.rb
 - test/helper.rb
+- test/input_test.rb
 - test/label_test.rb
 - test/match_test.rb
 - test/multibyte_test.rb
@@ -143,8 +146,10 @@ test_files:
 - test/calc_file_test.rb
 - test/calc_test.rb
 - test/choice_test.rb
+- test/debug_test.rb
 - test/file_test.rb
 - test/grammar_test.rb
+- test/input_test.rb
 - test/label_test.rb
 - test/match_test.rb
 - test/multibyte_test.rb