treetop 1.4.8 → 1.4.9

data/doc/using_in_ruby.markdown

@@ -1,6 +1,6 @@
 #Using Treetop Grammars in Ruby
 ##Using the Command Line Compiler
-You can `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
+You can compile `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
 
     tt foo.treetop bar.treetop
     tt foo.treetop -o foogrammar.rb
@@ -16,7 +16,10 @@ In order to use Polyglot dynamic loading of `.treetop` or `.tt` files though, yo
 in order to use Polyglot auto loading with Treetop in Ruby.
 
 ##Instantiating and Using Parsers
-If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class. To parse input, create an instance and call its `parse` method with a string. The parser will return the syntax tree of the match or `nil` if there is a failure.
+If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class.
+To parse input, create an instance and call its `parse` method with a string.
+The parser will return the syntax tree of the match or `nil` if there is a failure.
+Note that by default, the parser will fail unless *all* input is consumed.
 
     Treetop.load "arithmetic"
     
@@ -26,3 +29,77 @@ If a grammar by the name of `Foo` is defined, the compiled Ruby source will defi
     else
       puts 'failure'
     end
+
+##Parser Options
+A Treetop parser has several options you may set.
+Some are settable permanently by methods on the parser, but all may be passed in as options to the `parse` method.
+
+    parser = ArithmeticParser.new
+    input = 'x = 2; y = x+3;'
+
+    # Temporarily override an option:
+    result1 = parser.parse(input, :consume_all_input => false)
+    puts "consumed #{parser.index} characters"
+
+    parser.consume_all_input = false
+    result1 = parser.parse(input)
+    puts "consumed #{parser.index} characters"
+
+    # Continue the parse with the next character:
+    result2 = parser.parse(input, :index => parser.index)
+
+    # Parse, but match rule `variable` instead of the normal root rule:
+    parser.parse(input, :root => :variable)
+    parser.root = :variable	# Permanent setting
+
+If you have a statement-oriented language, you can save memory by parsing just one statement at a time,
+and discarding the parse tree after each statement.
+
+
+##Learning From Failure
+If a parse fails, it returns nil. In this case, you can ask the parser for an explanation.
+The failure reasons include the terminal nodes which were tried at the furthermost point the parse reached.
+
+    parser = ArithmeticParser.new
+    result = parser.parse('4+=3')
+
+    if !result
+      puts parser.failure_reason
+      puts parser.failure_line
+      puts parser.failure_column
+    end
+
+    =>
+    Expected one of (, - at line 1, column 3 (byte 3) after +
+    1
+    3
+
+
+##Using Parse Results
+Please don't try to walk down the syntax tree yourself, and please don't use the tree as your own convenient data structure.
+It contains many more nodes than your application needs, often even more than one for every character of input.
+
+    parser = ArithmeticParser.new
+    p parser.parse('2+3')
+
+    =>
+    SyntaxNode+Additive1 offset=0, "2+3" (multitive):
+      SyntaxNode+Multitive1 offset=0, "2" (primary):
+        SyntaxNode+Number0 offset=0, "2":
+          SyntaxNode offset=0, ""
+          SyntaxNode offset=0, "2"
+          SyntaxNode offset=1, ""
+        SyntaxNode offset=1, ""
+      SyntaxNode offset=1, "+3":
+        SyntaxNode+Additive0 offset=1, "+3" (multitive):
+          SyntaxNode offset=1, "+"
+          SyntaxNode+Multitive1 offset=2, "3" (primary):
+            SyntaxNode+Number0 offset=2, "3":
+              SyntaxNode offset=2, ""
+              SyntaxNode offset=2, "3"
+              SyntaxNode offset=3, ""
+            SyntaxNode offset=3, ""
+
+Instead, add methods to the root rule which return the information you require in a sensible form.
+Each rule can call its sub-rules, and this method of walking the syntax tree is much preferable to
+attempting to walk it from the outside.

data/examples/lambda_calculus/arithmetic.treetop

@@ -4,7 +4,10 @@ grammar Arithmetic
   end
   
   rule comparative
-    operand_1:additive space operator:equality_op space operand_2:additive <BinaryOperation>
+    head:additive
+    tail:(
+      space operator:equality_op
+      space operand:additive)* <BinaryOperation>
   end
   
   rule equality_op
@@ -16,11 +19,10 @@ grammar Arithmetic
   end
   
   rule additive
-    operand_1:multitive
-    space operator:additive_op space
-    operand_2:additive <BinaryOperation>
-    /
-    multitive
+    head:multitive
+    tail:(
+      space operator:additive_op
+      space operand:multitive)* <BinaryOperation>
   end
   
   rule additive_op
@@ -38,11 +40,10 @@ grammar Arithmetic
   end
 
   rule multitive
-    operand_1:primary
-    space operator:multitive_op space
-    operand_2:multitive <BinaryOperation>
-    /
-    primary
+    head:primary
+    tail:(
+      space operator:multitive_op
+      space operand:primary)* <BinaryOperation>
   end
   
   rule multitive_op
@@ -94,4 +95,4 @@ grammar Arithmetic
   rule space
     ' '*
   end
-end
+end

data/examples/lambda_calculus/arithmetic_node_classes.rb

@@ -1,7 +1,9 @@
 module Arithmetic
   class BinaryOperation < Treetop::Runtime::SyntaxNode
     def eval(env={})
-      operator.apply(operand_1.eval(env), operand_2.eval(env))      
+      tail.elements.inject(head.eval(env)) do |value, element|
+        element.operator.apply(value, element.operand.eval(env))
+      end
     end
   end
-end
+end

data/examples/lambda_calculus/arithmetic_test.rb

@@ -43,6 +43,10 @@ class ArithmeticParserTest < Test::Unit::TestCase
     assert_equal 11, parse('1 + 2 * 3 + 4').eval
   end
   
+  def test_left_to_right
+    assert_equal 2, parse('5 - 2 - 1').eval
+  end
+  
   def test_parentheses
     assert_equal 25, parse('(5 + x) * (10 - y)').eval('x' => 0, 'y' => 5)
   end

data/lib/treetop/runtime/compiled_parser.rb

@@ -15,8 +15,9 @@ module Treetop
       def parse(input, options = {})
         prepare_to_parse(input)
         @index = options[:index] if options[:index]
-        result = send("_nt_#{root}")
-        return nil if (consume_all_input? && index != input.size)
+        result = send("_nt_#{options[:root] || root}")
+        should_consume_all = options.include?(:consume_all_input) ? options[:consume_all_input] : consume_all_input?
+        return nil if (should_consume_all && index != input.size)
         return SyntaxNode.new(input, index...(index + 1)) if result == true
         return result
       end

data/lib/treetop/version.rb

@@ -2,7 +2,7 @@ module Treetop #:nodoc:
   module VERSION #:nodoc:
     MAJOR = 1
     MINOR = 4
-    TINY  = 8
+    TINY  = 9
 
     STRING = [MAJOR, MINOR, TINY].join('.')
   end

data/spec/runtime/compiled_parser_spec.rb

@@ -3,45 +3,67 @@ require 'spec_helper'
 module CompiledParserSpec
   describe Runtime::CompiledParser, "for a grammar with two rules" do
     attr_reader :parser
-    
+
     testing_grammar %{
       grammar TwoRules
         rule a
           'a'
         end
-        
+
         rule b
           'b'
         end
       end
     }
-    
+
     before do
       @parser = parser_class_under_test.new
     end
-  
+
     it "allows its root to be specified" do
       parser.parse('a').should_not be_nil
       parser.parse('b').should be_nil
-      
+
+      # Check that the temporary-override works:
+      parser.parse('b', :root => :b).should_not be_nil
+      parser.parse('a', :root => :b).should be_nil
+
+      # Check that the temporary-override isn't sticky:
+      parser.parse('a').should_not be_nil
+
+      # Try a permanent override:
       parser.root = :b
       parser.parse('b').should_not be_nil
       parser.parse('a').should be_nil
     end
-    
+
     it "allows the requirement that all input be consumed to be disabled" do
       parser.parse('ab').should be_nil
+
+      # Try a temporary override, and check it's not sticky:
+      result = parser.parse('ab', :consume_all_input => false)
+      result.should_not be_nil
+      result.interval.should == (0...1)
+      parser.parse('ab').should be_nil
+
+      # Now a permanent override:
       parser.consume_all_input = false
       result = parser.parse('ab')
       result.should_not be_nil
       result.interval.should == (0...1)
     end
-    
+
     it "allows input to be parsed at a given index" do
       parser.parse('ba').should be_nil
       parser.parse('ba', :index => 1).should_not be_nil
+      # Check that the index defaults again to zero:
+      parser.parse('a').should_not be_nil
+
+      result = parser.parse('ba', :consume_all_input => false, :index => 1)
+      result.should_not be_nil
+      result.interval.should == (1...2)
     end
-    
+
   end
 
   describe Runtime::CompiledParser, "for a grammar with a choice between terminals" do
@@ -81,7 +103,7 @@ module CompiledParserSpec
     before do
       @parser = parser_class_under_test.new
     end
-    
+
     it "is reset between parses" do
       parser.parse('ac')
       terminal_failures = parser.terminal_failures

metadata

@@ -1,12 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: treetop
 version: !ruby/object:Gem::Version 
+  hash: 21
   prerelease: false
   segments: 
   - 1
   - 4
-  - 8
-  version: 1.4.8
+  - 9
+  version: 1.4.9
 platform: ruby
 authors: 
 - Nathan Sobo
@@ -14,7 +15,7 @@ autorequire: treetop
 bindir: bin
 cert_chain: []
 
-date: 2010-05-31 00:00:00 +10:00
+date: 2010-11-16 00:00:00 +11:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -25,6 +26,7 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
+        hash: 17
         segments: 
         - 0
         - 3
@@ -139,6 +141,19 @@ files:
 - doc/index.markdown
 - doc/pitfalls_and_advanced_techniques.markdown
 - doc/semantic_interpretation.markdown
+- doc/site/contribute.html
+- doc/site/images/bottom_background.png
+- doc/site/images/middle_background.png
+- doc/site/images/paren_language_output.png
+- doc/site/images/pivotal.gif
+- doc/site/images/top_background.png
+- doc/site/index.html
+- doc/site/pitfalls_and_advanced_techniques.html
+- doc/site/robots.txt
+- doc/site/screen.css
+- doc/site/semantic_interpretation.html
+- doc/site/syntactic_recognition.html
+- doc/site/using_in_ruby.html
 - doc/site.rb
 - doc/sitegen.rb
 - doc/syntactic_recognition.markdown
@@ -167,6 +182,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"
@@ -175,6 +191,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"