treetop 1.4.5 → 1.4.7

data/README.md

@@ -1,3 +1,9 @@
+Support
+=======
+
+Support for Treetop is provided through the mailing list you can join or browse here:
+http://groups.google.com/group/treetop-dev 
+
 Tutorial
 ========
 Languages can be split into two components, their *syntax* and their *semantics*. It's your understanding of English syntax that tells you the stream of words "Sleep furiously green ideas colorless" is not a valid sentence. Semantics is deeper. Even if we rearrange the above sentence to be "Colorless green ideas sleep furiously", which is syntactically correct, it remains nonsensical on a semantic level. With Treetop, you'll be dealing with languages that are much simpler than English, but these basic concepts apply. Your programs will need to address both the syntax and the semantics of the languages they interpret.
@@ -112,7 +118,16 @@ Any item in a rule may be followed by a '+' or a '*' character, signifying one-o
       end
     end
 
-The 'a'* will always eat up any 'a's that follow, and the subsequent 'a' will find none there, so the whole rule will fail. You might need to use lookahead to avoid matching too much.
+The 'a'* will always eat up any 'a's that follow, and the subsequent 'a' will find none there, so the whole rule will fail. You might need to use lookahead to avoid matching too much. Alternatively, you can use an occurrence range:
+
+    # toogreedy.treetop
+    grammar TooGreedy
+      rule two_to_four_as
+      	'a' 2..4
+      end
+    end
+
+In an occurrence range, you may omit either the minimum count or the maximum count, so that "0.. " works like "*" and "1.. " works like '+'.
 
 Negative Lookahead
 ------------------
@@ -141,24 +156,33 @@ Positive lookahead
 
 Sometimes you want an item to match, but only if the *following* text would match some pattern. You don't want to consume that following text, but if it's not there, you want this rule to fail. You can append a positive lookahead like this to a rule by appending the lookahead rule preceeded by an & character.
 
+Semantic predicates
+-------------------
+
+Warning: This is an advanced feature. You need to understand the way a packrat parser operates to use it correctly. The result of computing a rule containing a semantic predicate will be memoized, even if the same rule, applied later at the same location in the input, would work differently due to a semantic predicate returning a different value. If you don't understand the previous sentence yet still use this feature, you're on your own, so test carefully!
+
+Sometimes, you need to run external Ruby code to decide whether this syntax rule should continue or should fail. You can do this using either positive or negative semantic predicates. These are Ruby code blocks (lambdas) which are called when the parser reaches that location. For this rule to succeed, the value must be true for a positive predicate (a block like &{ ... }), or false for a negative predicate (a block like !{ ... }).
+
+The block is called with one argument, the array containing the preceding syntax nodes in the current sequence. Within the block, you cannot use node names or labels for the preceding nodes, as the node for the current rule does not yet exist. You must refer to preceding nodes using their position in the sequence.
 
+    grammar Keywords
+      rule sequence_of_reserved_and_nonreserved_words
+      	( reserved / word )*
+      end
+
+      rule reserved
+        word &{ |s| symbol_reserved?(s[0].text_value) }
+      end
+
+      rule word
+        ([a-zA-Z]+ [ \t]+) 
+      end
+    end
+
+One case where it is always safe to use a semantic predicate is to invoke the Ruby debugger, but don't forget to return true so the rule succeeds! Assuming you have required the 'ruby-debug' module somewhere, it looks like this:
+
+      rule problems
+        word &{ |s| debugger; true }
+      end
 
-Features to cover in the talk
-=============================
-
-* Treetop files
-* Grammar definition
-* Rules
-* Loading a grammar
-* Compiling a grammar with the `tt` command
-* Accessing a parser for the grammar from Ruby
-* Parsing Expressions of all kinds
-? Left recursion and factorization
-  - Here I can talk about function application, discussing how the operator
-    could be an arbitrary expression
-* Inline node class eval blocks
-* Node class declarations
-* Labels
-* Use of super within within labels
-* Grammar composition with include
-* Use of super with grammar composition
+When the debugger stops here, you can inspect the contents of the SyntaxNode for "word" by looking at s[0], and the stack trace will show how you got there.

data/lib/treetop/compiler/metagrammar.rb

@@ -575,7 +575,7 @@ module Treetop
         def declarations
           [head] + tail
         end
-    
+
         def tail
           super.elements.map { |elt| elt.declaration }
         end
@@ -955,11 +955,11 @@ module Treetop
         def alternatives
           [head] + tail
         end
-        
+
         def tail
           super.elements.map {|elt| elt.alternative}
         end
-        
+
         def inline_modules
           (alternatives.map {|alt| alt.inline_modules }).flatten
         end
@@ -1076,17 +1076,17 @@ module Treetop
         def sequence_elements
           [head] + tail
         end
-  
+
         def tail
           super.elements.map {|elt| elt.labeled_sequence_primary }
         end
-    
+
         def inline_modules
           (sequence_elements.map {|elt| elt.inline_modules}).flatten +
           [sequence_element_accessor_module] +
           node_class_declarations.inline_modules
         end
-    
+
         def inline_module_name
           node_class_declarations.inline_module_name
         end
@@ -1199,15 +1199,15 @@ module Treetop
         def compile(address, builder, parent_expression=nil)
           prefix.compile(address, builder, self)
         end
-    
+
         def prefixed_expression
           atomic
         end
-    
+
         def inline_modules
           atomic.inline_modules
         end
-    
+
         def inline_module_name
           nil
         end
@@ -1253,19 +1253,19 @@ module Treetop
         def compile(address, builder, parent_expression=nil)
           suffix.compile(address, builder, self)
         end
-    
+
         def optional_expression
           atomic
         end
-    
+
         def node_class_name
           node_class_declarations.node_class_name
         end
-    
+
         def inline_modules
           atomic.inline_modules + node_class_declarations.inline_modules
         end
-    
+
         def inline_module_name
           node_class_declarations.inline_module_name
         end
@@ -1285,15 +1285,15 @@ module Treetop
         def compile(address, builder, parent_expression=nil)
           atomic.compile(address, builder, self)
         end
-    
+
         def node_class_name
           node_class_declarations.node_class_name
         end
-    
+
         def inline_modules
           atomic.inline_modules + node_class_declarations.inline_modules
         end
-    
+
         def inline_module_name
           node_class_declarations.inline_module_name
         end
@@ -1422,11 +1422,11 @@ module Treetop
         def compile(lexical_address, builder)
           sequence_primary.compile(lexical_address, builder)
         end
-    
+
         def inline_modules
           sequence_primary.inline_modules
         end
-    
+
         def label_name
           if label.name
             label.name
@@ -1585,15 +1585,15 @@ module Treetop
         def compile(lexical_address, builder)
           prefix.compile(lexical_address, builder, self)
         end
-    
+
         def prefixed_expression
           elements[1]
         end
-    
+
         def inline_modules
           atomic.inline_modules
         end
-    
+
         def inline_module_name
           nil
         end
@@ -1635,15 +1635,15 @@ module Treetop
         def compile(lexical_address, builder)
           suffix.compile(lexical_address, builder, self)
         end
-    
+
         def node_class_name
           nil
         end
-    
+
         def inline_modules
           atomic.inline_modules
         end
-    
+
         def inline_module_name
           nil
         end
@@ -1808,15 +1808,15 @@ module Treetop
         def node_class_name
           node_class_expression.node_class_name
         end
-    
+
         def inline_modules
           trailing_inline_module.inline_modules
         end
-    
+
         def inline_module
           trailing_inline_module.inline_module
         end
-    
+
         def inline_module_name
           inline_module.module_name if inline_module
         end
@@ -1886,8 +1886,13 @@ module Treetop
           if r2
             r0 = r2
           else
-            @index = i0
-            r0 = nil
+            r3 = _nt_occurrence_range
+            if r3
+              r0 = r3
+            else
+              @index = i0
+              r0 = nil
+            end
           end
         end
 
@@ -1896,6 +1901,94 @@ module Treetop
         r0
       end
 
+      module OccurrenceRange0
+        def min
+          elements[1]
+        end
+
+        def max
+          elements[3]
+        end
+      end
+
+      def _nt_occurrence_range
+        start_index = index
+        if node_cache[:occurrence_range].has_key?(index)
+          cached = node_cache[:occurrence_range][index]
+          if cached
+            cached = SyntaxNode.new(input, index...(index + 1)) if cached == true
+            @index = cached.interval.end
+          end
+          return cached
+        end
+
+        i0, s0 = index, []
+        r2 = _nt_space
+        if r2
+          r1 = r2
+        else
+          r1 = instantiate_node(SyntaxNode,input, index...index)
+        end
+        s0 << r1
+        if r1
+          s3, i3 = [], index
+          loop do
+            if has_terminal?('\G[0-9]', true, index)
+              r4 = true
+              @index += 1
+            else
+              r4 = nil
+            end
+            if r4
+              s3 << r4
+            else
+              break
+            end
+          end
+          r3 = instantiate_node(SyntaxNode,input, i3...index, s3)
+          s0 << r3
+          if r3
+            if has_terminal?('..', false, index)
+              r5 = instantiate_node(SyntaxNode,input, index...(index + 2))
+              @index += 2
+            else
+              terminal_parse_failure('..')
+              r5 = nil
+            end
+            s0 << r5
+            if r5
+              s6, i6 = [], index
+              loop do
+                if has_terminal?('\G[0-9]', true, index)
+                  r7 = true
+                  @index += 1
+                else
+                  r7 = nil
+                end
+                if r7
+                  s6 << r7
+                else
+                  break
+                end
+              end
+              r6 = instantiate_node(SyntaxNode,input, i6...index, s6)
+              s0 << r6
+            end
+          end
+        end
+        if s0.last
+          r0 = instantiate_node(OccurrenceRange,input, i0...index, s0)
+          r0.extend(OccurrenceRange0)
+        else
+          @index = i0
+          r0 = nil
+        end
+
+        node_cache[:occurrence_range][start_index] = r0
+
+        r0
+      end
+
       def _nt_prefix
         start_index = index
         if node_cache[:prefix].has_key?(index)
@@ -2816,7 +2909,7 @@ module Treetop
         def inline_modules
           [inline_module]
         end
-              
+
         def inline_module_name
           inline_module.module_name
         end
@@ -2826,11 +2919,11 @@ module Treetop
         def inline_modules
           []
         end
-    
+
         def inline_module
-          nil 
+          nil
         end
-    
+
         def inline_module_name
           nil
         end

data/lib/treetop/compiler/metagrammar.treetop

@@ -8,11 +8,11 @@ module Treetop
           end
         }
       end
-      
+
       rule require_statement
         prefix:space? "require" [ \t]+ [^\n\r]+ [\n\r]
       end
-      
+
       rule module_declaration
         prefix:('module' space [A-Z] alphanumeric_char* space) module_contents:(module_declaration / grammar) suffix:(space 'end') {
           def compile
@@ -34,7 +34,7 @@ module Treetop
           def declarations
             [head] + tail
           end
-      
+
           def tail
             super.elements.map { |elt| elt.declaration }
           end
@@ -45,11 +45,11 @@ module Treetop
           end
         }
       end
-      
+
       rule declaration
         parsing_rule / include_declaration
       end
-      
+
       rule include_declaration
         'include' space [A-Z] (alphanumeric_char / '::')* {
           def compile(builder)
@@ -71,11 +71,11 @@ module Treetop
           def alternatives
             [head] + tail
           end
-          
+
           def tail
             super.elements.map {|elt| elt.alternative}
           end
-          
+
           def inline_modules
             (alternatives.map {|alt| alt.inline_modules }).flatten
           end
@@ -87,17 +87,17 @@ module Treetop
           def sequence_elements
             [head] + tail
           end
-    
+
           def tail
             super.elements.map {|elt| elt.labeled_sequence_primary }
           end
-      
+
           def inline_modules
             (sequence_elements.map {|elt| elt.inline_modules}).flatten +
             [sequence_element_accessor_module] +
             node_class_declarations.inline_modules
           end
-      
+
           def inline_module_name
             node_class_declarations.inline_module_name
           end
@@ -113,15 +113,15 @@ module Treetop
           def compile(address, builder, parent_expression=nil)
             prefix.compile(address, builder, self)
           end
-      
+
           def prefixed_expression
             atomic
           end
-      
+
           def inline_modules
             atomic.inline_modules
           end
-      
+
           def inline_module_name
             nil
           end
@@ -143,19 +143,19 @@ module Treetop
           def compile(address, builder, parent_expression=nil)
             suffix.compile(address, builder, self)
           end
-      
+
           def optional_expression
             atomic
           end
-      
+
           def node_class_name
             node_class_declarations.node_class_name
           end
-      
+
           def inline_modules
             atomic.inline_modules + node_class_declarations.inline_modules
           end
-      
+
           def inline_module_name
             node_class_declarations.inline_module_name
           end
@@ -165,15 +165,15 @@ module Treetop
           def compile(address, builder, parent_expression=nil)
             atomic.compile(address, builder, self)
           end
-      
+
           def node_class_name
             node_class_declarations.node_class_name
           end
-      
+
           def inline_modules
             atomic.inline_modules + node_class_declarations.inline_modules
           end
-      
+
           def inline_module_name
             node_class_declarations.inline_module_name
           end
@@ -185,11 +185,11 @@ module Treetop
           def compile(lexical_address, builder)
             sequence_primary.compile(lexical_address, builder)
           end
-      
+
           def inline_modules
             sequence_primary.inline_modules
           end
-      
+
           def label_name
             if label.name
               label.name
@@ -201,7 +201,7 @@ module Treetop
           end
         }
       end
-  
+
       rule label
         (alpha_char alphanumeric_char*) ':' {
           def name
@@ -221,15 +221,15 @@ module Treetop
           def compile(lexical_address, builder)
             prefix.compile(lexical_address, builder, self)
           end
-      
+
           def prefixed_expression
             elements[1]
           end
-      
+
           def inline_modules
             atomic.inline_modules
           end
-      
+
           def inline_module_name
             nil
           end
@@ -251,15 +251,15 @@ module Treetop
           def compile(lexical_address, builder)
             suffix.compile(lexical_address, builder, self)
           end
-      
+
           def node_class_name
             nil
           end
-      
+
           def inline_modules
             atomic.inline_modules
           end
-      
+
           def inline_module_name
             nil
           end
@@ -267,29 +267,29 @@ module Treetop
         /
         atomic
       end
-  
+
       rule suffix
         repetition_suffix / optional_suffix
       end
-  
+
       rule optional_suffix
         '?' <Optional>
       end
-      
+
       rule node_class_declarations
         node_class_expression trailing_inline_module {
           def node_class_name
             node_class_expression.node_class_name
           end
-      
+
           def inline_modules
             trailing_inline_module.inline_modules
           end
-      
+
           def inline_module
             trailing_inline_module.inline_module
           end
-      
+
           def inline_module_name
             inline_module.module_name if inline_module
           end
@@ -297,7 +297,11 @@ module Treetop
       end
 
       rule repetition_suffix
-        '+' <OneOrMore> / '*' <ZeroOrMore>
+        '+' <OneOrMore> / '*' <ZeroOrMore> / occurrence_range
+      end
+
+      rule occurrence_range
+	space? min:([0-9])* '..' max:([0-9])* <OccurrenceRange>
       end
 
       rule prefix
@@ -327,7 +331,7 @@ module Treetop
       rule terminal
         quoted_string / character_class / anything_symbol
       end
-      
+
       rule quoted_string
         (single_quoted_string / double_quoted_string) {
           def string
@@ -335,7 +339,7 @@ module Treetop
           end
         }
       end
-      
+
       rule double_quoted_string
         '"' string:(!'"' ("\\\\" / '\"' / .))* '"' <Terminal>
       end
@@ -375,7 +379,7 @@ module Treetop
           def inline_modules
             [inline_module]
           end
-                
+
           def inline_module_name
             inline_module.module_name
           end
@@ -385,11 +389,11 @@ module Treetop
           def inline_modules
             []
           end
-      
+
           def inline_module
-            nil 
+            nil
           end
-      
+
           def inline_module_name
             nil
           end