treetop 1.1.1 → 1.1.2

data/doc/syntactic_recognition.markdown

@@ -0,0 +1,103 @@
+#Syntactic Recognition 
+Treetop grammars are written in a custom language based on parsing expression grammars. Literature on the subject of parsing expression grammars is useful in writing Treetop grammars.
+
+#Grammar Structure
+Treetop grammars look like this:
+
+    grammar GrammarName
+      rule rule_name
+        ...
+      end
+      
+      rule rule_name
+        ...
+      end
+      
+      ...
+    end
+
+The main keywords are:
+
+* `grammar` : This introduces a new grammar. It is followed by a constant name to which the grammar will be bound when it is loaded.
+
+* `rule` : This defines a parsing rule within the grammar. It is followed by a name by which this rule can be referenced within other rules. It is then followed by a parsing expression defining the rule.
+
+#Parsing Expressions
+Each rule associates a name with a _parsing expression_. Parsing expressions are a generalization of vanilla regular expressions. Their key feature is the ability to reference other expressions in the grammar by name.
+
+##Atomic Expressions
+###Terminal Symbols
+####Strings
+Strings are surrounded in double or single quotes and must be matched exactly.
+
+* `"foo"`
+* `'foo'`
+  
+####Character Classes
+Character classes are surrounded by brackets. Their semantics are identical to those used in Ruby's regular expressions.
+
+* `[a-zA-Z]`
+* `[0-9]`
+
+####The Anything Symbol
+The anything symbol is represented by a dot (`.`) and matches any single character.
+
+###Nonterminal Symbols
+Nonterminal symbols are unquoted references to other named rules. They are equivalent to an inline substitution of the named expression.
+
+    rule foo
+      "the dog " bar
+    end
+    
+    rule bar
+      "jumped"
+    end
+
+The above grammar is equivalent to:
+
+    rule foo
+      "the dog jumped"
+    end
+
+##Composite Expressions
+###Ordered Choice
+Parsers attempt to match ordered choices in left-to-right order, and stop after the first successful match.
+
+    "foobar" / "foo" / "bar"
+    
+Note that if `"foo"` in the above expression came first, `"foobar"` would never be matched.
+
+###Sequences
+
+Sequences are a space-separated list of parsing expressions. They have higher precedence than choices, so choices must be parenthesized to be used as the elements of a sequence. 
+
+    "foo" "bar" ("baz" / "bop")
+
+###Repetitions
+####Zero or More
+Parsers will greedily match an expression zero or more times if it is followed by the star (`*`) symbol.
+
+* `'foo'*` matches the empty string, `"foo"`, `"foofoo"`, etc.
+
+####One or More
+Parsers will greedily match an expression one or more times if it is followed by the star (`+`) symbol.
+
+* `'foo'+` does not match the empty string, but matches `"foo"`, `"foofoo"`, etc.
+
+###Optional Expressions
+An expression can be declared optional by following it with a question mark (`?`).
+
+* `'foo'?` matches `"foo"` or the empty string.
+
+##Lookahead Assertions
+Lookahead assertions can be used to give parsing expressions a limited degree of context-sensitivity. The parser will look ahead into the buffer and attempt to match an expression without consuming input.
+
+###Positive Lookahead Assertion
+Preceding an expression with an ampersand `(&)` indicates that it must match, but no input will be consumed in the process of determining whether this is true.
+
+* `"foo" &"bar"` matches `"foobar"` but only consumes up to the end `"foo"`. It will not match `"foobaz"`.
+
+###Negative Lookahead Assertion
+Preceding an expression with a bang `(!)` indicates that the expression must not match, but no input will be consumed in the process of determining whether this is true.
+
+* `"foo" !"bar"` matches `"foobaz"` but only consumes up to the end `"foo"`. It will not match `"foobar"`.

data/doc/using_in_ruby.markdown

@@ -0,0 +1,17 @@
+#Using Treetop Grammars in Ruby
+##Using the Command Line Compiler
+You can `.treetop` files into Ruby source code with the `tt` command line script. `tt` takes an list of files with a `.treetop` extension and compiles them into `.rb` files of the same name. You can then `require` these files like any other Ruby script. Alternately, you can supply just one `.treetop` file and a `-o` flag to name specify the name of the output file. Improvements to this compilation script are welcome.
+
+    tt foo.treetop bar.treetop
+    tt foo.treetop -o foogrammar.rb
+
+##Loading A Grammar Directly
+The `load_grammar` method takes the path to a `.treetop` file (where the extension is optional), and automatically compiles and evaluates the Ruby source. If you are getting errors in methods you define on the syntax tree, try using the command line compiler for better stack trace feedback. The need to do this is being addressed.
+
+##Instantiating and Using Parsers
+If a grammar by the name of `Foo` is defined, the compiled Ruby source will define a `FooParser` class. To parse input, create an instance and call its `parse` method with a string.
+
+    load_grammar "arithmetic"
+    
+    parser = ArithmeticParser.new
+    puts parser.parse('1+1').success?

data/examples/lambda_calculus/lambda_calculus.rb

@@ -0,0 +1,751 @@
+module LambdaCalculus
+  include ::Treetop::Runtime
+  
+  def root
+    result = _nt_program
+    if index == input.size
+      return result
+    else
+      return ParseFailure.new(input, index, result.nested_failures)
+    end
+  end
+  
+  include Arithmetic
+  
+  module Program0
+    def space
+      elements[1]
+    end
+    
+    def expression
+      elements[2]
+    end
+  end
+  
+  module Program1
+    def expression
+      elements[0]
+    end
+    
+    def more_expressions
+      elements[1]
+    end
+  end
+  
+  module Program2
+    def eval(env={})
+      env = env.clone
+      last_eval = nil
+      expressions.each do |exp|
+        last_eval = exp.eval(env)
+      end
+      last_eval
+    end
+    
+    def expressions
+      [expression] + more_expressions.elements.map {|elt| elt.expression}
+    end
+  end
+  
+  def _nt_program
+    start_index = index
+    cached = node_cache[:program][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    r1 = _nt_expression
+    s0 << r1
+    if r1.success?
+      s2, nr2, i2 = [], [], index
+      loop do
+        i3, s3, nr3 = index, [], []
+        r4 = parse_terminal(';', SyntaxNode)
+        s3 << r4
+        if r4.success?
+          r5 = _nt_space
+          s3 << r5
+          if r5.success?
+            r6 = _nt_expression
+            s3 << r6
+          end
+        end
+        if s3.last.success?
+          r3 = (SyntaxNode).new(input, i3...index, s3)
+          r3.extend(Program0)
+        else
+          self.index = i3
+          r3 = ParseFailure.new(input, i3, s3)
+        end
+        nr2 << r3
+        if r3.success?
+          s2 << r3
+        else
+          break
+        end
+      end
+      r2 = SyntaxNode.new(input, i2...index, s2, nr2)
+      s0 << r2
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Program1)
+      r0.extend(Program2)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:program][start_index] = r0
+    
+    return r0
+  end
+  
+  def _nt_expression
+    start_index = index
+    cached = node_cache[:expression][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, nr0 = index, []
+    r1 = _nt_definition
+    nr0 << r1
+    if r1.success?
+      r0 = r1
+      r1.update_nested_results(nr0)
+    else
+      r2 = _nt_conditional
+      nr0 << r2
+      if r2.success?
+        r0 = r2
+        r2.update_nested_results(nr0)
+      else
+        r3 = _nt_application
+        nr0 << r3
+        if r3.success?
+          r0 = r3
+          r3.update_nested_results(nr0)
+        else
+          r4 = _nt_function
+          nr0 << r4
+          if r4.success?
+            r0 = r4
+            r4.update_nested_results(nr0)
+          else
+            r5 = super
+            nr0 << r5
+            if r5.success?
+              r0 = r5
+              r5.update_nested_results(nr0)
+            else
+              self.index = i0
+              r0 = ParseFailure.new(input, i0, nr0)
+            end
+          end
+        end
+      end
+    end
+    
+    node_cache[:expression][start_index] = r0
+    
+    return r0
+  end
+  
+  module Definition0
+    def space
+      elements[1]
+    end
+    
+    def variable
+      elements[2]
+    end
+    
+    def space
+      elements[3]
+    end
+    
+    def expression
+      elements[4]
+    end
+  end
+  
+  module Definition1
+    def eval(env)
+      env[variable.name] = expression.eval(env)
+    end
+  end
+  
+  def _nt_definition
+    start_index = index
+    cached = node_cache[:definition][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    r1 = parse_terminal('def', SyntaxNode)
+    s0 << r1
+    if r1.success?
+      r2 = _nt_space
+      s0 << r2
+      if r2.success?
+        r3 = _nt_variable
+        s0 << r3
+        if r3.success?
+          r4 = _nt_space
+          s0 << r4
+          if r4.success?
+            r5 = _nt_expression
+            s0 << r5
+          end
+        end
+      end
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Definition0)
+      r0.extend(Definition1)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:definition][start_index] = r0
+    
+    return r0
+  end
+  
+  module Conditional0
+    def space
+      elements[1]
+    end
+    
+    def space
+      elements[3]
+    end
+    
+    def condition
+      elements[4]
+    end
+    
+    def space
+      elements[5]
+    end
+    
+    def space
+      elements[7]
+    end
+    
+    def true_case
+      elements[8]
+    end
+    
+    def space
+      elements[9]
+    end
+    
+    def space
+      elements[11]
+    end
+    
+    def false_case
+      elements[12]
+    end
+  end
+  
+  module Conditional1
+    def eval(env)
+      if condition.eval(env)
+        true_case.eval(env)
+      else
+        false_case.eval(env)
+      end
+    end
+  end
+  
+  def _nt_conditional
+    start_index = index
+    cached = node_cache[:conditional][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    r1 = parse_terminal('if', SyntaxNode)
+    s0 << r1
+    if r1.success?
+      r2 = _nt_space
+      s0 << r2
+      if r2.success?
+        r3 = parse_terminal('(', SyntaxNode)
+        s0 << r3
+        if r3.success?
+          r4 = _nt_space
+          s0 << r4
+          if r4.success?
+            r5 = _nt_expression
+            s0 << r5
+            if r5.success?
+              r6 = _nt_space
+              s0 << r6
+              if r6.success?
+                r7 = parse_terminal(')', SyntaxNode)
+                s0 << r7
+                if r7.success?
+                  r8 = _nt_space
+                  s0 << r8
+                  if r8.success?
+                    r9 = _nt_expression
+                    s0 << r9
+                    if r9.success?
+                      r10 = _nt_space
+                      s0 << r10
+                      if r10.success?
+                        r11 = parse_terminal('else', SyntaxNode)
+                        s0 << r11
+                        if r11.success?
+                          r12 = _nt_space
+                          s0 << r12
+                          if r12.success?
+                            r13 = _nt_expression
+                            s0 << r13
+                          end
+                        end
+                      end
+                    end
+                  end
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Conditional0)
+      r0.extend(Conditional1)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:conditional][start_index] = r0
+    
+    return r0
+  end
+  
+  def _nt_primary
+    start_index = index
+    cached = node_cache[:primary][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, nr0 = index, []
+    r1 = _nt_application
+    nr0 << r1
+    if r1.success?
+      r0 = r1
+      r1.update_nested_results(nr0)
+    else
+      r2 = super
+      nr0 << r2
+      if r2.success?
+        r0 = r2
+        r2.update_nested_results(nr0)
+      else
+        self.index = i0
+        r0 = ParseFailure.new(input, i0, nr0)
+      end
+    end
+    
+    node_cache[:primary][start_index] = r0
+    
+    return r0
+  end
+  
+  module Application0
+    def operator
+      elements[0]
+    end
+    
+    def space
+      elements[1]
+    end
+    
+    def expression
+      elements[2]
+    end
+  end
+  
+  module Application1
+    def eval(env={})
+      left_associative_apply(operator.eval(env), env)
+    end
+    
+    def left_associative_apply(operator, env)
+      if expression.instance_of?(Application)
+        expression.left_associative_apply(operator.apply(expression.operator.eval(env)), env)
+      else
+        operator.apply(expression.eval(env))
+      end
+    end
+    
+    def to_s(env={})
+      operator.to_s(env) + ' ' + expression.to_s(env)
+    end
+  end
+  
+  def _nt_application
+    start_index = index
+    cached = node_cache[:application][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    r1 = _nt_operator
+    s0 << r1
+    if r1.success?
+      r2 = _nt_space
+      s0 << r2
+      if r2.success?
+        r3 = _nt_expression
+        s0 << r3
+      end
+    end
+    if s0.last.success?
+      r0 = (Application).new(input, i0...index, s0)
+      r0.extend(Application0)
+      r0.extend(Application1)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:application][start_index] = r0
+    
+    return r0
+  end
+  
+  def _nt_operator
+    start_index = index
+    cached = node_cache[:operator][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, nr0 = index, []
+    r1 = _nt_function
+    nr0 << r1
+    if r1.success?
+      r0 = r1
+      r1.update_nested_results(nr0)
+    else
+      r2 = _nt_variable
+      nr0 << r2
+      if r2.success?
+        r0 = r2
+        r2.update_nested_results(nr0)
+      else
+        self.index = i0
+        r0 = ParseFailure.new(input, i0, nr0)
+      end
+    end
+    
+    node_cache[:operator][start_index] = r0
+    
+    return r0
+  end
+  
+  def _nt_non_application
+    start_index = index
+    cached = node_cache[:non_application][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, nr0 = index, []
+    r1 = _nt_function
+    nr0 << r1
+    if r1.success?
+      r0 = r1
+      r1.update_nested_results(nr0)
+    else
+      r2 = _nt_variable
+      nr0 << r2
+      if r2.success?
+        r0 = r2
+        r2.update_nested_results(nr0)
+      else
+        self.index = i0
+        r0 = ParseFailure.new(input, i0, nr0)
+      end
+    end
+    
+    node_cache[:non_application][start_index] = r0
+    
+    return r0
+  end
+  
+  module Function0
+    def param
+      elements[1]
+    end
+    
+    def body
+      elements[3]
+    end
+    
+  end
+  
+  module Function1
+    class Closure
+      attr_reader :env, :function
+      
+      def initialize(function, env)
+        @function = function
+        @env = env
+      end
+    
+      def apply(arg)
+        function.body.eval(function.param.bind(arg, env))
+      end
+    
+      def to_s(other_env={})
+        "\\#{function.param.to_s}(#{function.body.to_s(other_env.merge(env))})"
+      end
+    end
+    
+    def eval(env={})
+      Closure.new(self, env)
+    end
+    
+    def to_s(env={})
+      eval(env).to_s
+    end
+  end
+  
+  def _nt_function
+    start_index = index
+    cached = node_cache[:function][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    r1 = parse_terminal('\\', SyntaxNode)
+    s0 << r1
+    if r1.success?
+      r2 = _nt_variable
+      s0 << r2
+      if r2.success?
+        r3 = parse_terminal('(', SyntaxNode)
+        s0 << r3
+        if r3.success?
+          r4 = _nt_expression
+          s0 << r4
+          if r4.success?
+            r5 = parse_terminal(')', SyntaxNode)
+            s0 << r5
+          end
+        end
+      end
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Function0)
+      r0.extend(Function1)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:function][start_index] = r0
+    
+    return r0
+  end
+  
+  module Variable0
+    def bind(value, env)
+      env.merge(name => value)
+    end
+
+    def to_s(env={})
+      env.has_key?(name) ? env[name].to_s : name
+  end
+  end
+  
+  module Variable1
+  end
+  
+  def _nt_variable
+    start_index = index
+    cached = node_cache[:variable][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    i1 = index
+    r2 = _nt_keyword
+    if r2.success?
+      r1 = ParseFailure.new(input, i1, r2.nested_failures)
+    else
+      self.index = i1
+      r1 = SyntaxNode.new(input, index...index, r2.nested_failures)
+    end
+    s0 << r1
+    if r1.success?
+      r3 = super
+      r3.extend(Variable0)
+      s0 << r3
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Variable1)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:variable][start_index] = r0
+    
+    return r0
+  end
+  
+  module Keyword0
+  end
+  
+  def _nt_keyword
+    start_index = index
+    cached = node_cache[:keyword][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    i1, nr1 = index, []
+    r2 = parse_terminal('if', SyntaxNode)
+    nr1 << r2
+    if r2.success?
+      r1 = r2
+      r2.update_nested_results(nr1)
+    else
+      r3 = parse_terminal('else', SyntaxNode)
+      nr1 << r3
+      if r3.success?
+        r1 = r3
+        r3.update_nested_results(nr1)
+      else
+        self.index = i1
+        r1 = ParseFailure.new(input, i1, nr1)
+      end
+    end
+    s0 << r1
+    if r1.success?
+      i4 = index
+      r5 = _nt_non_space_char
+      if r5.success?
+        r4 = ParseFailure.new(input, i4, r5.nested_failures)
+      else
+        self.index = i4
+        r4 = SyntaxNode.new(input, index...index, r5.nested_failures)
+      end
+      s0 << r4
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(Keyword0)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:keyword][start_index] = r0
+    
+    return r0
+  end
+  
+  module NonSpaceChar0
+  end
+  
+  def _nt_non_space_char
+    start_index = index
+    cached = node_cache[:non_space_char][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    i0, s0, nr0 = index, [], []
+    i1 = index
+    r2 = parse_char_class(/[ \n]/, ' \n', SyntaxNode)
+    if r2.success?
+      r1 = ParseFailure.new(input, i1, r2.nested_failures)
+    else
+      self.index = i1
+      r1 = SyntaxNode.new(input, index...index, r2.nested_failures)
+    end
+    s0 << r1
+    if r1.success?
+      r3 = parse_anything(SyntaxNode)
+      s0 << r3
+    end
+    if s0.last.success?
+      r0 = (SyntaxNode).new(input, i0...index, s0)
+      r0.extend(NonSpaceChar0)
+    else
+      self.index = i0
+      r0 = ParseFailure.new(input, i0, s0)
+    end
+    
+    node_cache[:non_space_char][start_index] = r0
+    
+    return r0
+  end
+  
+  def _nt_space
+    start_index = index
+    cached = node_cache[:space][index]
+    if cached
+      @index = cached.interval.end
+      return cached
+    end
+    
+    s0, nr0, i0 = [], [], index
+    loop do
+      r1 = parse_char_class(/[ \n]/, ' \n', SyntaxNode)
+      nr0 << r1
+      if r1.success?
+        s0 << r1
+      else
+        break
+      end
+    end
+    r0 = SyntaxNode.new(input, i0...index, s0, nr0)
+    
+    node_cache[:space][start_index] = r0
+    
+    return r0
+  end
+  
+end
+
+class LambdaCalculusParser < ::Treetop::Runtime::CompiledParser
+  include LambdaCalculus
+end