kpeg 0.8.0 → 0.8.1

data/README.md

@@ -23,16 +23,20 @@ After that a block of ruby code can be defined that will be added into the class
 
 ### Defining literals
 
-Literals are static declarations of characters or regular expressions designed for reuse in the grammar. These can be constants or variables.
+Literals are static declarations of characters or regular expressions designed for reuse in the grammar. These can be constants or variables. Literals can take strings, regular expressions or character ranges
 
     ALPHA = /[A-Za-z]/
     DIGIT = /[0-9]/
     period = "."
+    string = "a string"
+    regex = /(regexs?)+/
+    char_range = [b-t]
 
 Literals can also accept multiple definitions
 
     vowel = "a" | "e" | "i" | "o" | "u"
     alpha = /[A-Z]/ | /[a-z]/
+    
 
 ### Defining Rules for Values
 
@@ -41,29 +45,111 @@ Before you can start parsing a string you will need to define rules that you wil
 The most basic of these rules is a string capture
   
     alpha = < /[A-Za-z]/ > { text }
+    
 
 While this looks very much like the ALPHA literal defined above it differs in one important way, the text captured by the rule defined between the < and > symbols will be set as the text variable in block that follows. You can also explicitly define the variable that you would like but only with existing rules or literals.
     
+    letter = alpha:a { a }
+    
+Additionally blocks can return true or false values based upon an expression within the block. To return true if a test passes do the following:
+
+    match_greater_than_10 = < num:n > &{ n > 10 }
+    
+To test and return a false value if the test passes do the following:
+
+    do_not_match_greater_than_10 = < num:n > !{ n > 10 }
+    
+Rules can also act like functions and take parameters. An example of this is lifted from the [Email List Validator](https://github.com/larb/email_address_validator), where an ascii value is passed in and the character is evaluated against it returning a true if it matches
+    
+    d(num) = <.> &{ text[0] == num }
+
+Rules support some regular expression syntax for matching
+ 
++ maybe ?
++ many 
++ kleene * 
++ groupings () 
+
+Examples
+
+    letters = alpha+
+    words = alpha+ space* period?
+    sentence = (letters+ | space+)+
+    
+Kpeg also allows a rule to define the acceptable number of matches in the form of a range. In regular expressions this is often denoted with syntax like {0,3}. Kpeg uses this syntax to accomplish match ranges [min, max].
+
+    matches_3_to_5_times = letter[3,5]
+    matches_3_to_any_times = letter[3,*]
+  
+    
+### Defining Actions
+
+Illustrated above in some of the examples, kpeg allows you to perform actions based upon a match that are described in block provided or in the rule definition itself.
+
     num = /[1-9][0-9]*/
     sum = < num:n1 "+" num:n2 > { n1 + n2 }
+
+### Referencing an external grammar
+
+Kpeg allows you to run a rule that is defined in an external grammar. This is useful if there is a defined set of rules that you would like to reuse in another parser. To do this, create your grammar and generate a parser using the kpeg command line tool.
+
+    kpeg literals.kpeg
+
+Once you have the generated parser, include that file into your new grammar
+
+    %{
+      require "literals.kpeg.rb"
+    }
     
-Additionally blocks can return true or false values based upon an expression within the block. To test if something is true do the following:
+Then create a variable to hold to foreign interface and pass it the class name of your parser. In this case my parser class name is Literal 
+
+    %foreign_grammer = Literal
 
-    greater_than_10 = < num:n > &{ n > 10 }
+You can then use rules defined in the foreign grammar in the local grammar file like so
+
+    sentence = (%foreign_grammer.alpha %foreign_grammer.space*)+ %foreign_grammer.period
+
+### Comments
+
+Kpeg allows comments to be added to the grammar file by using the # symbol
+
+    # This is a comment in my grammar
     
-To test for a false value do the following:
+## Generating and running your parser
+
+Before you can generate your parser you will need to define a root rule. This will be the first rule run against the string provided to the parser
 
-    not_greater_than_10 = < num:n > !{ n > 10 }
+    root = sentence
     
-Rules can also act like functions and take parameters, an example of this is can be lifted from the [Email List Validator](https://github.com/andrewvc/email_address_validator), where an ascii value is passed in and the character is evaluated against it returning a true if it matches
+To generate the parser run the kpeg command with the kpeg file(s) as an argument. This will generate a ruby file with the same name as your grammar file.
+
+    kpeg example.kpeg
     
-    d(num) = <.> &{ text[0] == num }
+Include your generated parser file into an application that you want to use the parser in and run it. Create a new instance of the parser and pass in the string you want to evaluate. When parse is called on the parser instance it will return a true if the sting is matched, or false if it doesn't. 
+
+    require "example.kpeg.rb"
     
+    parser = Example::Parser.new(string_to_evaluate)
+    parser.parse
+
+## Shortcuts and other techniques
+
+Per vito, you can get the current line or current column in the following way 
 
+    line = { current_line }
+    column = { current_column }
+    foo = line:line ... { # use line here }
+
+## Examples
+
+There are several examples available in the /examples directory. The upper parser has a readme with a step by step description of the grammar.
 
 ## Projects using kpeg
 
 [Dang](https://github.com/veganstraightedge/dang)
-[Email Address Validator](https://github.com/andrewvc/email_address_validator)
+
+[Email Address Validator](https://github.com/larb/email_address_validator)
+
 [Callisto](https://github.com/dwaite/Callisto)
-[Doodle](https://github.com/vito/doodle)
+
+[Doodle](https://github.com/vito/doodle)

data/Rakefile

@@ -19,6 +19,7 @@ task :grammar do
   gr.render(STDOUT)
 end
 
+desc "rebuild parser"
 task :parser do
   sh "ruby -Ilib bin/kpeg -o lib/kpeg/format_parser.rb -s -f lib/kpeg/format.kpeg"
 end

data/kpeg.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
   rb = Dir["lib/**/*.rb"] << "bin/kpeg"
   docs = Dir["doc/**/*"]
 
-  s.files = rb + docs + ["LICENSE", "README.md", "Rakefile", "kpeg.gemspec", "Gemfile"]
+  s.files = rb + docs + ["README.md", "Rakefile", "kpeg.gemspec", "Gemfile"]
   s.test_files    = Dir["test/**/*.rb"]
   s.bindir = "bin"
   s.executables = ["kpeg"]

data/lib/kpeg/code_generator.rb

@@ -63,7 +63,11 @@ module KPeg
 
       methods = []
 
-      @grammar.variables.each do |name, val|
+      vars = @grammar.variables.keys.sort
+
+      vars.each do |name|
+        val = @grammar.variables[name]
+
         if val.index("ast ") == 0
           unless output_node
             code << "\n"
@@ -86,22 +90,32 @@ module KPeg
         end
       end
     end
-
-    def output_op(code, op)
+    
+    def indentify(code, indent)
+      "#{"  " * indent}#{code}"
+    end
+    
+    # Default indent is 4 spaces (indent=2)
+    def output_op(code, op, indent=2)
       case op
       when Dot
-        code << "    _tmp = get_byte\n"
+        code << indentify("_tmp = get_byte\n", indent)
       when LiteralString
-        code << "    _tmp = match_string(#{op.string.dump})\n"
+        code << indentify("_tmp = match_string(#{op.string.dump})\n", indent)
       when LiteralRegexp
-        lang = op.regexp.kcode.to_s[0,1]
-        code << "    _tmp = scan(/\\A#{op.regexp}/#{lang})\n"
+        if op.regexp.respond_to?(:kcode)
+          lang = op.regexp.kcode.to_s[0,1]
+        else
+          # Let default ruby string handling figure it out
+          lang = ""
+        end
+        code << indentify("_tmp = scan(/\\A#{op.regexp}/#{lang})\n", indent)
       when CharRange
         ss = save()
         if op.start.bytesize == 1 and op.fin.bytesize == 1
-          code << "    #{ss} = self.pos\n"
-          code << "    _tmp = get_byte\n"
-          code << "    if _tmp\n"
+          code << indentify("#{ss} = self.pos\n", indent)
+          code << indentify("_tmp = get_byte\n", indent)
+          code << indentify("if _tmp\n", indent)
 
           if op.start.respond_to? :getbyte
             left  = op.start.getbyte 0
@@ -110,178 +124,178 @@ module KPeg
             left  = op.start[0]
             right = op.fin[0]
           end
-
-          code << "      unless _tmp >= #{left} and _tmp <= #{right}\n"
-          code << "        self.pos = #{ss}\n"
-          code << "        _tmp = nil\n"
-          code << "      end\n"
-          code << "    end\n"
+          
+          code << indentify("  unless _tmp >= #{left} and _tmp <= #{right}\n", indent)
+          code << indentify("    self.pos = #{ss}\n", indent)
+          code << indentify("    _tmp = nil\n", indent)
+          code << indentify("  end\n", indent)
+          code << indentify("end\n", indent)
         else
           raise "Unsupported char range - #{op.inspect}"
         end
       when Choice
         ss = save()
-        code << "\n    #{ss} = self.pos\n"
-        code << "    while true # choice\n"
+        code << "\n"
+        code << indentify("#{ss} = self.pos\n", indent)
+        code << indentify("while true # choice\n", indent)
         op.ops.each_with_index do |n,idx|
-          output_op code, n
-
-          code << "    break if _tmp\n"
-          code << "    self.pos = #{ss}\n"
+          output_op code, n, (indent+1)
+          
+          code << indentify("  break if _tmp\n", indent)
+          code << indentify("  self.pos = #{ss}\n", indent)
           if idx == op.ops.size - 1
-            code << "    break\n"
+            code << indentify("  break\n", indent)
           end
         end
-        code << "    end # end choice\n\n"
+        code << indentify("end # end choice\n\n", indent)
       when Multiple
         ss = save()
         if op.min == 0 and op.max == 1
-          code << "    #{ss} = self.pos\n"
-          output_op code, op.op
+          code << indentify("#{ss} = self.pos\n", indent)
+          output_op code, op.op, indent
           if op.save_values
-            code << "    @result = nil unless _tmp\n"
+            code << indentify("@result = nil unless _tmp\n", indent)
           end
-          code << "    unless _tmp\n"
-          code << "      _tmp = true\n"
-          code << "      self.pos = #{ss}\n"
-          code << "    end\n"
+          code << indentify("unless _tmp\n", indent)
+          code << indentify("  _tmp = true\n", indent)
+          code << indentify("  self.pos = #{ss}\n", indent)
+          code << indentify("end\n", indent)
         elsif op.min == 0 and !op.max
           if op.save_values
-            code << "    _ary = []\n"
+            code << indentify("_ary = []\n", indent)
           end
 
-          code << "    while true\n"
-          output_op code, op.op
+          code << indentify("while true\n", indent)
+          output_op code, op.op, (indent+1)
           if op.save_values
-            code << "    _ary << @result if _tmp\n"
+            code << indentify("  _ary << @result if _tmp\n", indent)
           end
-          code << "    break unless _tmp\n"
-          code << "    end\n"
-          code << "    _tmp = true\n"
+          code << indentify("  break unless _tmp\n", indent)
+          code << indentify("end\n", indent)
+          code << indentify("_tmp = true\n", indent)
 
           if op.save_values
-            code << "    @result = _ary\n"
+            code << indentify("@result = _ary\n", indent)
           end
 
         elsif op.min == 1 and !op.max
-          code << "    #{ss} = self.pos\n"
+          code << indentify("#{ss} = self.pos\n", indent)
           if op.save_values
-            code << "    _ary = []\n"
+            code << indentify("_ary = []\n", indent)
           end
-          output_op code, op.op
-          code << "    if _tmp\n"
+          output_op code, op.op, indent
+          code << indentify("if _tmp\n", indent)
           if op.save_values
-            code << "      _ary << @result\n"
+            code << indentify("  _ary << @result\n", indent)
           end
-          code << "      while true\n"
-          code << "    "
-          output_op code, op.op
+          code << indentify("  while true\n", indent)
+          output_op code, op.op, (indent+2)
           if op.save_values
-            code << "        _ary << @result if _tmp\n"
+            code << indentify("    _ary << @result if _tmp\n", indent)
           end
-          code << "        break unless _tmp\n"
-          code << "      end\n"
-          code << "      _tmp = true\n"
+          code << indentify("    break unless _tmp\n", indent)
+          code << indentify("  end\n", indent)
+          code << indentify("  _tmp = true\n", indent)
           if op.save_values
-            code << "      @result = _ary\n"
+            code << indentify("  @result = _ary\n", indent)
           end
-          code << "    else\n"
-          code << "      self.pos = #{ss}\n"
-          code << "    end\n"
+          code << indentify("else\n", indent)
+          code << indentify("  self.pos = #{ss}\n", indent)
+          code << indentify("end\n", indent)
         else
-          code << "    #{ss} = self.pos\n"
-          code << "    _count = 0\n"
-          code << "    while true\n"
-          code << "  "
-          output_op code, op.op
-          code << "      if _tmp\n"
-          code << "        _count += 1\n"
-          code << "        break if _count == #{op.max}\n"
-          code << "      else\n"
-          code << "        break\n"
-          code << "      end\n"
-          code << "    end\n"
-          code << "    if _count >= #{op.min}\n"
-          code << "      _tmp = true\n"
-          code << "    else\n"
-          code << "      self.pos = #{ss}\n"
-          code << "      _tmp = nil\n"
-          code << "    end\n"
+          code << indentify("#{ss} = self.pos\n", indent)
+          code << indentify("_count = 0\n", indent)
+          code << indentify("while true\n", indent)
+          output_op code, op.op, (indent+1)
+          code << indentify("  if _tmp\n", indent)
+          code << indentify("    _count += 1\n", indent)
+          code << indentify("    break if _count == #{op.max}\n", indent)
+          code << indentify("  else\n", indent)
+          code << indentify("    break\n", indent)
+          code << indentify("  end\n", indent)
+          code << indentify("end\n", indent)
+          code << indentify("if _count >= #{op.min}\n", indent)
+          code << indentify("  _tmp = true\n", indent)
+          code << indentify("else\n", indent)
+          code << indentify("  self.pos = #{ss}\n", indent)
+          code << indentify("  _tmp = nil\n", indent)
+          code << indentify("end\n", indent)
         end
 
       when Sequence
         ss = save()
-        code << "\n    #{ss} = self.pos\n"
-        code << "    while true # sequence\n"
+        code << "\n"
+        code << indentify("#{ss} = self.pos\n", indent)
+        code << indentify("while true # sequence\n", indent)
         op.ops.each_with_index do |n, idx|
-          output_op code, n
+          output_op code, n, (indent+1)
 
           if idx == op.ops.size - 1
-            code << "    unless _tmp\n"
-            code << "      self.pos = #{ss}\n"
-            code << "    end\n"
-            code << "    break\n"
+            code << indentify("  unless _tmp\n", indent)
+            code << indentify("    self.pos = #{ss}\n", indent)
+            code << indentify("  end\n", indent)
+            code << indentify("  break\n", indent)
           else
-            code << "    unless _tmp\n"
-            code << "      self.pos = #{ss}\n"
-            code << "      break\n"
-            code << "    end\n"
+            code << indentify("  unless _tmp\n", indent)
+            code << indentify("    self.pos = #{ss}\n", indent)
+            code << indentify("    break\n", indent)
+            code << indentify("  end\n", indent)
           end
         end
-        code << "    end # end sequence\n\n"
+        code << indentify("end # end sequence\n\n", indent)
       when AndPredicate
         ss = save()
-        code << "    #{ss} = self.pos\n"
+        code << indentify("#{ss} = self.pos\n", indent)
         if op.op.kind_of? Action
-          code << "    _tmp = begin; #{op.op.action}; end\n"
+          code << indentify("_tmp = begin; #{op.op.action}; end\n", indent)
         else
-          output_op code, op.op
+          output_op code, op.op, indent
         end
-        code << "    self.pos = #{ss}\n"
+        code << indentify("self.pos = #{ss}\n", indent)
       when NotPredicate
         ss = save()
-        code << "    #{ss} = self.pos\n"
+        code << indentify("#{ss} = self.pos\n", indent)
         if op.op.kind_of? Action
-          code << "    _tmp = begin; #{op.op.action}; end\n"
+          code << indentify("_tmp = begin; #{op.op.action}; end\n", indent)
         else
-          output_op code, op.op
+          output_op code, op.op, indent
         end
-        code << "    _tmp = _tmp ? nil : true\n"
-        code << "    self.pos = #{ss}\n"
+        code << indentify("_tmp = _tmp ? nil : true\n", indent)
+        code << indentify("self.pos = #{ss}\n", indent)
       when RuleReference
-        code << "    _tmp = apply(:#{method_name op.rule_name})\n"
+        code << indentify("_tmp = apply(:#{method_name op.rule_name})\n", indent)
       when InvokeRule
         if op.arguments
-          code << "    _tmp = #{method_name op.rule_name}#{op.arguments}\n"
+          code << indentify("_tmp = #{method_name op.rule_name}#{op.arguments}\n", indent)
         else
-          code << "    _tmp = #{method_name op.rule_name}()\n"
+          code << indentify("_tmp = #{method_name op.rule_name}()\n", indent)
         end
       when ForeignInvokeRule
         if op.arguments
-          code << "    _tmp = @_grammar_#{op.grammar_name}.external_invoke(self, :#{method_name op.rule_name}, #{op.arguments[1..-2]})\n"
+          code << indentify("_tmp = @_grammar_#{op.grammar_name}.external_invoke(self, :#{method_name op.rule_name}, #{op.arguments[1..-2]})\n", indent)
         else
-          code << "    _tmp = @_grammar_#{op.grammar_name}.external_invoke(self, :#{method_name op.rule_name})\n"
+          code << indentify("_tmp = @_grammar_#{op.grammar_name}.external_invoke(self, :#{method_name op.rule_name})\n", indent)
         end
       when Tag
         if op.tag_name and !op.tag_name.empty?
-          output_op code, op.op
-          code << "    #{op.tag_name} = @result\n"
+          output_op code, op.op, indent
+          code << indentify("#{op.tag_name} = @result\n", indent)
         else
-          output_op code, op.op
+          output_op code, op.op, indent
         end
       when Action
-        code << "    @result = begin; "
+        code << indentify("@result = begin; ", indent)
         code << op.action << "; end\n"
         if @debug
-          code << "    puts \"   => \" #{op.action.dump} \" => \#{@result.inspect} \\n\"\n"
+          code << indentify("puts \"   => \" #{op.action.dump} \" => \#{@result.inspect} \\n\"\n", indent)
         end
-        code << "    _tmp = true\n"
+        code << indentify("_tmp = true\n", indent)
       when Collect
-        code << "    _text_start = self.pos\n"
-        output_op code, op.op
-        code << "    if _tmp\n"
-        code << "      text = get_text(_text_start)\n"
-        code << "    end\n"
+        code << indentify("_text_start = self.pos\n", indent)
+        output_op code, op.op, indent
+        code << indentify("if _tmp\n", indent)
+        code << indentify("  text = get_text(_text_start)\n", indent)
+        code << indentify("end\n", indent)
       else
         raise "Unknown op - #{op.class}"
       end
@@ -339,13 +353,6 @@ module KPeg
           code << "    @_grammar_#{name} = #{gram}.new(nil)\n"
         end
         code << "  end\n"
-
-        @grammar.foreign_grammars.each do |name, gram|
-          code << "\n"
-          code << "  def invoke_#{name}(*args)\n"
-          code << "    @_grammar_#{name}.external_invoke(self, :_root, *args)\n"
-          code << "  end\n"
-        end
       end
 
       render = GrammarRenderer.new(@grammar)