kpeg 0.8.2 → 0.8.3

data/README.md

@@ -15,10 +15,14 @@ All grammars start with with the class/module name that will be your parser
 
     %% name = Example::Parser
 
-After that a block of ruby code can be defined that will be added into the class body of your parser. Attributes that are defines in this block can be accessed within your parser as instance variables
+After that a block of ruby code can be defined that will be added into the class body of your parser. Attributes that are defined in this block can be accessed within your parser as instance variables. Methods can also be defined in this block and used in action blocks as well.
 
     %% {
       attr_accessor :something_cool
+      
+      def something_awesome
+        # do something awesome
+      end
     }
 
 ### Defining literals
@@ -36,52 +40,50 @@ Literals can also accept multiple definitions
 
     vowel = "a" | "e" | "i" | "o" | "u"
     alpha = /[A-Z]/ | /[a-z]/
-    
 
 ### Defining Rules for Values
 
 Before you can start parsing a string you will need to define rules that you will use to accept or reject that string. There are many different types of rules available in kpeg 
 
 The most basic of these rules is a string capture
-  
+
     alpha = < /[A-Za-z]/ > { text }
-    
+
 
 While this looks very much like the ALPHA literal defined above it differs in one important way, the text captured by the rule defined between the < and > symbols will be set as the text variable in block that follows. You can also explicitly define the variable that you would like but only with existing rules or literals.
-    
+
     letter = alpha:a { a }
-    
+
 Additionally blocks can return true or false values based upon an expression within the block. To return true if a test passes do the following:
 
     match_greater_than_10 = < num:n > &{ n > 10 }
-    
+
 To test and return a false value if the test passes do the following:
 
     do_not_match_greater_than_10 = < num:n > !{ n > 10 }
-    
+
 Rules can also act like functions and take parameters. An example of this is lifted from the [Email List Validator](https://github.com/larb/email_address_validator), where an ascii value is passed in and the character is evaluated against it returning a true if it matches
-    
+
     d(num) = <.> &{ text[0] == num }
 
 Rules support some regular expression syntax for matching
- 
+
 + maybe ?
-+ many 
-+ kleene * 
-+ groupings () 
++ many +
++ kleene *
++ groupings ()
 
 Examples
 
     letters = alpha+
     words = alpha+ space* period?
     sentence = (letters+ | space+)+
-    
+
 Kpeg also allows a rule to define the acceptable number of matches in the form of a range. In regular expressions this is often denoted with syntax like {0,3}. Kpeg uses this syntax to accomplish match ranges [min, max].
 
     matches_3_to_5_times = letter[3,5]
     matches_3_to_any_times = letter[3,*]
-  
-    
+
 ### Defining Actions
 
 Illustrated above in some of the examples, kpeg allows you to perform actions based upon a match that are described in block provided or in the rule definition itself.
@@ -89,6 +91,16 @@ Illustrated above in some of the examples, kpeg allows you to perform actions ba
     num = /[1-9][0-9]*/
     sum = < num:n1 "+" num:n2 > { n1 + n2 }
 
+As of version 0.8 an alternate syntax has been added for calling defined methods as actions.
+
+    %% {
+      def add(n1, n2){
+        n1 + n2
+      }
+    }
+    num = /[1-9][0-9]*/
+    sum = < num:n1 "+" num:n2 > ~add(n1, n2)
+
 ### Referencing an external grammar
 
 Kpeg allows you to run a rule that is defined in an external grammar. This is useful if there is a defined set of rules that you would like to reuse in another parser. To do this, create your grammar and generate a parser using the kpeg command line tool.
@@ -100,7 +112,7 @@ Once you have the generated parser, include that file into your new grammar
     %{
       require "literals.kpeg.rb"
     }
-    
+
 Then create a variable to hold to foreign interface and pass it the class name of your parser. In this case my parser class name is Literal 
 
     %foreign_grammer = Literal
@@ -114,21 +126,21 @@ You can then use rules defined in the foreign grammar in the local grammar file
 Kpeg allows comments to be added to the grammar file by using the # symbol
 
     # This is a comment in my grammar
-    
+
 ## Generating and running your parser
 
 Before you can generate your parser you will need to define a root rule. This will be the first rule run against the string provided to the parser
 
     root = sentence
-    
+
 To generate the parser run the kpeg command with the kpeg file(s) as an argument. This will generate a ruby file with the same name as your grammar file.
 
     kpeg example.kpeg
-    
+
 Include your generated parser file into an application that you want to use the parser in and run it. Create a new instance of the parser and pass in the string you want to evaluate. When parse is called on the parser instance it will return a true if the sting is matched, or false if it doesn't. 
 
     require "example.kpeg.rb"
-    
+
     parser = Example::Parser.new(string_to_evaluate)
     parser.parse
 
@@ -140,6 +152,20 @@ Per vito, you can get the current line or current column in the following way
     column = { current_column }
     foo = line:line ... { # use line here }
 
+## AST Generation
+
+As of Kpeg 0.8 a parser can now generate an AST. To define an AST node use the following syntax
+
+    %% assign = ast Assignment(name, value)
+
+Once you have a defined AST node, it can be used in your grammar like so
+
+    assignment = identifier:i space* = space* value:v ~assign(i,v)
+
+This will create a new Assign node that you can add into your AST.
+
+For a good example of usage check out [Talon](https://github.com/evanphx/talon)
+
 ## Examples
 
 There are several examples available in the /examples directory. The upper parser has a readme with a step by step description of the grammar.
@@ -153,3 +179,5 @@ There are several examples available in the /examples directory. The upper parse
 [Callisto](https://github.com/dwaite/Callisto)
 
 [Doodle](https://github.com/vito/doodle)
+
+[Kanbanpad](https://kanbanpad.com) (uses kpeg for parsing of the 'enter something' bar)

data/lib/kpeg/code_generator.rb

@@ -263,7 +263,11 @@ module KPeg
         code << indentify("_tmp = _tmp ? nil : true\n", indent)
         code << indentify("self.pos = #{ss}\n", indent)
       when RuleReference
-        code << indentify("_tmp = apply(:#{method_name op.rule_name})\n", indent)
+        if op.arguments
+          code << indentify("_tmp = apply_with_args(:#{method_name op.rule_name}, #{op.arguments[1..-2]})\n", indent)
+        else
+          code << indentify("_tmp = apply(:#{method_name op.rule_name})\n", indent)
+        end
       when InvokeRule
         if op.arguments
           code << indentify("_tmp = #{method_name op.rule_name}#{op.arguments}\n", indent)

data/lib/kpeg/compiled_parser.rb

@@ -32,8 +32,8 @@ module KPeg
     end
 
     attr_reader :string
-    attr_reader :result, :failing_rule_offset
-    attr_accessor :pos
+    attr_reader :failing_rule_offset
+    attr_accessor :result, :pos
 
     include Position
 
@@ -218,6 +218,7 @@ module KPeg
       begin
         if val = __send__(rule, *args)
           other.pos = @pos
+          other.result = @result
         else
           other.set_failed_rule "#{self.class}##{rule}"
         end
@@ -228,6 +229,43 @@ module KPeg
       end
     end
 
+    def apply_with_args(rule, *args)
+      memo_key = [rule, args]
+      if m = @memoizations[memo_key][@pos]
+        m.inc!
+
+        prev = @pos
+        @pos = m.pos
+        if m.ans.kind_of? LeftRecursive
+          m.ans.detected = true
+          return nil
+        end
+
+        @result = m.result
+
+        return m.ans
+      else
+        lr = LeftRecursive.new(false)
+        m = MemoEntry.new(lr, @pos)
+        @memoizations[memo_key][@pos] = m
+        start_pos = @pos
+
+        ans = __send__ rule, *args
+
+        m.move! ans, @pos, @result
+
+        # Don't bother trying to grow the left recursion
+        # if it's failing straight away (thus there is no seed)
+        if ans and lr.detected
+          return grow_lr(rule, args, start_pos, m)
+        else
+          return ans
+        end
+
+        return ans
+      end
+    end
+
     def apply(rule)
       if m = @memoizations[rule][@pos]
         m.inc!
@@ -255,7 +293,7 @@ module KPeg
         # Don't bother trying to grow the left recursion
         # if it's failing straight away (thus there is no seed)
         if ans and lr.detected
-          return grow_lr(rule, start_pos, m)
+          return grow_lr(rule, nil, start_pos, m)
         else
           return ans
         end
@@ -264,12 +302,16 @@ module KPeg
       end
     end
 
-    def grow_lr(rule, start_pos, m)
+    def grow_lr(rule, args, start_pos, m)
       while true
         @pos = start_pos
         @result = m.result
 
-        ans = __send__ rule
+        if args
+          ans = __send__ rule, *args
+        else
+          ans = __send__ rule
+        end
         return nil unless ans
 
         break if @pos <= m.pos

data/lib/kpeg/format_parser.rb

@@ -20,8 +20,8 @@ class KPeg::FormatParser
     end
 
     attr_reader :string
-    attr_reader :result, :failing_rule_offset
-    attr_accessor :pos
+    attr_reader :failing_rule_offset
+    attr_accessor :result, :pos
 
     # STANDALONE START
     def current_column(target=pos)
@@ -234,6 +234,7 @@ class KPeg::FormatParser
       begin
         if val = __send__(rule, *args)
           other.pos = @pos
+          other.result = @result
         else
           other.set_failed_rule "#{self.class}##{rule}"
         end
@@ -244,6 +245,43 @@ class KPeg::FormatParser
       end
     end
 
+    def apply_with_args(rule, *args)
+      memo_key = [rule, args]
+      if m = @memoizations[memo_key][@pos]
+        m.inc!
+
+        prev = @pos
+        @pos = m.pos
+        if m.ans.kind_of? LeftRecursive
+          m.ans.detected = true
+          return nil
+        end
+
+        @result = m.result
+
+        return m.ans
+      else
+        lr = LeftRecursive.new(false)
+        m = MemoEntry.new(lr, @pos)
+        @memoizations[memo_key][@pos] = m
+        start_pos = @pos
+
+        ans = __send__ rule, *args
+
+        m.move! ans, @pos, @result
+
+        # Don't bother trying to grow the left recursion
+        # if it's failing straight away (thus there is no seed)
+        if ans and lr.detected
+          return grow_lr(rule, args, start_pos, m)
+        else
+          return ans
+        end
+
+        return ans
+      end
+    end
+
     def apply(rule)
       if m = @memoizations[rule][@pos]
         m.inc!
@@ -271,7 +309,7 @@ class KPeg::FormatParser
         # Don't bother trying to grow the left recursion
         # if it's failing straight away (thus there is no seed)
         if ans and lr.detected
-          return grow_lr(rule, start_pos, m)
+          return grow_lr(rule, nil, start_pos, m)
         else
           return ans
         end
@@ -280,12 +318,16 @@ class KPeg::FormatParser
       end
     end
 
-    def grow_lr(rule, start_pos, m)
+    def grow_lr(rule, args, start_pos, m)
       while true
         @pos = start_pos
         @result = m.result
 
-        ans = __send__ rule
+        if args
+          ans = __send__ rule, *args
+        else
+          ans = __send__ rule
+        end
         return nil unless ans
 
         break if @pos <= m.pos
@@ -536,7 +578,7 @@ class KPeg::FormatParser
     return _tmp
   end
 
-  # dbl_escapes = ("\\\"" { '"' } | "\\n" { "\n" } | "\\t" { "\t" } | "\\b" { "\b" } | "\\\\" { "\\" })
+  # dbl_escapes = ("n" { "\n" } | "s" { " " } | "r" { "\r" } | "t" { "\t" } | "v" { "\v" } | "f" { "\f" } | "b" { "\b" } | "a" { "\a" } | "e" { "\e" } | "\\" { "\\" } | "\"" { "\"" } | num_escapes)
   def _dbl_escapes
 
     _save = self.pos
@@ -544,12 +586,12 @@ class KPeg::FormatParser
 
       _save1 = self.pos
       while true # sequence
-        _tmp = match_string("\\\"")
+        _tmp = match_string("n")
         unless _tmp
           self.pos = _save1
           break
         end
-        @result = begin;  '"' ; end
+        @result = begin;  "\n" ; end
         _tmp = true
         unless _tmp
           self.pos = _save1
@@ -562,12 +604,12 @@ class KPeg::FormatParser
 
       _save2 = self.pos
       while true # sequence
-        _tmp = match_string("\\n")
+        _tmp = match_string("s")
         unless _tmp
           self.pos = _save2
           break
         end
-        @result = begin;  "\n" ; end
+        @result = begin;  " " ; end
         _tmp = true
         unless _tmp
           self.pos = _save2
@@ -580,12 +622,12 @@ class KPeg::FormatParser
 
       _save3 = self.pos
       while true # sequence
-        _tmp = match_string("\\t")
+        _tmp = match_string("r")
         unless _tmp
           self.pos = _save3
           break
         end
-        @result = begin;  "\t" ; end
+        @result = begin;  "\r" ; end
         _tmp = true
         unless _tmp
           self.pos = _save3
@@ -598,12 +640,12 @@ class KPeg::FormatParser
 
       _save4 = self.pos
       while true # sequence
-        _tmp = match_string("\\b")
+        _tmp = match_string("t")
         unless _tmp
           self.pos = _save4
           break
         end
-        @result = begin;  "\b" ; end
+        @result = begin;  "\t" ; end
         _tmp = true
         unless _tmp
           self.pos = _save4
@@ -616,12 +658,12 @@ class KPeg::FormatParser
 
       _save5 = self.pos
       while true # sequence
-        _tmp = match_string("\\\\")
+        _tmp = match_string("v")
         unless _tmp
           self.pos = _save5
           break
         end
-        @result = begin;  "\\" ; end
+        @result = begin;  "\v" ; end
         _tmp = true
         unless _tmp
           self.pos = _save5
@@ -629,6 +671,117 @@ class KPeg::FormatParser
         break
       end # end sequence
 
+      break if _tmp
+      self.pos = _save
+
+      _save6 = self.pos
+      while true # sequence
+        _tmp = match_string("f")
+        unless _tmp
+          self.pos = _save6
+          break
+        end
+        @result = begin;  "\f" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save6
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save7 = self.pos
+      while true # sequence
+        _tmp = match_string("b")
+        unless _tmp
+          self.pos = _save7
+          break
+        end
+        @result = begin;  "\b" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save7
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save8 = self.pos
+      while true # sequence
+        _tmp = match_string("a")
+        unless _tmp
+          self.pos = _save8
+          break
+        end
+        @result = begin;  "\a" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save8
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save9 = self.pos
+      while true # sequence
+        _tmp = match_string("e")
+        unless _tmp
+          self.pos = _save9
+          break
+        end
+        @result = begin;  "\e" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save9
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save10 = self.pos
+      while true # sequence
+        _tmp = match_string("\\")
+        unless _tmp
+          self.pos = _save10
+          break
+        end
+        @result = begin;  "\\" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save10
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save11 = self.pos
+      while true # sequence
+        _tmp = match_string("\"")
+        unless _tmp
+          self.pos = _save11
+          break
+        end
+        @result = begin;  "\"" ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save11
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+      _tmp = apply(:_num_escapes)
       break if _tmp
       self.pos = _save
       break
@@ -638,6 +791,67 @@ class KPeg::FormatParser
     return _tmp
   end
 
+  # num_escapes = (< /[0-7]{3}/ > { [text.to_i(8)].pack("U") } | "x" < /[0-9a-fA-F]{2}/ > { [text.to_i(16)].pack("U") })
+  def _num_escapes
+
+    _save = self.pos
+    while true # choice
+
+      _save1 = self.pos
+      while true # sequence
+        _text_start = self.pos
+        _tmp = scan(/\A(?-mix:[0-7]{3})/)
+        if _tmp
+          text = get_text(_text_start)
+        end
+        unless _tmp
+          self.pos = _save1
+          break
+        end
+        @result = begin;  [text.to_i(8)].pack("U") ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save1
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+
+      _save2 = self.pos
+      while true # sequence
+        _tmp = match_string("x")
+        unless _tmp
+          self.pos = _save2
+          break
+        end
+        _text_start = self.pos
+        _tmp = scan(/\A(?-mix:[0-9a-fA-F]{2})/)
+        if _tmp
+          text = get_text(_text_start)
+        end
+        unless _tmp
+          self.pos = _save2
+          break
+        end
+        @result = begin;  [text.to_i(16)].pack("U") ; end
+        _tmp = true
+        unless _tmp
+          self.pos = _save2
+        end
+        break
+      end # end sequence
+
+      break if _tmp
+      self.pos = _save
+      break
+    end # end choice
+
+    set_failed_rule :_num_escapes unless _tmp
+    return _tmp
+  end
+
   # dbl_seq = < /[^\\"]+/ > { text }
   def _dbl_seq
 
@@ -664,58 +878,52 @@ class KPeg::FormatParser
     return _tmp
   end
 
-  # dbl_not_quote = (dbl_escapes:s | dbl_seq:s)+:ary { ary }
+  # dbl_not_quote = ("\\" dbl_escapes:s | dbl_seq:s)*:ary { Array(ary) }
   def _dbl_not_quote
 
     _save = self.pos
     while true # sequence
-      _save1 = self.pos
       _ary = []
+      while true
 
-      _save2 = self.pos
-      while true # choice
-        _tmp = apply(:_dbl_escapes)
-        s = @result
-        break if _tmp
-        self.pos = _save2
-        _tmp = apply(:_dbl_seq)
-        s = @result
-        break if _tmp
-        self.pos = _save2
-        break
-      end # end choice
-
-      if _tmp
-        _ary << @result
-        while true
+        _save2 = self.pos
+        while true # choice
 
           _save3 = self.pos
-          while true # choice
+          while true # sequence
+            _tmp = match_string("\\")
+            unless _tmp
+              self.pos = _save3
+              break
+            end
             _tmp = apply(:_dbl_escapes)
             s = @result
-            break if _tmp
-            self.pos = _save3
-            _tmp = apply(:_dbl_seq)
-            s = @result
-            break if _tmp
-            self.pos = _save3
+            unless _tmp
+              self.pos = _save3
+            end
             break
-          end # end choice
+          end # end sequence
 
-          _ary << @result if _tmp
-          break unless _tmp
-        end
-        _tmp = true
-        @result = _ary
-      else
-        self.pos = _save1
+          break if _tmp
+          self.pos = _save2
+          _tmp = apply(:_dbl_seq)
+          s = @result
+          break if _tmp
+          self.pos = _save2
+          break
+        end # end choice
+
+        _ary << @result if _tmp
+        break unless _tmp
       end
+      _tmp = true
+      @result = _ary
       ary = @result
       unless _tmp
         self.pos = _save
         break
       end
-      @result = begin;  ary ; end
+      @result = begin;  Array(ary) ; end
       _tmp = true
       unless _tmp
         self.pos = _save
@@ -808,54 +1016,36 @@ class KPeg::FormatParser
     return _tmp
   end
 
-  # sgl_not_quote = (sgl_escape_quote | sgl_seq)+:segs { segs.join }
+  # sgl_not_quote = (sgl_escape_quote | sgl_seq)*:segs { Array(segs) }
   def _sgl_not_quote
 
     _save = self.pos
     while true # sequence
-      _save1 = self.pos
       _ary = []
+      while true
 
-      _save2 = self.pos
-      while true # choice
-        _tmp = apply(:_sgl_escape_quote)
-        break if _tmp
-        self.pos = _save2
-        _tmp = apply(:_sgl_seq)
-        break if _tmp
-        self.pos = _save2
-        break
-      end # end choice
-
-      if _tmp
-        _ary << @result
-        while true
-
-          _save3 = self.pos
-          while true # choice
-            _tmp = apply(:_sgl_escape_quote)
-            break if _tmp
-            self.pos = _save3
-            _tmp = apply(:_sgl_seq)
-            break if _tmp
-            self.pos = _save3
-            break
-          end # end choice
+        _save2 = self.pos
+        while true # choice
+          _tmp = apply(:_sgl_escape_quote)
+          break if _tmp
+          self.pos = _save2
+          _tmp = apply(:_sgl_seq)
+          break if _tmp
+          self.pos = _save2
+          break
+        end # end choice
 
-          _ary << @result if _tmp
-          break unless _tmp
-        end
-        _tmp = true
-        @result = _ary
-      else
-        self.pos = _save1
+        _ary << @result if _tmp
+        break unless _tmp
       end
+      _tmp = true
+      @result = _ary
       segs = @result
       unless _tmp
         self.pos = _save
         break
       end
-      @result = begin;  segs.join ; end
+      @result = begin;  Array(segs) ; end
       _tmp = true
       unless _tmp
         self.pos = _save
@@ -867,7 +1057,7 @@ class KPeg::FormatParser
     return _tmp
   end
 
-  # sgl_string = "'" sgl_not_quote:s "'" { @g.str(s) }
+  # sgl_string = "'" sgl_not_quote:s "'" { @g.str(s.join) }
   def _sgl_string
 
     _save = self.pos
@@ -888,7 +1078,7 @@ class KPeg::FormatParser
         self.pos = _save
         break
       end
-      @result = begin;  @g.str(s) ; end
+      @result = begin;  @g.str(s.join) ; end
       _tmp = true
       unless _tmp
         self.pos = _save
@@ -1401,7 +1591,7 @@ class KPeg::FormatParser
     return _tmp
   end
 
-  # value = (value:v ":" var:n { @g.t(v,n) } | value:v "?" { @g.maybe(v) } | value:v "+" { @g.many(v) } | value:v "*" { @g.kleene(v) } | value:v mult_range:r { @g.multiple(v, *r) } | "&" value:v { @g.andp(v) } | "!" value:v { @g.notp(v) } | "(" - expression:o - ")" { o } | "@<" - expression:o - ">" { @g.bounds(o) } | "<" - expression:o - ">" { @g.collect(o) } | curly_block | "~" method:m < nested_paren? > { @g.action("#{m}#{text}") } | "." { @g.dot } | "@" var:name !(- "=") { @g.invoke(name) } | "^" var:name < nested_paren? > { @g.foreign_invoke("parent", name, text) } | "%" var:gram "." var:name < nested_paren? > { @g.foreign_invoke(gram, name, text) } | var:name < nested_paren? > !(- "=") { text.empty? ? @g.ref(name) : @g.invoke(name, text) } | char_range | regexp | string)
+  # value = (value:v ":" var:n { @g.t(v,n) } | value:v "?" { @g.maybe(v) } | value:v "+" { @g.many(v) } | value:v "*" { @g.kleene(v) } | value:v mult_range:r { @g.multiple(v, *r) } | "&" value:v { @g.andp(v) } | "!" value:v { @g.notp(v) } | "(" - expression:o - ")" { o } | "@<" - expression:o - ">" { @g.bounds(o) } | "<" - expression:o - ">" { @g.collect(o) } | curly_block | "~" method:m < nested_paren? > { @g.action("#{m}#{text}") } | "." { @g.dot } | "@" var:name < nested_paren? > !(- "=") { @g.invoke(name, text.empty? ? nil : text) } | "^" var:name < nested_paren? > { @g.foreign_invoke("parent", name, text) } | "%" var:gram "." var:name < nested_paren? > { @g.foreign_invoke(gram, name, text) } | var:name < nested_paren? > !(- "=") { @g.ref(name, nil, text.empty? ? nil : text) } | char_range | regexp | string)
   def _value
 
     _save = self.pos
@@ -1771,29 +1961,43 @@ class KPeg::FormatParser
           self.pos = _save14
           break
         end
+        _text_start = self.pos
         _save15 = self.pos
-
+        _tmp = apply(:_nested_paren)
+        unless _tmp
+          _tmp = true
+          self.pos = _save15
+        end
+        if _tmp
+          text = get_text(_text_start)
+        end
+        unless _tmp
+          self.pos = _save14
+          break
+        end
         _save16 = self.pos
+
+        _save17 = self.pos
         while true # sequence
           _tmp = apply(:__hyphen_)
           unless _tmp
-            self.pos = _save16
+            self.pos = _save17
             break
           end
           _tmp = match_string("=")
           unless _tmp
-            self.pos = _save16
+            self.pos = _save17
           end
           break
         end # end sequence
 
         _tmp = _tmp ? nil : true
-        self.pos = _save15
+        self.pos = _save16
         unless _tmp
           self.pos = _save14
           break
         end
-        @result = begin;  @g.invoke(name) ; end
+        @result = begin;  @g.invoke(name, text.empty? ? nil : text) ; end
         _tmp = true
         unless _tmp
           self.pos = _save14
@@ -1804,37 +2008,37 @@ class KPeg::FormatParser
       break if _tmp
       self.pos = _save
 
-      _save17 = self.pos
+      _save18 = self.pos
       while true # sequence
         _tmp = match_string("^")
         unless _tmp
-          self.pos = _save17
+          self.pos = _save18
           break
         end
         _tmp = apply(:_var)
         name = @result
         unless _tmp
-          self.pos = _save17
+          self.pos = _save18
           break
         end
         _text_start = self.pos
-        _save18 = self.pos
+        _save19 = self.pos
         _tmp = apply(:_nested_paren)
         unless _tmp
           _tmp = true
-          self.pos = _save18
+          self.pos = _save19
         end
         if _tmp
           text = get_text(_text_start)
         end
         unless _tmp
-          self.pos = _save17
+          self.pos = _save18
           break
         end
         @result = begin;  @g.foreign_invoke("parent", name, text) ; end
         _tmp = true
         unless _tmp
-          self.pos = _save17
+          self.pos = _save18
         end
         break
       end # end sequence
@@ -1842,48 +2046,48 @@ class KPeg::FormatParser
       break if _tmp
       self.pos = _save
 
-      _save19 = self.pos
+      _save20 = self.pos
       while true # sequence
         _tmp = match_string("%")
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
           break
         end
         _tmp = apply(:_var)
         gram = @result
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
           break
         end
         _tmp = match_string(".")
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
           break
         end
         _tmp = apply(:_var)
         name = @result
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
           break
         end
         _text_start = self.pos
-        _save20 = self.pos
+        _save21 = self.pos
         _tmp = apply(:_nested_paren)
         unless _tmp
           _tmp = true
-          self.pos = _save20
+          self.pos = _save21
         end
         if _tmp
           text = get_text(_text_start)
         end
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
           break
         end
         @result = begin;  @g.foreign_invoke(gram, name, text) ; end
         _tmp = true
         unless _tmp
-          self.pos = _save19
+          self.pos = _save20
         end
         break
       end # end sequence
@@ -1891,54 +2095,54 @@ class KPeg::FormatParser
       break if _tmp
       self.pos = _save
 
-      _save21 = self.pos
+      _save22 = self.pos
       while true # sequence
         _tmp = apply(:_var)
         name = @result
         unless _tmp
-          self.pos = _save21
+          self.pos = _save22
           break
         end
         _text_start = self.pos
-        _save22 = self.pos
+        _save23 = self.pos
         _tmp = apply(:_nested_paren)
         unless _tmp
           _tmp = true
-          self.pos = _save22
+          self.pos = _save23
         end
         if _tmp
           text = get_text(_text_start)
         end
         unless _tmp
-          self.pos = _save21
+          self.pos = _save22
           break
         end
-        _save23 = self.pos
-
         _save24 = self.pos
+
+        _save25 = self.pos
         while true # sequence
           _tmp = apply(:__hyphen_)
           unless _tmp
-            self.pos = _save24
+            self.pos = _save25
             break
           end
           _tmp = match_string("=")
           unless _tmp
-            self.pos = _save24
+            self.pos = _save25
           end
           break
         end # end sequence
 
         _tmp = _tmp ? nil : true
-        self.pos = _save23
+        self.pos = _save24
         unless _tmp
-          self.pos = _save21
+          self.pos = _save22
           break
         end
-        @result = begin;  text.empty? ? @g.ref(name) : @g.invoke(name, text) ; end
+        @result = begin;  @g.ref(name, nil, text.empty? ? nil : text) ; end
         _tmp = true
         unless _tmp
-          self.pos = _save21
+          self.pos = _save22
         end
         break
       end # end sequence
@@ -2871,14 +3075,15 @@ class KPeg::FormatParser
   Rules[:_kleene] = rule_info("kleene", "\"*\"")
   Rules[:_var] = rule_info("var", "< (\"-\" | /[a-zA-Z][\\-_a-zA-Z0-9]*/) > { text }")
   Rules[:_method] = rule_info("method", "< /[a-zA-Z_][a-zA-Z0-9_]*/ > { text }")
-  Rules[:_dbl_escapes] = rule_info("dbl_escapes", "(\"\\\\\\\"\" { '\"' } | \"\\\\n\" { \"\\n\" } | \"\\\\t\" { \"\\t\" } | \"\\\\b\" { \"\\b\" } | \"\\\\\\\\\" { \"\\\\\" })")
+  Rules[:_dbl_escapes] = rule_info("dbl_escapes", "(\"n\" { \"\\n\" } | \"s\" { \" \" } | \"r\" { \"\\r\" } | \"t\" { \"\\t\" } | \"v\" { \"\\v\" } | \"f\" { \"\\f\" } | \"b\" { \"\\b\" } | \"a\" { \"\\a\" } | \"e\" { \"\\e\" } | \"\\\\\" { \"\\\\\" } | \"\\\"\" { \"\\\"\" } | num_escapes)")
+  Rules[:_num_escapes] = rule_info("num_escapes", "(< /[0-7]{3}/ > { [text.to_i(8)].pack(\"U\") } | \"x\" < /[0-9a-fA-F]{2}/ > { [text.to_i(16)].pack(\"U\") })")
   Rules[:_dbl_seq] = rule_info("dbl_seq", "< /[^\\\\\"]+/ > { text }")
-  Rules[:_dbl_not_quote] = rule_info("dbl_not_quote", "(dbl_escapes:s | dbl_seq:s)+:ary { ary }")
+  Rules[:_dbl_not_quote] = rule_info("dbl_not_quote", "(\"\\\\\" dbl_escapes:s | dbl_seq:s)*:ary { Array(ary) }")
   Rules[:_dbl_string] = rule_info("dbl_string", "\"\\\"\" dbl_not_quote:s \"\\\"\" { @g.str(s.join) }")
   Rules[:_sgl_escape_quote] = rule_info("sgl_escape_quote", "\"\\\\'\" { \"'\" }")
   Rules[:_sgl_seq] = rule_info("sgl_seq", "< /[^']/ > { text }")
-  Rules[:_sgl_not_quote] = rule_info("sgl_not_quote", "(sgl_escape_quote | sgl_seq)+:segs { segs.join }")
-  Rules[:_sgl_string] = rule_info("sgl_string", "\"'\" sgl_not_quote:s \"'\" { @g.str(s) }")
+  Rules[:_sgl_not_quote] = rule_info("sgl_not_quote", "(sgl_escape_quote | sgl_seq)*:segs { Array(segs) }")
+  Rules[:_sgl_string] = rule_info("sgl_string", "\"'\" sgl_not_quote:s \"'\" { @g.str(s.join) }")
   Rules[:_string] = rule_info("string", "(dbl_string | sgl_string)")
   Rules[:_not_slash] = rule_info("not_slash", "< (\"\\\\/\" | /[^\\/]/)+ > { text }")
   Rules[:_regexp_opts] = rule_info("regexp_opts", "< [a-z]* > { text }")
@@ -2891,7 +3096,7 @@ class KPeg::FormatParser
   Rules[:_curly_block] = rule_info("curly_block", "curly")
   Rules[:_curly] = rule_info("curly", "\"{\" < (/[^{}\"']+/ | string | curly)* > \"}\" { @g.action(text) }")
   Rules[:_nested_paren] = rule_info("nested_paren", "\"(\" (/[^()\"']+/ | string | nested_paren)* \")\"")
-  Rules[:_value] = rule_info("value", "(value:v \":\" var:n { @g.t(v,n) } | value:v \"?\" { @g.maybe(v) } | value:v \"+\" { @g.many(v) } | value:v \"*\" { @g.kleene(v) } | value:v mult_range:r { @g.multiple(v, *r) } | \"&\" value:v { @g.andp(v) } | \"!\" value:v { @g.notp(v) } | \"(\" - expression:o - \")\" { o } | \"@<\" - expression:o - \">\" { @g.bounds(o) } | \"<\" - expression:o - \">\" { @g.collect(o) } | curly_block | \"~\" method:m < nested_paren? > { @g.action(\"\#{m}\#{text}\") } | \".\" { @g.dot } | \"@\" var:name !(- \"=\") { @g.invoke(name) } | \"^\" var:name < nested_paren? > { @g.foreign_invoke(\"parent\", name, text) } | \"%\" var:gram \".\" var:name < nested_paren? > { @g.foreign_invoke(gram, name, text) } | var:name < nested_paren? > !(- \"=\") { text.empty? ? @g.ref(name) : @g.invoke(name, text) } | char_range | regexp | string)")
+  Rules[:_value] = rule_info("value", "(value:v \":\" var:n { @g.t(v,n) } | value:v \"?\" { @g.maybe(v) } | value:v \"+\" { @g.many(v) } | value:v \"*\" { @g.kleene(v) } | value:v mult_range:r { @g.multiple(v, *r) } | \"&\" value:v { @g.andp(v) } | \"!\" value:v { @g.notp(v) } | \"(\" - expression:o - \")\" { o } | \"@<\" - expression:o - \">\" { @g.bounds(o) } | \"<\" - expression:o - \">\" { @g.collect(o) } | curly_block | \"~\" method:m < nested_paren? > { @g.action(\"\#{m}\#{text}\") } | \".\" { @g.dot } | \"@\" var:name < nested_paren? > !(- \"=\") { @g.invoke(name, text.empty? ? nil : text) } | \"^\" var:name < nested_paren? > { @g.foreign_invoke(\"parent\", name, text) } | \"%\" var:gram \".\" var:name < nested_paren? > { @g.foreign_invoke(gram, name, text) } | var:name < nested_paren? > !(- \"=\") { @g.ref(name, nil, text.empty? ? nil : text) } | char_range | regexp | string)")
   Rules[:_spaces] = rule_info("spaces", "(space | comment)+")
   Rules[:_values] = rule_info("values", "(values:s spaces value:v { @g.seq(s, v) } | value:l spaces value:r { @g.seq(l, r) } | value)")
   Rules[:_choose_cont] = rule_info("choose_cont", "- \"|\" - values:v { v }")

data/lib/kpeg/grammar.rb

@@ -396,13 +396,14 @@ module KPeg
   end
 
   class RuleReference < Operator
-    def initialize(name, grammar=nil)
+    def initialize(name, grammar=nil, args=nil)
       super()
       @rule_name = name
       @grammar = grammar
+      @arguments = args
     end
 
-    attr_reader :rule_name
+    attr_reader :rule_name, :arguments
 
     def match(x)
       if @grammar and @grammar != x.grammar
@@ -421,14 +422,19 @@ module KPeg
     def ==(obj)
       case obj
       when RuleReference
-        @rule_name == obj.rule_name
+        @rule_name == obj.rule_name and @arguments == obj.arguments
       else
         super
       end
     end
 
     def inspect
-      inspect_type "ref", @rule_name
+      if @arguments
+        body = "#{@rule_name} #{@arguments}"
+      else
+        body = @rule_name
+      end
+      inspect_type "ref", body
     end
   end
 
@@ -807,14 +813,14 @@ module KPeg
       NotPredicate.new Grammar.resolve(node)
     end
 
-    def ref(name, other_grammar=nil)
-      RuleReference.new name.to_s, other_grammar
+    def ref(name, other_grammar=nil, args=nil)
+      RuleReference.new name.to_s, other_grammar, args
     end
 
     def invoke(name, args=nil)
       InvokeRule.new name.to_s, args
     end
-    
+
     # Invoke a rule defined on a foreign grammar
     # == Parameters:
     # gram::

data/lib/kpeg/grammar_renderer.rb

@@ -132,10 +132,14 @@ module KPeg
           render_op io, op.op
         end
       when RuleReference
-        io.print op.rule_name
-      when InvokeRule
         if op.arguments
           io.print "#{op.rule_name}#{op.arguments}"
+        else
+          io.print "#{op.rule_name}"
+        end
+      when InvokeRule
+        if op.arguments
+          io.print "@#{op.rule_name}#{op.arguments}"
         else
           io.print "@#{op.rule_name}"
         end

data/lib/kpeg/version.rb

@@ -1,3 +1,3 @@
 module KPeg
-  VERSION = "0.8.2"
+  VERSION = "0.8.3"
 end

data/test/test_kpeg_code_generator.rb

@@ -799,7 +799,7 @@ class Test < KPeg::CompiledParser
     return _tmp
   end
 
-  # root = greeting(1,2)
+  # root = @greeting(1,2)
   def _root
     _tmp = _greeting(1,2)
     set_failed_rule :_root unless _tmp
@@ -808,7 +808,7 @@ class Test < KPeg::CompiledParser
 
   Rules = {}
   Rules[:_greeting] = rule_info("greeting", "\\\"hello\\\"")
-  Rules[:_root] = rule_info("root", "greeting(1,2)")
+  Rules[:_root] = rule_info("root", "@greeting(1,2)")
 end
     STR
 

data/test/test_kpeg_format.rb

@@ -27,6 +27,10 @@ class TestKPegFormat < Test::Unit::TestCase
     assert_rule G.ref("b"), match("a=b"), "a"
   end
 
+  def test_apply_with_arg
+    assert_rule G.ref("b", nil, "(x)"), match("a=b(x)"), "a"
+  end
+
   def test_invoke
     assert_rule G.invoke("b"), match("a=@b"), "a"
   end
@@ -107,7 +111,7 @@ b(p) = x
 
 
   def test_invoke_with_multiple_args
-    assert_rule G.invoke("b", "(1,2)"), match("a=b(1,2)"), "a"
+    assert_rule G.invoke("b", "(1,2)"), match("a=@b(1,2)"), "a"
   end
 
   def test_invoke_foreign_rule
@@ -140,7 +144,11 @@ b(p) = x
   end
 
   def test_string
+    assert_rule G.str(""), match('a=""')
     assert_rule G.str("hello"), match('a="hello"')
+    assert_rule G.str("hello\ngoodbye"), match('a="hello\ngoodbye"')
+    assert_rule G.str("\n\s\r\t\v\f\b\a\r\\\"\012\x1b"),
+                match('a="\n\s\r\t\v\f\b\a\r\\\\\\"\012\x1b"')
     assert_rule G.str("h\"ello"), match('a="h\"ello"')
   end
 
@@ -453,7 +461,7 @@ fact = fact "*" num
   def test_allow_ends_with_comment
     path = File.expand_path("../inputs/comments.kpeg", __FILE__)
     parser = KPeg::FormatParser.new File.read(path), true
-    assert true, parser.parse
+    assert_equal true, parser.parse
   end
 
   def test_roundtrip

data/test/test_kpeg_grammar_renderer.rb

@@ -36,7 +36,7 @@ class TestKPegGrammarRenderer < Test::Unit::TestCase
     gr = KPeg::GrammarRenderer.new(gram)
     gr.render(io)
 
-    assert_equal "root = greeting(1,2)\n", io.string
+    assert_equal "root = @greeting(1,2)\n", io.string
   end
 
   def test_foreign_invoke

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: kpeg
 version: !ruby/object:Gem::Version 
-  hash: 59
+  hash: 57
   prerelease: 
   segments: 
   - 0
   - 8
-  - 2
-  version: 0.8.2
+  - 3
+  version: 0.8.3
 platform: ruby
 authors: 
 - Evan Phoenix
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-04-18 00:00:00 -07:00
+date: 2011-05-05 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency