neg 0.3.0 → 1.0.0

data/CHANGELOG.md

@@ -2,7 +2,7 @@
 # neg - CHANGELOG.md
 
 
-## neg - 1.0.0    not yet released
+## neg - 1.0.0    released 2013-01-16
 
 - initial release
 

data/LICENSE.txt

@@ -1,5 +1,5 @@
 
-Copyright (c) 2012-2012, John Mettraux, jmettraux@gmail.com
+Copyright (c) 2012-2013, John Mettraux, jmettraux@gmail.com
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,10 +1,177 @@
 
 # neg
 
-A neg parser.
+A neg narser.
 
 A silly little exploration project.
 
+It could have been "peg" as in "peg, a peg parser" but that would have been presomptuous, it could have been "leg" as in "leg, a leg larser", but there is already a [leg](http://piumarta.com/software/peg/peg.1.html), so it became "neg" as in "neg, a neg narser". It sounds neg-ative, but whatever, it's just a toy project.
+
+
+## Ruby PEG libraries
+
+Ruby has many such libraries. Here are three preeminent ones:
+
+* Treetop: <http://treetop.rubyforge.org/>
+* Citrus: <http://mjijackson.com/citrus/>
+* Parslet: <http://kschiess.github.com/parslet/>
+
+My favourite is Parslet. Neg is born out of the ashes of contribution attempts to Parslet. Studying this great library made me want to implement my own mini PEG library, for the fun of it.
+
+So if you're looking for something robust and battle-tested, something for the long term, stop reading here and use one of the three gems above. IMHO, [Parslet](http://kschiess.github.com/parslet/) stands above for its error reporting.
+
+
+## expressing a grammar with neg
+
+Here is the classical arithmetic example:
+
+```ruby
+  class ArithParser < Neg::Parser
+
+    expression  == operation
+
+    operator    == `+` | `-` | `*` | `/`
+    operation   == value + (operator + value) * 0
+    value       == parenthese | number
+    parenthese  == `(` + expression + `)`
+    number      == `-` * -1 + _('0-9') * 1
+  end
+
+  tree = ArithParser.parse("1+(2*12)")
+```
+
+(Note: this is Ruby code)
+
+
+## grammar building blocks
+
+```ruby
+# leaves
+
+  StringParser
+    text == `foreach`
+
+  CharacterParser
+    stuff == _             # any character
+    stuff == _ * 1         # one or more of any character
+    stuff == _("0-9") * 1  # like /[0-9]+/
+
+# composite
+
+  SequenceParser
+    sentence == subject + verb + object
+
+  AlternativeParser
+    subject == person | animal | place
+
+  # parentheses
+    sentence = (person | animal) + verb + (object | (`in ` + place))
+
+# modifiers
+
+  RepetitionParser
+    text == `x` * 0       # 0 or more
+    text == `x` * 1       # 1 or more
+    text == `x` * -1      # 0 or 1
+    text == `x` * [2, 4]  # 2, 3 or 4
+
+  LookaheadParser
+    x_then_z     == `x` + ~`z`  # presence
+    x_then_not_z == `x` + -`z`  # absence
+
+# naming
+
+  NonTerminalParser
+    brand == `mazda` | `ford`  # "brand" is the non-terminal
+
+  NonTerminalParser (name is omitted in output parse tree)
+    _operator == `+` | `*` | `-` | `/`
+
+  Embedded naming (here "operator")
+    operation == number + (`+` | `-`)["operator"] + number
+```
+
+
+## parser output
+
+Without a translator, the parser outputs a raw parse tree, something like:
+
+```ruby
+[ :json,
+  [ 0, 1, 1 ],
+  true,
+  nil,
+  [ [ :spaces?, [ 0, 1, 1 ], true, '', [] ],
+    [ :value, [ 0, 1, 1 ], true, nil, [
+      [ :bfalse, [ 0, 1, 1 ], true, 'false', [] ] ] ],
+    [ :spaces?, [ 5, 1, 6 ], true, '', [] ] ] ]
+```
+
+It's a nested assemblage of result nodes.
+
+```ruby
+[ rule_name, [ offset, line, column ], success?, result, children ]
+  #
+  # for example
+[ :bfalse, [ 0, 1, 1 ], true, 'false', [] ]
+```
+
+In case of successful parsing, the succes? == false also get all pruned. In case of failed parsing, they are left in the output parse tree.
+
+A translator turns a raw parse tree into some final result. Look below and at the JSON parser sample in the specs for more information. If the parse failed and a translator is present, a ParseError is raised.
+
+
+## parser + translator
+
+It's OK to stuff the translator inside of the parser:
+
+```ruby
+class CompactArithParser < Neg::Parser
+
+  parser do
+
+    expression  == operation
+
+    operator    == `+` | `-` | `*` | `/`
+    operation   == value + (operator + value) * 0
+    value       == parenthese | number
+    parenthese  == `(` + expression + `)`
+    number      == `-` * -1 + _('0-9') * 1
+  end
+
+  translator do
+
+    on(:number)    { |n| n.result.to_i }
+    on(:operator)  { |n| n.result }
+    on(:value)     { |n| n.results.first }
+
+    on(:expression) { |n|
+      results = n.results.flatten(2)
+      results.size == 1 ? results.first : results
+    }
+  end
+end
+
+CompactArithParser.parse("1+2+3")
+  # => [ 1, '+', 2, '+', 3 ]
+```
+
+As said above, when a translator is present and the parsing fails (before the translator kicks in), a ParseError is raised, with fancy methods to navigate the failed parse tree.
+
+
+## presentations
+
+Neg was published on the 2012-10-06 as it was presented to [Hiroshima.rb](http://hiroshimarb.github.com/).
+
+The \[very dry\] deck of slides that accompanied it can be found at <https://speakerdeck.com/u/jmettraux/p/neg-a-neg-narser>.
+
+
+## links
+
+* source: <https://github.com/jmettraux/neg>
+* issues: <https://github.com/jmettraux/neg/issues>
+* irc: freenode.net #ruote
+
 
 ## license
 

data/TODO.txt

@@ -4,15 +4,14 @@
 [o] switch from ^ to * (how * is related to +)
 [o] _ (any)
 [o] chars
-[ ] lookahead present/absent
+[o] lookahead present/absent
   ~ x --> present
   ! x --> absent
+
 [ ] blankslate
 [ ] drop UnconsumedInput, replace with regular [ false, ... ] output
 [ ] x * '?' / x * '+' / x * '*' as shortcuts
+[ ] memoization (only at non-terminal level?)
 
-  `x` + [.]
-  `x` + [a-z]
-  `x` + c('a-z')
-  `x` + _('a-z')
+[ ] "xxx" instead of `xxx` (trick on the right side)
 

data/lib/neg.rb

@@ -1,3 +1,4 @@
 
+require 'neg/version'
 require 'neg/parser'
 

data/lib/neg/errors.rb

@@ -0,0 +1,66 @@
+#--
+# Copyright (c) 2012-2013, John Mettraux, jmettraux@gmail.com
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+#
+# Made in Japan.
+#++
+
+
+module Neg
+
+  class NegError < StandardError; end
+
+  class UnconsumedInputError < NegError; end
+  class ParserError < NegError; end
+
+  class ParseError < NegError
+
+    attr_reader :tree
+
+    def initialize(tree)
+
+      @tree = tree
+      @nodes = list_nodes(tree)
+
+      super(deepest_error[3])
+    end
+
+    def errors
+
+      @nodes.select { |n| n[2] == false && n[3].is_a?(String) }
+    end
+
+    def deepest_error
+
+      errors.inject { |e, n| e[1][0] < n[1][0] ? n : e }
+    end
+
+    protected
+
+    def list_nodes(start, accumulator=[])
+
+      accumulator << start
+      start[4].each { |n| list_nodes(n, accumulator) }
+
+      accumulator
+    end
+  end
+end
+

data/lib/neg/input.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2012-2012, John Mettraux, jmettraux@gmail.com
+# Copyright (c) 2012-2013, John Mettraux, jmettraux@gmail.com
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal

data/lib/neg/parser.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2012-2012, John Mettraux, jmettraux@gmail.com
+# Copyright (c) 2012-2013, John Mettraux, jmettraux@gmail.com
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -22,20 +22,29 @@
 # Made in Japan.
 #++
 
-require 'neg/version'
 require 'neg/input'
+require 'neg/errors'
+require 'neg/translator'
 
 
 module Neg
 
-  class UnconsumedInputError < StandardError; end
-  class ParseError < StandardError; end
-
   class Parser
 
     def self.`(s)      ; StringParser.new(s); end
     def self._(c=nil)  ; CharacterParser.new(c); end
 
+    def self.parser(&block)
+
+      self.instance_eval(&block)
+    end
+
+    def self.translator(&block)
+
+      @translator = Class.new(Neg::Translator)
+      @translator.instance_eval(&block)
+    end
+
     def self.method_missing(m, *args)
 
       return super if args.any?
@@ -44,22 +53,30 @@ module Neg
       @root ||= m
       pa = NonTerminalParser.new(m)
 
-      (class << self; self; end).send(:define_method, m) { pa }
+      (class << self; self; end).__send__(:define_method, m) { pa }
 
       pa
     end
 
-    def self.parse(s)
+    def self.parse(s, opts={})
 
       i = Neg::Input(s)
 
-      result = send(@root).parse(i)
+      result = __send__(@root).parse(i, opts)
 
       raise UnconsumedInputError.new(
         "remaining: #{i.remains.inspect}"
       ) if result[2] && ( ! i.eoi?)
 
-      result
+      if @translator && opts[:translate] != false
+        if result[2]
+          @translator.translate(result)
+        else
+          raise ParseError.new(result)
+        end
+      else
+        result
+      end
     end
 
     def self.to_s
@@ -71,12 +88,12 @@ module Neg
         m = method(mname)
 
         next if m.owner == Class
-        next if %w[ _ to_s ].include?(mname.to_s)
+        next if %w[ _ to_s parser translator ].include?(mname.to_s)
         next unless m.arity == (RUBY_VERSION > '1.9' ? 0 : -1)
         next unless m.owner.ancestors.include?(Class)
         next unless m.receiver.ancestors.include?(Neg::Parser)
 
-        s << "  #{send(mname).to_s}"
+        s << "  #{__send__(mname).to_s}"
       end
 
       s << "  root: #{@root}"
@@ -97,15 +114,19 @@ module Neg
       def ~         ; LookaheadParser.new(self, true); end
       def -@        ; LookaheadParser.new(self, false); end
 
-      def parse(input_or_string)
+      def parse(input_or_string, opts)
 
         input = Neg::Input(input_or_string)
         start = input.position
 
-        success, result, children = do_parse(input)
+        success, result, children = do_parse(input, opts)
 
         input.rewind(start) unless success
 
+        #if success && children.size == 1 && children.first[1] == start
+        #  return children.first
+        #end
+
         [ nil, start, success, result, children ]
       end
     end
@@ -123,39 +144,38 @@ module Neg
         @child = pa
       end
 
-#      def reduce(children_results)
-#
-#        children_results.collect { |cr|
-#          if cr[0] && cr[0] != :digit
-#            false
-#          elsif cr[2]
-#            cr[3] ? cr[3] : reduce(cr[4])
-#          else
-#            nil
-#          end
-#        }.flatten.compact
-#      end
+      def reduce(children_results)
+
+        children_results.collect { |cr|
+          if cr[0] && cr[0].to_s.match(/^_/).nil?
+            false
+          elsif cr[2]
+            cr[3] ? cr[3] : reduce(cr[4])
+          else
+            nil
+          end
+        }.flatten.compact
+      end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
-        raise ParseError.new("\"#{@name}\" is missing") if @child.nil?
+        raise ParserError.new("\"#{@name}\" is missing") if @child.nil?
 
-        r = @child.do_parse(i)
+        r = @child.do_parse(i, opts)
 
-        return r
+        return r if r[0] == false
+        return r if r[1].is_a?(String)
 
-#        return r if r[0] == false
-#
-#        report = reduce(r[2])
-#
-#        return r if report.include?(false)
-#
-#        [ true, report.join, [] ]
+        report = reduce(r[2])
+
+        return r if report.include?(false)
+
+        [ true, report.join, opts[:noreduce] ? r[2] : [] ]
       end
 
-      def parse(input_or_string)
+      def parse(input_or_string, opts)
 
-        r = super(input_or_string)
+        r = super
         r[0] = @name
 
         r
@@ -189,12 +209,12 @@ module Neg
         end
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         rs = []
 
         loop do
-          r = @child.parse(i)
+          r = @child.parse(i, opts)
           break if ! r[2] && rs.size >= @min && (@max.nil? || rs.size <= @max)
           rs << r
           break if ! r[2]
@@ -219,7 +239,7 @@ module Neg
         @s = s
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         if (s = i.read(@s.length)) == @s
           [ true, @s, [] ]
@@ -242,7 +262,7 @@ module Neg
         @r = Regexp.new(c ? "[#{c}]" : '.')
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         if (s = i.read(1)).match(@r)
           [ true, s, [] ]
@@ -274,13 +294,13 @@ module Neg
         self
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         results = []
 
         @children.each do |c|
 
-          results << c.parse(i)
+          results << c.parse(i, opts)
           break unless results.last[2]
         end
 
@@ -302,15 +322,17 @@ module Neg
         self
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         results = []
 
         @children.each { |c|
-          results << c.parse(i)
+          results << c.parse(i, opts)
           break if results.last[2]
         }
 
+        results = results[-1, 1] if results.last[2] && ! opts[:noreduce]
+
         [ results.last[2], nil, results ]
       end
 
@@ -328,18 +350,18 @@ module Neg
         @presence = presence
       end
 
-      def do_parse(i)
+      def do_parse(i, opts)
 
         start = i.position
 
-        r = @child.parse(i)
+        r = @child.parse(i, opts)
         i.rewind(start)
 
         success = r[2]
         success = ! success if ! @presence
 
         result = if success
-          nil
+          '' # for NonTerminal#reduce not to continue
         else
           [
             @child.to_s(nil), 'is not', @presence ? 'present' : 'absent'