ghazel-parslet 1.4.0.1

data/HISTORY.txt

@@ -0,0 +1,195 @@
+= 2.0 / ?? (future release changes, like a reminder to self)
+  
+  - prsnt? and absnt? are now finally banned into oblivion. Wasting vocals for
+    the win. 
+    
+= 1.4.0 / 25May2012
+
+  + Revised documentation. A few new API features have finally made it into
+    the documentation. Examples in the documentation are now curated and
+    run against the current code so that they really really work. 
+    Also, the website generation tools have been replaced with 2012-style
+    tools. Much less pain to update now. 
+
+  + Parslet::Source now doesn't hold a StringIO, it directly holds the 
+    buffer to be parsed. The api of Source has changed a tiny bit. This change
+    has been made for speed optimisation reasons.
+
+  + :reporter argument to parse, allowing to customize error reporting within
+    wide boundaries. See issue #64 for a discussion. 
+    Included are two error reporters, one (default) with the existing error
+    tree functionality, one reporting deepest errors as defined by the above
+    ticket.
+
+  + Optimistic parse: Parsing is two phase, with the first phase assuming 
+    there will be no errors. This yields ~ 20% speed improvement in the 
+    case where the parse succeeds.
+    Also, internal error handling is now using tuples. This and other 
+    optimizations have yielded ~ 30% overall improvement. 
+
+  ! #error_tree and #cause removed from all of parslet. The 
+    Parslet::ParseFailed exception now contains a #cause field that can
+    be asked for an #ascii_tree as before. 
+    Cleaner internal error handling, not stateful in atoms anymore. Some 
+    parsers will see correct error reporting for the first time. (issue #65)
+
+  + Made it possible to pass a custom Parslet::Source implementor to #parse.
+    (see #63)
+    
+  + #parse has now a second argument that is an options hash. See 
+    Parslet::Atoms::Base#parse for documentation.
+
+  - VM engine on the way out. No benefit except for the intellectual
+    challenge.
+    
+= 1.3.0 / 5Mar2012
+
+  ! Parslet::Transform::Context is now much more well-behaved. It has
+    #respond_to? and #method_missing; it now looks like a plain old Ruby
+    object with instance variables and attribute readers.
+
+  - Grammar transforms turned out to be a dead end and have been removed. 
+
+  ! A few problems in error message generation have been fixed. This will
+  	improve diagnostics further.
+	
+  + A VM driven parser engine: Removes the limitation that parsing needs a 
+    lot of stack space, something dearly missing from Ruby 1.9.3 fibers.
+    This engine is experimental and might be removed in the future. 
+
+  ! Interaction with mathn fixed - Line number generation will terminate. 
+	
+  . Internal reorganisation, removing cruft and bit rot.
+    
+= 1.2.3 / 22Sep2011
+
+  + Transform#apply can now be called with a hash as second argument. This 
+    provides bindings and a way to inject context.
+
+  ! Fixes a bug thar modified parslet atoms in place, defeating oop chaining. 
+    (#50)
+    
+= 1.2.1 / 6Jun2011
+
+  ! FIX: Input at the end of a parse raises Parslet::UnconsumedInput. (see
+    issue 18)
+
+  ! FIX: Unicode parsing should now work as expected. (see issue 38)
+  
+  ! FIX: Slice#slice returned wrong bits at times (see issue 36).
+
+= 1.2.0 / 4Feb2011
+  
+  + Parslet::Parser is now also a grammar atom, it can be composed freely with
+    other atoms. (str('f') >> MiniLispParser.new >> str('b'))
+    
+  + No strings, only slices are returned as part of the parser result. 
+    Parslet::Slice is almost a string class, but one that remembers the 
+    source offset. This has also bought us a slight speedup.
+    
+  + require 'parslet/convenience' now brings #parse_with_debug to all parslets.
+    This is a consequence of the above change. 
+  
+  + Deprecates prsnt? and absnt? in favor of the more readable absent? and
+    prsnt?. Uses 3 bytes more RAM. The old variants will exist until we release
+    2.0. 
+  
+  INTERNALLY
+  
+  + Visitors now should have methods that all begin with 'visit_*'. #str 
+    becomes #visit_str.
+
+  + Parslet::Atoms::Entity now takes only a block argument instead of context
+    and block. 
+
+= 1.1.1 / 4Feb2011
+
+  ! FIX: Line counting was broken by performance optimisations. 
+  
+  + Squeezed out another few drops of performance. 
+
+= 1.1.0 / 2Feb2011
+  
+  + Uses return (fail/success), cached line counts, memoizing of parse results 
+    and other tricks internally for at least an order of magnitude increase 
+    in execution speed.
+    
+  + str('foo').maybe will now return an empty string again. Use .as(...) to 
+    name things and get back [] from #repeat and nil from #maybe.
+    
+  + If you require 'parslet/atoms/visitor', you'll get an accept method on
+    all known Parslet::Atoms.
+    
+  + If you require 'parslet/export', you can call #to_citrus and #to_treetop
+    to produce string versions of your grammar in those dialects.
+  
+  + Requiring 'parslet/convenience' will given you a parse_with_debug on 
+    your Parslet::Parser class. This prints some diagnostics on parse failure. 
+    (Thanks to Florian Hanke)
+        
+= 1.0.1 / 17Jan2011
+
+  A happy new year!
+  
+  ! FIX: Parslet::Transform was wrongly fixed earlier - it now wont mangle 
+    hashes anymore. (Blake Sweeney)
+    
+  + parslet/rig/rspec.rb contains useful rspec matchers. (R. Konstantin Haase)
+
+= 1.0.0 / 29Dez2010
+
+  - #each_match was removed. There was some duplication of code that even 
+    confused me - and we should not have 2 methods of achieving the same
+    goal. 
+    
+  + Full documentation. Fixed sdoc. 
+
+= 0.11.0 / 25Nov2010
+
+  ! Bugfixes to tree handling. Let's hope that was the last such significant
+    change to the core.
+
+= 0.10.1 / 22Nov2010
+
+  + Allow match['a-z'], shortcut for match('[a-z]')
+
+  ! Fixed output inconsistencies (behaviour in connection to 'maybe')
+
+= 0.10.0 / 22Nov2010
+
+  + Parslet::Transform now takes a block on initialisation, wherein you can
+    define all the rules directly.
+    
+  + Parslet::Transform now only passes a hash to the block during transform
+    when its arity is 1. Otherwise all hash contents as bound as local     
+    variables.
+    
+  + Both inline and other documentation have been improved. 
+  
+  + You can now use 'subtree(:x)' to bind any subtree to x during tree pattern
+    matching. 
+    
+  + Transform classes can now include rules into class definition. This makes
+    Parser and Transformer behave the same. 
+  
+= 0.9.0 / 28Oct2010
+  * More of everything: Examples, documentation, etc...
+
+  * Breaking change: Ruby's binary or ('|') is now used for alternatives, 
+    instead of the division sign ('/') - this reduces the amount of 
+    parenthesis needed for a grammar overall. 
+
+  * parslet.maybe now yields the result or nil in case of parse failure. This
+    is probably better than the array it did before; the jury is still out on
+    that. 
+    
+  * parslet.repeat(min, max) is now valid syntax
+
+= 0.1.0 / not released.
+
+  * Initial version. Classes for parsing, matching in the resulting trees
+    and transforming the trees into something more useful.  
+    
+  * Parses and outputs intermediary trees
+    
+  * Matching of single elements and sequences

data/LICENSE

@@ -0,0 +1,23 @@
+
+ Copyright (c) 2010 Kaspar Schiess
+
+ Permission is hereby granted, free of charge, to any person
+ obtaining a copy of this software and associated documentation
+ files (the "Software"), to deal in the Software without
+ restriction, including without limitation the rights to use,
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following
+ conditions:
+
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,70 @@
+INTRODUCTION
+
+Parslet makes developing complex parsers easy. It does so by
+
+* providing the best error reporting possible
+* not generating reams of code for you to debug
+
+Parslet takes the long way around to make your job easier. It allows for
+incremental language construction. Often, you start out small, implementing
+the atoms of your language first; _parslet_ takes pride in making this
+possible.
+
+Eager to try this out? Please see the associated web site:
+http://kschiess.github.com/parslet
+
+SYNOPSIS
+
+  require 'parslet'
+  include Parslet
+
+  # parslet parses strings
+  str('foo').
+    parse('foo') # => "foo"@0
+
+  # it matches character sets
+  match['abc'].parse('a') # => "a"@0
+  match['abc'].parse('b') # => "b"@0
+  match['abc'].parse('c') # => "c"@0
+
+  # and it annotates its output
+  str('foo').as(:important_bit).
+    parse('foo') # => {:important_bit=>"foo"@0}
+
+  # you can construct parsers with just a few lines
+  quote = str('"')
+  simple_string = quote >> (quote.absent? >> any).repeat >> quote
+
+  simple_string.
+    parse('"Simple Simple Simple"') # => "\"Simple Simple Simple\""@0
+
+  # or by making a fuss about it 
+  class Smalltalk < Parslet::Parser
+    root :smalltalk
+
+    rule(:smalltalk) { statements }
+    rule(:statements) { 
+      # insert smalltalk parser here (outside of the scope of this readme)
+    }
+  end
+
+  # and then
+  Smalltalk.new.parse('smalltalk')
+
+COMPATIBILITY
+
+This library should work with most rubies. I've tested it with MRI 1.8 
+(except 1.8.6), 1.9, rbx-head, jruby. Please report as a bug if you encounter 
+issues.
+
+Note that due to Ruby 1.8 internals, Unicode parsing is not supported on that 
+version. 
+
+On Mac OS X Lion, ruby-1.8.7-p352 has been known to segfault. Use
+ruby-1.8.7-p334 for better results.
+
+STATUS 
+
+At version 1.4.0 - See HISTORY.txt for changes.
+
+(c) 2010, 2011, 2012 Kaspar Schiess

data/Rakefile

@@ -0,0 +1,49 @@
+require 'rdoc/task'
+require 'sdoc'
+
+require 'rspec/core/rake_task'
+require "rubygems/package_task"
+
+desc "Run all tests: Exhaustive."
+RSpec::Core::RakeTask.new
+
+namespace :spec do
+  desc "Only run unit tests: Fast. "
+  RSpec::Core::RakeTask.new(:unit) do |task|
+    task.pattern = "spec/parslet/**/*_spec.rb"
+  end
+end
+
+task :default => :spec
+
+# Generate documentation
+RDoc::Task.new do |rdoc|
+  rdoc.title    = "parslet - construction of parsers made easy"
+  rdoc.options << '--line-numbers'
+  rdoc.options << '--fmt' << 'shtml' # explictly set shtml generator
+  rdoc.template = 'direct' # lighter template used on railsapi.com
+  rdoc.main = "README"
+  rdoc.rdoc_files.include("README", "lib/**/*.rb")
+  rdoc.rdoc_dir = "rdoc"
+end
+
+desc 'Clear out RDoc'
+task :clean => [:clobber_rdoc, :clobber_package]
+
+# This task actually builds the gem. 
+task :gem => :spec
+spec = eval(File.read('parslet.gemspec'))
+
+desc "Generate the gem package."
+Gem::PackageTask.new(spec) do |pkg|
+  pkg.gem_spec = spec
+end
+
+desc "Prints LOC stats"
+task :stat do
+  %w(lib spec example).each do |dir|
+    loc = %x(find #{dir} -name "*.rb" | xargs wc -l | grep 'total').split.first.to_i
+    printf("%20s %d\n", dir, loc)
+  end
+end
+

data/example/boolean_algebra.rb

@@ -0,0 +1,70 @@
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require "parslet"
+require "pp"
+
+# Parses strings like "var1 and (var2 or var3)" respecting operator precedence
+# and parentheses. After that transforms the parse tree into an array of
+# arrays like this:
+#
+# [["1", "2"], ["1", "3"]]
+#
+# The array represents a DNF (disjunctive normal form). Elements of outer
+# array are connected with "or" operator, while elements of inner arrays are
+# joined with "and".
+#
+class Parser < Parslet::Parser
+  rule(:space)  { match[" "].repeat(1) }
+  rule(:space?) { space.maybe }
+
+  rule(:lparen) { str("(") >> space? }
+  rule(:rparen) { str(")") >> space? }
+
+  rule(:and_operator) { str("and") >> space? }
+  rule(:or_operator)  { str("or")  >> space? }
+
+  rule(:var) { str("var") >> match["0-9"].repeat(1).as(:var) >> space? }
+
+  # The primary rule deals with parentheses.
+  rule(:primary) { lparen >> or_operation >> rparen | var }
+
+  # Note that following rules are both right-recursive.
+  rule(:and_operation) { 
+    (primary.as(:left) >> and_operator >> 
+      and_operation.as(:right)).as(:and) | 
+    primary }
+    
+  rule(:or_operation)  { 
+    (and_operation.as(:left) >> or_operator >> 
+      or_operation.as(:right)).as(:or) | 
+    and_operation }
+
+  # We start at the lowest precedence rule.
+  root(:or_operation)
+end
+
+class Transformer < Parslet::Transform
+  rule(:var => simple(:var)) { [[String(var)]] }
+
+  rule(:or => { :left => subtree(:left), :right => subtree(:right) }) do
+    (left + right)
+  end
+
+  rule(:and => { :left => subtree(:left), :right => subtree(:right) }) do
+     res = []
+     left.each do |l|
+       right.each do |r|
+         res << (l + r)
+       end
+     end
+     res
+  end
+end
+
+pp tree = Parser.new.parse("var1 and (var2 or var3)")
+# {:and=>
+#   {:left=>{:var=>"1"@3},
+#    :right=>{:or=>{:left=>{:var=>"2"@13}, :right=>{:var=>"3"@21}}}}}
+pp Transformer.new.apply(tree)
+# [["1", "2"], ["1", "3"]]
+

data/example/calc.rb

@@ -0,0 +1,153 @@
+# A simple integer calculator to answer the question about how to do 
+# left and right associativity in parslet (PEG) once and for all. 
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require 'rspec'
+require 'parslet'
+require 'parslet/rig/rspec'
+
+# This is the parsing stage. It expresses left associativity by compiling
+# list of things that have the same associativity. 
+class CalcParser < Parslet::Parser
+  root :addition
+  
+  rule(:addition) {
+    multiplication.as(:l) >> (add_op >> multiplication.as(:r)).repeat(1) |
+    multiplication
+  }
+  
+  rule(:multiplication) { 
+    integer.as(:l) >> (mult_op >> integer.as(:r)).repeat(1) |
+    integer }
+  
+  rule(:integer) { digit.repeat(1).as(:i) >> space? }
+
+  rule(:mult_op) { match['*/'].as(:o) >> space? }
+  rule(:add_op) { match['+-'].as(:o) >> space? }
+
+  rule(:digit) { match['0-9'] }
+  rule(:space?) { match['\s'].repeat }
+end
+
+# Classes for the abstract syntax tree.
+Int    = Struct.new(:int) {
+  def eval; self end
+  def op(operation, other)
+    left = int
+    right = other.int 
+
+    Int.new(
+      case operation
+        when '+'
+          left + right
+        when '-'
+          left - right
+        when '*'
+          left * right
+        when '/'
+          left / right
+      end)
+  end
+  def to_i
+    int
+  end
+}
+Seq    = Struct.new(:sequence) {
+  def eval
+    sequence.reduce { |accum, operation| 
+      operation.call(accum) }
+  end
+}
+LeftOp = Struct.new(:operation, :right) {
+  def call(left)
+    left = left.eval
+    right = self.right.eval
+
+    left.op(operation, right)
+  end
+}
+
+# Transforming intermediary syntax tree into a real AST.
+class CalcTransform < Parslet::Transform
+  rule(i: simple(:i)) { Int.new(Integer(i)) }
+  rule(o: simple(:o), r: simple(:i)) { LeftOp.new(o, i) }
+  rule(l: simple(:i)) { i }
+  rule(sequence(:seq)) { Seq.new(seq) }
+end
+
+# And this calls everything in the right order.
+def calculate(str)
+  intermediary_tree = CalcParser.new.parse(str)
+  abstract_tree = CalcTransform.new.apply(intermediary_tree)
+  result = abstract_tree.eval
+  
+  result.to_i
+end
+
+# A test suite for the above parser
+describe CalcParser do
+  let(:p) { described_class.new }
+  describe '#integer' do
+    let(:i) { p.integer }
+    it "parses integers" do
+      i.should parse('1')
+      i.should parse('123')
+    end 
+    it "consumes trailing white space" do
+      i.should parse('123   ')
+    end 
+    it "doesn't parse floats" do
+      i.should_not parse('1.3')
+    end 
+  end
+  describe '#multiplication' do
+    let(:m) { p.multiplication }
+    it "parses simple multiplication" do
+      m.should parse('1*2')
+    end
+    it "parses division" do
+      m.should parse('1/2')
+    end 
+  end
+  describe '#addition' do
+    let(:a) { p.addition }
+    
+    it "parses simple addition" do
+      a.should parse('1+2')
+      a.should parse('1+2+3-4')
+    end 
+  end
+end
+describe CalcTransform do
+  def t(obj)
+    described_class.new.apply(obj)
+  end
+  
+  it "transforms integers" do
+    t(i: '1').should == Int.new(1)
+  end 
+  it "unwraps left operand" do
+    t(l: :obj).should == :obj
+  end 
+end
+describe 'whole computation specs' do
+  def self.result_of(str, int)
+    it(str) { calculate(str).should == int } 
+  end
+
+  result_of '1+1', 2
+  result_of '1-1-1', -1
+  result_of '1+1+3*5/2', 9
+  result_of '123*2', 246
+end
+
+
+# Enable these if you want to change the code.
+# RSpec::Core::Runner.run([], $stderr, $stdout)
+
+str = ARGV.join
+str = '123*2' if str.match(/^\s*$/)
+
+print "#{str} (command line): -> "
+puts calculate(str)