rockit 0.7.1 → 0.7.2

data/README

@@ -1,172 +1,36 @@
-= Rockit -- Ruby Object-oriented Compiler construction toolKIT
+This is a preview release of the upcoming Rockit 0.8.0. It has 
+*VERY LITTLE* in the form of documentation and examples so it is mainly for
+*VERY INTERESTED* inidividuals who want to check out what is coming. You are
+also encouraged to give feedback on the "pure-Ruby" way of specing grammars.
 
-This package contains Rockit an object-oriented compiler construction 
-toolKIT in/for Ruby.
+Lots of stuff that will be in 0.8.0 is not in this release:
 
-NOTE! This README is incomplete and will be updated for the 0.7 release!!
+  * Proper memoization (à la packrat)
+  * Optimization
+  * Error reporting
+  * Java grammar
+  * Ruby grammar
+  * many of the good stuff from older Rockit versions (tree pattern matching, fully automated AST generation etc)
+  * GPL only on things related to generation, MIT/BSD on everything needed for
+      using the generated parsers etc (as much as possible)
 
-Rockit has the following features:
+so *BE WARNED*... ;)
 
-* Scannerless Generalized LR Parsing with a fast C backend
-   * Means: Can parse any context-free grammar and is still pretty fast
-* Grammars are specified in Ruby so no special Grammar file format
-   * Means: You can use your ordinary Ruby skills when writing grammars (=>
-            shorter, more easily understandable, and powerful grammars)
-* The actions taken during parsing can be easily changed by specifying Handler
-    objects.
-* Rockit can directly build the AST (Abstract Syntax Tree) for you
-* AST classes can be pattern matched
-* Rockit has a library for pretty-printing AST's
+----------------------------------------------------------
 
-Svn id: $Id: README 131 2004-09-03 12:11:29Z robert_feldt $
+If you still want to check this out I suggest you start with the example in
 
-== Download
+tests/acceptance/packrat/minibasic
 
-The latest version of rockit can be found at
+which is a grammar and interpreter for a mini version of Basic.
 
-* http://www.pronovomundo.com/projects/rockit
+You might also get some background by checking out the starts of a paper
+describing Rockit (in file rockit_paper.pdf).
 
-Documentation can be found at 
+----------------------------------------------------------
 
-* http://www.pronovomundo.com/projects/rockit/svn/doc
+If you have comments I'd appreciate if you send them to me at:
 
-== Installation
+robert.feldt@gmail.com
 
-=== Normal Installation
-
-You can install rake with the following command.
-
-  % rake install
-
-from the top of its distribution directory. For this to work you need to have
-rake installed and a C compiler (that Ruby knows of). If you don't have the 
-latter you can install from prebuilt binaries, see below.
-
-=== Installation from prebuilt binaries
-
-To simplify installation rockit comes with prebuilt extensions so that you
-can use it even if you have no C compiler installed. Rockit currently
-comes with prebuilt extensions for cygwin and mingw32. Since the mswin32
-platform uses the same format as the mingw32 platform you can use the
-mingw32 files with the Ruby one-click installer.
-
-To install from the prebuilt binaries for your platform and ruby version:
-
-  % rake install_prebuilt
-  % rake install_lib
-
-Be sure to test your installation and report any problems since these
-prebuilt installations have not had extensive testing.
-
-=== Test the installation
-
-You can test the installed rockit with the following command:
-
-  % rake test
-
-from the top of its distribution directory. For this to work you need to have
-rake installed.
-
-== Roadmap
-
-* If you want to see how to invoke rockit to generate your own parsers, read on.
-* If you want to see the format of a Rockit grammar file
-  doc/rockitfile.rdoc[http://rockit.rubyforge.org/files/doc/rockitfile_rdoc.html].
-* If you want to see a glossary of terms, see
-  doc/glossary.rdoc[http://rockit.rubyforge.org/files/doc/glossary_rdoc.html].
-
-== Simple Example
-
-Once installed, you can use rockit as follows:
-
-== Other Parser generators...
-
-There are a large number of parser generators available for Ruby.
-Here are links to other projects with similar (and not so similar) goals.
-
-* Racc
-
-== Credits
-
-[<b>asdsad</b>] For the .
-
-== License
-
-Rockit is available under the GPL2 license.
-
-:include: LICENSE
-
-== Support
-
-The Rockit homepage is http://.
-
-For other information, feel free to ask on the ruby-talk mailing list
-(which is mirrored to comp.lang.ruby) or contact
-mailto:robert.feldt@pronovomundo.com.
-
-----
-
-= Usage
-
-Rockit is invoked from the command line using:
-
-   % rockit <options> [targets...]
-
-Options are:
-
-[<tt>--help</tt>  (-H)]
-    Display some help text and exit.
-
-[<tt>--quiet</tt> (-q)]
-    Do not echo Sys commands.
-
-[<tt>--trace</tt> (-t)]
-    Turn on invoke/execute tracing.
-
-[<tt>--usage</tt> (-h)]
-    Display a usage message and exit.
-
-[<tt>--verbose</tt> (-v)]
-    Echo the Sys commands to standard output.
-
-[<tt>--version</tt> (-V)]
-    Display the program version and exit.
-
----
-
-= Rockit grammar Format
-
-See doc/grammar.rdoc[http://www.pronovomundo.com/projects/rockit/doc/grammar_rdoc.html]
-for details on the Rockit grammar format.
-
----
-
-= Other stuff
-
-Author::   Robert Feldt <robert.feldt@pronovomundo.com>
-Address::  Robert Feldt c/o ProNovoMundo
-           Brolyckan 1
-           433 69 S�vedalen
-Requires:: Ruby 1.8.2 or later (might work but not tested with earlier version)
-License::  Copyright (C) 2001-2004 by Robert Feldt
-           Released under GNU General Public License 2. See the LICENSE file
-           included in this distribution.
-
-== License and Warranty
-
-    Rockit - Ruby Object-oriented Compiler construction toolKIT
-    Copyright (C) 2001-2004 by Robert Feldt
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; either version 2 of the License, or
-    (at your option) any later version.
-
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with this program; if not, write to the Free Software
-    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+Thank you for your interest!

data/TODO

@@ -1,30 +1,30 @@
 ------------------------------------------------------------------------------
 New Interpretive Packrat Engine
 ------------------------------------------------------------------------------
-* Change ast so that the constant grammar elements need not be given
-  when creating and ast instance. This should simplify the testing
-  considerably.
+* Simplify the grammar-in-ruby specing.
+  * Need not both rule and prod methods only rule.
+  * Short hand "r" instead of "rule" like in old code
+  * Specify grammars as subclasses to Grammar instead of in a block. This
+      makes constants live in the proper namespace and is cleaner and more
+      semantically closer to Ruby.
 
-* Add more atest_minibasic tests.
+* While porting Java grammar:
+  * What happens if ast trees are only partially specified? Should it mix
+      sexprs with asts? Or should it create anonymous parse trees (ie the
+      class has no fixed name but there is a generic class on which you can 
+      specify the name)?
 
-* Add tests of the MiniBasic interpreter.
+* Separate things out into different logical files depending on purpose.
 
-* Fix the MiniBasic interpreter.
-
-* Add tests for running the MB interpreter from the command line.
-
-* Add ability to run MiniBasic interpreter from command line.
-
-* Start porting the next sample: Java grammar in xtc.lang
+* Then we go on to parse Ruby.
 
-* When we can parse Java, we simplify Rockit so that only this new IPE is
-  used and packed up in a gem.
+* Then we release Rockit with MB and Java as a gem and rockit_ruby as a separate gem (with the ruby-related stuff). Possibly also ruby_parser.rb as a separate standalone file for people who just wants a parser (or generate multiple stand-alonoe files from rockit_ruby which includes different parts of the ruby-specific stuff (parser, pretty-printer, analyzer).
 
-* Start porting C grammar in xtc.
+* Add the simple memoization (always memoize on non-transient prod level)
 
-* Then we go on to parse Ruby.
+* Then we start adding the adaptive optimizations.
 
-* Then we release as a gem. Check if we have a RubyForge account for Rockit.
+* Then we add the automatic creation of pretty-printers
 
 ------------------------------------------------------------------------------
 OLD

data/VERSION

@@ -1 +1 @@
-0.7.1
+0.7.2

data/lib/packrat/grammar.rb

@@ -57,11 +57,12 @@ end
 # a special class for it is so that we can inspect it in a more natural way
 # (as a string instead of a Regexp).
 class Packrat::StringLiteral < Packrat::RegexpLiteral
+  attr_reader :string
   def initialize(str)
     super(Regexp.new(Regexp.escape(str)))
-    @str = str
+    @string = str
   end
-  def inspect; @str.inspect; end
+  def inspect; @string.inspect; end
 end
 
 class Packrat::RuleRef < Packrat::GrammarElement
@@ -99,20 +100,23 @@ end
 # symbol should be parsed.
 class Packrat::Production
   attr_accessor :grammar
-  attr_reader :name, :rhs, :result_modifier
+  attr_reader :name, :rhs, :result_producer
   def initialize(name, rhs)
     @name, @rhs = name, rhs
-    if Packrat::ResultModifier === rhs.last
-      @result_modifier = @rhs.pop
+    if Packrat::ResultProducer === rhs.last
+      @result_producer = @rhs.pop
     else
-      # Default modifier is to create a Sexpr with the production name
+      # Default producer is to create a Sexpr with the production name
       # as the head of the returned array.
-      @result_modifier = Packrat::SexprModifier.new(@name)
+      @result_producer = Packrat::SexprProducer.new(@name)
     end
     @rhs.map! {|e| e.to_packrat_grammar_element}
   end
+  def finalize!
+    @result_producer.production = self
+  end
   def inspect(withLhs = true)
-    rhs = @rhs.map {|e| e.inspect}.join(' ')
+    rhs = @rhs.map {|e| e.hidden ? nil : e.inspect}.compact.join(' ')
     withLhs ? "#{name.to_s} -> " + rhs : rhs
   end
 end
@@ -124,15 +128,16 @@ class Packrat::ErrorReporter < Packrat::GrammarElement
   end
   def parse(parser)
     res = @sub.parse(parser)
-    if res == false
+    if false == res
       lputs "\t\t\t  FAIL #{@sub.inspect}"
       puts ""
     else
-      lputs "    Match #{@sub.inspect}"
+      lputs "    #{parser.pos}: Match #{@sub.inspect}"
       puts ""
     end
     res
   end
+  def inspect; @sub.inspect; end
   def method_missing(method, *args)
     @sub.send(method, *args)
   end
@@ -142,7 +147,7 @@ module Packrat::GrammarBuild
   attr_reader :start
   def start_symbol(name); @start = name; end
   def rules; @rules ||= (Hash.new {|h,k| h[k] = Packrat::Rule.new(k)}); end
-  def rule(name, rhss)
+  def rule(name, *rhss)
     rhss.each {|rhs| prod(name, rhs)}
   end
   def prod(name, rhs)
@@ -160,20 +165,16 @@ module Packrat::GrammarBuild
   # Finalize the building of the grammar by conducting postprocessing.
   def finalize!
     postprocess_set_grammar_on_rules
-    postprocess_create_ast_classes
+    each_prod {|p| p.finalize!}
   end
   def postprocess_set_grammar_on_rules
-    rules.values.each {|r| r.grammar = self}
+    each_prod {|r| r.grammar = self}
   end
-  def postprocess_create_ast_classes
-    each_prod do |p|
-      if Packrat::ASTBuilder === p.result_modifier
-        ast_class(p.result_modifier.name, p)
-      end
-    end
+  def each_rule
+    rules.values.each {|r| yield(r)}
   end
   def each_prod
-    rules.values.each {|r| r.prods.each {|p| yield(p)}}
+    each_rule {|r| r.prods.each {|p| yield(p)}}
   end
 end
 
@@ -225,8 +226,7 @@ class Packrat::Maybe < Packrat::GrammarElement
   end
   def parse(parser)
     res = @sub.parse(parser)
-    return res if res
-    return nil
+    false == res ? nil : res
   end
   def inspect
     "(#{@sub.inspect})?"
@@ -237,39 +237,49 @@ module Packrat::GrammarBuild
   def maybe(element); Packrat::Maybe.new(element); end
 end
 
-# The last element of a prod can be a result modifier that modifies
-# the result to be returned by the prod when parsing.
-class Packrat::ResultModifier
+# The last element of a prod can be a result producer that produces
+# the result to be returned by the prod in case of a successfull parse.
+class Packrat::ResultProducer
+  # Before any results are produced we need to know the prod we are in
+  def production=(prod); @prod = prod; end
+
+  # A ResultProducer returns a result which it then updates. This is needed
+  # since multiple results can be in production at the same time.
+  def new_result; end
+  def update_result(res, subres, elem, index); end
+  def finalize_result(res); res; end
 end
 
 # Create a Sexpr based on the name of the matched production and the
 # result-array.
-class Packrat::SexprModifier < Packrat::ResultModifier
+class Packrat::SexprProducer < Packrat::ResultProducer
   def initialize(name)
     @name = name
   end
-  def modify_result(prod, result)
-    # Add the production name in the front
-    result.unshift @name
-    result
-  end
+  def new_result; [@name]; end
+  def update_result(res, subres, elem, index, nhi); res << subres; end
 end
 
 # Lift one of the sub-results as the result from parsing a production.
-class Packrat::LiftModifier < Packrat::ResultModifier
+# Optionally a block can be given. If so the block will get called with
+# the lifted result and can modify it.
+class Packrat::LiftOneResultProducer < Packrat::ResultProducer
   def initialize(valueIndex, &block)
     @value_index = valueIndex
     @block = block
   end
-  def modify_result(prod, result)
-    extracted_result = result[@value_index]
-    @block? @block.call(extracted_result) : extracted_result
+  def new_result; nil; end
+  def update_result(res, subres, elem, index, nonhiddenIndex)
+    index == @value_index ? subres : res
+  end
+  def finalize_result(res)
+    @block ? @block.call(res) : res
   end
 end
 
 module Packrat::GrammarBuild
-  def sexpr(name); Packrat::SexprModifier.new(name); end
-  def lift(index, &b); Packrat::LiftModifier.new(index, &b); end
+  def sexpr(name); Packrat::SexprProducer.new(name); end
+  def lift(index, &b); Packrat::LiftOneResultProducer.new(index, &b); end
 end
 
 module Packrat::GrammarBuild
@@ -282,7 +292,7 @@ module Packrat::GrammarBuild
       Packrat::RegexpLiteral.new(Regexp.new(re_string))
     else
       name = internal_rule_name() 
-      rule(name, subs.map {|s| [s, lift(0)]})
+      rule(name, *subs.map {|s| [s, lift(0)]})
       Packrat::RuleRef.new(name)
     end
   end
@@ -308,30 +318,40 @@ module Packrat::GrammarBuild
   def eos(); hidden(Packrat::EOS.new); end
 end
 
-class Packrat::ASTBuilder < Packrat::ResultModifier
+# Build AST tree as result of parsing a Production.
+class Packrat::ASTBuilder < Packrat::ResultProducer
   attr_reader :name
-  def initialize(nodeName)
-    @name = nodeName
+  def initialize(nodeName, nameMap = {})
+    @name, @name_map = nodeName, nameMap
   end
-  def modify_result(prod, result)
-    astklass = prod.grammar.ast_class(@name, prod)
-    astklass.new(*result)
+  def production=(prod)
+    super
+    @ast_class = prod.grammar.ast_class(@name, prod, @name_map)
+  end
+
+  def new_result; Array.new; end
+  def update_result(res, subres, elem, index, nhIndex)
+    res << subres unless @ast_class.constant_elem_at?(nhIndex)
+    res
+  end
+  def finalize_result(res)
+    @ast_class.new(res, {:only_nonconstant => true})
   end
 end
 
 module Packrat::GrammarBuild
-  def ast(name)
-    Packrat::ASTBuilder.new(name)
+  def ast(name, options = {})
+    Packrat::ASTBuilder.new(name, options)
   end
 
   # Return the ast class with the given <nodeName> for the given <production>.
   # If not previously created we create it and add it to the Tree module.
-  def ast_class(name, prod)
+  def ast_class(name, prod, nameMap)
     acn = ast_class_name(name)
     begin
       const_get("ASTs").const_get(acn)
     rescue
-      const_get("ASTs").const_set(acn, make_ast_class(acn, prod))
+      const_get("ASTs").const_set(acn, make_ast_class(acn, prod, nameMap))
     end
   end
 
@@ -340,8 +360,8 @@ module Packrat::GrammarBuild
     s[0,1].upcase + s[1..-1]
   end
 
-  def make_ast_class(klassName, production)
-    Packrat::AST.new_subclass(klassName, production)
+  def make_ast_class(klassName, production, nameMap)
+    Packrat::AST.new_subclass(klassName, production, nameMap)
   end
 end
 
@@ -349,9 +369,13 @@ end
 class Packrat::AST
   class <<self
     attr_accessor :sig
-    def new_subclass(name, production)
+    # Create a new AST subclass. The <nameMap> hash can specify names
+    # for certain element indices (such explicitly specified names
+    # will override the default naming scheme which is to use a downcase
+    # version of the production name).
+    def new_subclass(name, production, nameMap = {})
       klass = Class.new(self)
-      klass.sig = extract_sig(production)
+      klass.sig = extract_sig(production, nameMap)
       # Add accessor methods for all symbols in the sig
       num_strings = 0
       klass.sig.each_with_index do |sn, i|
@@ -374,32 +398,59 @@ class Packrat::AST
     # Return a sig for the given <production>. The sig has strings in the
     # positions where the production rhs has a String or StringLiteral,
     # has symbols in the positions where a rhs element refer to another
-    # production, and has nil in other positions.
-    def extract_sig(production)
-      production.rhs.map do |e|
-        case e
-        when String
-          e
-        when Packrat::StringLiteral
-          e.inspect # gives us the string itself
-        when Packrat::RuleRef
-          sub_element_name(e.rule_name)
-        else
-          nil  # Expand this so that names are lifted out of Maybe, and "s" is added when plus and mult etc
+    # production, and has nil in other positions. The <nameMap> can contain
+    # explicit names for certing indices (indices as key and name as symbol 
+    # value).
+    def extract_sig(production, nameMap = {})
+      sig = []
+      production.rhs.each_with_index do |e, i|
+        unless e.hidden
+          case e
+          when String
+            sig << e
+          when Packrat::StringLiteral
+            sig << e.string
+          when Packrat::RuleRef
+            sig << sub_element_name(e.rule_name)
+          else
+            sig << nil  # Expand this so that names are lifted out of Maybe, and "s" is added when plus and mult etc
+          end
         end
       end
+      number_multioccurences(sig).map {|n| nameMap[n] || n}
     end
-    
+
+    def number_multioccurences(sig)
+      num_sigs = sig.inject(Hash.new(0)) {|h, s| h[s] += 1 if Symbol === s; h}
+      counters = Hash.new(0)
+      sig.map do |s|
+        (num_sigs[s] > 1) ? (s.to_s + (counters[s] += 1).to_s).intern : s
+      end
+    end
+  
     def sub_element_name(name)
       parts = name.to_s.split(/([A-Z][a-z0-9]*)/).select {|e| e.length > 0}
       parts.map {|p| p.downcase}.join("_").intern
     end
 
-    def [](*args); new(*args); end
+    def constant_elem_at?(index)
+      self.sig[index].kind_of?(String)
+    end
+
+    def [](*args); new(args); end
   end
-    
-  def initialize(*children)
-    @children = children
+
+  DefaultOptions = {:only_nonconstant => true}
+
+  def initialize(children, options = {})
+    options = DefaultOptions.clone.update(options)
+    if options[:only_nonconstant]
+      @children = self.class.sig.map do |n|
+        n.kind_of?(String) ? n : children.shift
+      end
+    else
+      @children = children
+    end
   end
   attr_reader :children
 
@@ -408,7 +459,7 @@ class Packrat::AST
   def ==(other)
     self.class == other.class && @children == other.children
   end
-  
+
   def inspect
     self.class.inspect.split("::").last + "[" +
       @children.map {|c| c.inspect}.join(", ") + "]"
@@ -423,10 +474,10 @@ class Packrat::InterpretingParser
       klass.grammar = grammar
       klass
     end
-    def parse_string(str)
+    def parse_string(str, startSymbol = nil)
       # We always add a whitespace since StringScanner cannot match /\s*/
       # (typically used as whitespace) at EOS
-      new(str + " ").parse_string
+      new(str + " ").parse_string(startSymbol)
     end
   end
 
@@ -438,8 +489,9 @@ class Packrat::InterpretingParser
     @grammar = self.class.grammar
   end
 
-  def parse_string
-    @grammar.start_rule.parse(self)
+  def parse_string(startSymbol = nil)
+    startSymbol ||= @grammar.start
+    @grammar[startSymbol].parse(self)
   end
 
   # Get and Set current position in string.
@@ -484,25 +536,27 @@ end
 
 class Packrat::Production
   def parse(parser)
-    result = []
-    @rhs.each do |e|
-      res = e.parse(parser)
-      if res == false
-        return false
-      else
-        result << res unless e.hidden
+    res = @result_producer.new_result
+    nonhidden_index = 0
+    @rhs.each_with_index do |e, i|
+      subres = e.parse(parser)
+      return false if false == subres
+      unless e.hidden
+        res = @result_producer.update_result(res, subres, e, 
+                                             i, nonhidden_index)
+        nonhidden_index += 1
       end
     end
-    return @result_modifier.modify_result(self, result)
+    return @result_producer.finalize_result(res)
   end
 end
 
 class Packrat::Rule
   def parse(parser)
+    oldpos = parser.pos
     prods.each do |prod|
-      oldpos = parser.pos
       res = prod.parse(parser)
-      return res if res
+      return res unless false == res
       parser.pos = oldpos
     end
     return false
@@ -519,6 +573,7 @@ class Packrat::Repeat
   def parse(parser)
     result_list = []
     oldpos = parser.pos
+    # XXX: Should we take only amx number of results here if max != false?
     while (res = @sub.parse(parser))
       result_list << res
     end