redparse 0.8.3 → 0.8.4

data/History.txt

@@ -1,4 +1,63 @@
-=== 0.8.3 / 2009-07-01
+=== 0.8.4 / 21dec2009
+* 5 Major Enhancements:
+  * OpNode and related modules are now classes
+  * parse results are now cached -> substantial speedup on reparse
+  * moderate performance improvements for regular parser too
+  * inspect now dumps node trees in more readable tree-like output
+  * tests now ignore (with a warning) differences in just a :begin node
+
+* 18 Minor Enhancements:
+  * single code path utility now converts bare => in calls and between [ and ]
+  * reworked the way ternary rescue is parsed
+  * new build script & gemspec
+  * better way to deal with default of :rubyversion parser option
+  * various fixes to xform_tree! rewriter utility (still doesn't work, tho)
+  * improvements to constructors to make creating nodes by hand more pleasant
+  * parser now creates nodes via Node.create
+  * use AssignmentRhsListStart/EndToken to delimit right hand sides
+  * lhs* and rhs* should be considered unary ops again
+  * when parens in assign lhs, treat unary* and single arg like no parens
+  * VarNode#ident is now kept in a slot, not an ivar
+  * force body of a block to always be a SequenceNode
+  * added RedParse::Nodes; include it to get all the redparse node classes
+  * have each node class remember a list of its slot names
+  * added aliases and accessors in various nodes to make the api nicer
+  * moved some utilities into the support libraries where they belong
+  * slight improvements to parser compiler
+  * added a version of depthwalk which just visits the Nodes of the tree
+
+* 18 Bugfixes:
+  * parser now runs under MRI 1.9
+  * (more?) accurate version of Float#to_s, for 1.8
+  * minor tweaks to #unparse
+  * value of () is nil, not false
+  * get redparse/version.rb relative to current directory from gemspec
+  * when comparing trees, more insignificant differences are ignored
+  * Node#deep_copy makes more faithful copies now
+  * node marshalling should be more reliable
+  * tweaks to parse_tree support to improve conformance
+  * support automagicness of integer&regexp in flipflop (in parse_tree output)
+  * parse_tree's placement of :begin nodes is somewhat better emulated
+  * always put parse inputs into binary mode
+  * changed some operators (lhs, rescue3 unary* rhs*) to proper precedence
+  * numeric literals inserted directly in parsetrees should be autoquoted
+  * ensure @lhs_parens set in AssignNode when it should be
+  * make sure ListInNode is extended into arrays added to Nodes via writers
+  * permit empty symbol LiteralNode to be made
+  * fixed bad permissions in gem file
+
+* 9 Changes To Tests:
+  * test Node trees surviving Marshal/Ron round-trip and deep_copy unscathed
+  * tests for many of the new 1.9 syntax constructions
+  * parse_tree server process now started in a more portable way
+  * lots of new test cases
+  * rp-locatetest now has docs on how to use it
+  * keep track of problematic files if even the slightest problem occurs
+  * enable/disable fuzzing with ENV var rather than comments
+  * make sure inputs are unchanged by parse
+  * better organized some of the known failing testcases
+
+=== 0.8.3 / 1jul2009
 * 7 Minor Enhancements:
   * most nodes now have startline, endline, and offset
   * Node.[] is more fun to use now (most attributes can be left off)
@@ -12,7 +71,7 @@
   * more faithful ParseTree emulation in a couple of cases
   * minor fixes in node processing for some rare conditions 
 
-=== 0.8.2 / 2009-04-21
+=== 0.8.2 / 21apr2009
 * 14 Minor Enhancements:
   * should be no tokens left in Node tree now
   * lots of code trying to make things faster (no luck so far)
@@ -43,14 +102,14 @@
   * (so escape translation has to be done later, when Nodes are used)
   * #to_parsetree of an empty source text should be happier now
 
-=== 0.8.1 / 2009-04-21
+=== 0.8.1 / 21apr2009
 * 4 Minor Enhancements:
   * fixed embarassing permissions problems
   * version.rb and History.txt were missing from the release
   * I left off the dependancy on reg, oops!
   * hacked up tests to ignore sudden problem with extra nils
 
-=== 0.8.0 / 2008-10-10
+=== 0.8.0 / 10oct2008
 * 1 Major Enhancement:
   * Birthday!
 

data/Makefile

@@ -0,0 +1,43 @@
+name=RedParse
+lname=redparse
+gemname=redparse
+
+#everything after this line is generic
+
+version=$(shell ruby -r ./lib/$(lname)/version.rb -e "puts $(name)::VERSION")
+filelist=$(shell git ls-files)
+
+.PHONY: all test docs gem tar pkg email
+all: test
+
+test:
+	ruby -Ilib test/test_all.rb
+
+docs:
+	rdoc lib/*
+
+pkg: gem tar
+
+gem:
+	gem build $(lname).gemspec
+
+tar:
+	tar cf - $(filelist) | ( mkdir $(gemname)-$(version); cd $(gemname)-$(version); tar xf - )
+	tar czf $(gemname)-$(version).tar.gz $(gemname)-$(version)
+	rm -rf $(gemname)-$(version)
+
+email: README.txt History.txt
+	ruby -e ' \
+  require "rubygems"; \
+  load "./$(lname).gemspec"; \
+  spec= Gem::Specification.list.find{|x| x.name=="$(gemname)"}; \
+  puts "\
+Subject: [ANN] $(name) #{spec.version} Released \
+\n\n$(name) version #{spec.version} has been released! \n\n\
+#{Array(spec.homepage).map{|url| " * #{url}\n" }} \
+ \n\
+#{$(name)::Description} \
+\n\nChanges:\n\n \
+#{$(name)::Latest_changes} \
+"\
+'

data/README.txt

@@ -10,9 +10,11 @@ tool is LALR(1)-equivalent and the 'parse language' is pretty nice,
 even in it's current crude form.)
 
 My intent is to have a completely correct parser for ruby, in 100% 
-ruby. It's not all there yet, but I'm getting pretty close. Currently, 
-RedParse can parse slightly in excess of 99% of ruby files found in 
-the wild. For known problems, see below. 
+ruby. Currently, RedParse can parse all known ruby 1.8 constructions
+correctly. There might be some problems with unparsing or otherwise 
+working with texts in a character set other than ascii. Some of the
+new ruby 1.9 constructions are supported in 1.9 mode. For more
+details on known problems, see below. 
 
 == REQUIREMENTS:
 * RedParse requires RubyLexer, my hand-coded lexer for ruby. It also 
@@ -41,6 +43,7 @@ Please see COPYING.LGPL for details.
 * Pretty AST trees (at least, I think so). (To program for, not
   necessarily to look at.)
 * AST trees closely mirror the actual structure of source code.
+* unparser is built in
 * ParseTree format output too, if you want that.
 * Did I mention that there's no YACC at all? YACC grammars are 
   notoriously difficult to modify, (I've never successfully done it) 
@@ -63,7 +66,6 @@ Please see COPYING.LGPL for details.
 * Pathetically, ridiculously slow (ok, compiler-compilers are hard...)
 * Error handling is very minimal right now.
 * No warnings at all.
-* Some expressions aren't parsed correctly. see below.
 * Unit test takes a fairly long time.
 * Lots of warnings printed during unit test.
 * Debugging parse rules is not straightforward.
@@ -71,6 +73,8 @@ Please see COPYING.LGPL for details.
 * "loosey-goosey" parser happily parses many expressions which normal 
   ruby considers errors.
 
+
+
 == SYNOPSIS:
 
   #simple example of usage:
@@ -140,219 +144,150 @@ existing format in the future, but no incompatibility-creating changes.
  UnAmpNode means UnOpNode with op == "&"
 
  Node<Array        #abstract ancestor of all nodes
- +RescueNode       #a rescue clause in a def or begin statement
+ +-RescueNode      #a rescue clause in a def or begin statement
  |                  (exceptions: Array[Expr*], varname: VarNode|nil, action: Expr)
- +WhenNode         #a when clause in a case statement
+ +-WhenNode        #a when clause in a case statement
  |                  (when: Expr|Array[Expr+]  then: Expr|nil )
- +ElsifNode        #an elsif clause in an if statement
+ +-ElsifNode       #an elsif clause in an if statement
  |                  (elsif: Expr, then: Expr|nil) 
- +ValueNode        #abstract, a node which has a value (an expression)
- |+VarNode  #represents variables and constants
+ +-ValueNode       #abstract, a node which has a value (an expression)
+ |+-VarNode  #represents variables and constants
  ||                 (ident: String)
- |+ListOpNode      #abstract, ancestor for nodes which are lists of 
+ |+-ListOpNode     #abstract, ancestor for nodes which are lists of 
  |||               #things separated by some op
- ||+SequenceNode   #a sequence of statements
+ ||+-SequenceNode  #a sequence of statements
  |||                 (Array[Expr*])
- ||+ConstantNode   #a constant expression of the form A::B::C or the like
+ ||+-ConstantNode  #a constant expression of the form A::B::C or the like
  ||                #first expression can be anything
  ||                 (Array[String|Expr|nil,String+])
- |+RawOpNode       #ancestor of all operators (but not . :: ; , ?..:)
+ |+-RawOpNode      #ancestor of all binary operators (except . :: ; , ?..:)
  |||                (left: Expr, op: String, right: Expr)
- ||+OpNode         #ancestor of some operators
- |||+RangeNode     #a range literal node
- |||+KeywordOpNode #abstract, ancestor of keyword operators
- ||||+LogicalNode  #and or && || expressions
- ||||+WhileOpNode  #while as an operator
- ||||+UntilOpNode  #until as an operator
- ||||+IfOpNode     #if as an operator
- ||||+UnlessOpNode #unless as an operator
- ||||+RescueOpNode #rescue as an operator
- ||||               (body: Expr, rescues: Array[RescueNode*])
- |||+NotEqualNode  #!= expressions
- |||+MatchNode     #=~ expressions
- |||+NotMatchNode  #!~ expressions
- |+LiteralNode     #literal symbols, integers
+ ||+-RangeNode     #a range literal node
+ ||+-KeywordOpNode #abstract, ancestor of keyword operators
+ |||+-LogicalNode  #and or && || expressions
+ |||+-WhileOpNode  #while as an operator
+ |||+-UntilOpNode  #until as an operator
+ |||+-IfOpNode     #if as an operator
+ |||+-UnlessOpNode #unless as an operator
+ |||+-RescueOpNode #rescue as an operator
+ |||                (body: Expr, rescues: Array[RescueNode*])
+ ||+-OpNode        #ancestor of some binary operators (those with methods hidden in them)
+ |||+-NotEqualNode #!= expressions
+ |||+-MatchNode    #=~ expressions
+ |||+-NotMatchNode #!~ expressions
+ |+-LiteralNode    #literal symbols, integers
  ||                 (val: Numeric|Symbol|StringNode)
- |+StringNode      #literal strings
+ |+-StringNode     #literal strings
  |||                (Array[(String|Expr)+])
- ||+HereDocNode    #here documents
- |+StringCatNode   #adjacent strings are catenated ("foo" "bar" == "foobar")
+ ||+-HereDocNode   #here documents
+ |+-StringCatNode  #adjacent strings are catenated ("foo" "bar" == "foobar")
  ||                 (Array[StringNode+])
- |+NopNode         #an expression with no tokens at all in it
+ |+-NopNode        #an expression with no tokens at all in it
  ||                 (no attributes)
- |+VarLikeNode     #nil,false,true,__FILE__,__LINE__,self
+ |+-VarLikeNode    #nil,false,true,__FILE__,__LINE__,self
  ||                 (name: String)
- |+UnOpNode        #unary operators
+ |+-UnOpNode       #unary operators
  ||                 (op: String, val: Expr)
- ||+UnaryStarNode  #unary star (splat)
- |||+DanglingStarNode  #unary star with no argument
- |||||                    (no attributes)
- ||||+DanglingCommaNode  #comma with no rhs
- ||                       (no attributes)
- |+BeginNode       #begin..end block
+ ||+-UnaryStarNode #unary star (splat)
+ |||+-DanglingStarNode  #unary star with no argument
+ |||||                     (no attributes)
+ ||||+-DanglingCommaNode  #comma with no rhs
+ ||                        (no attributes)
+ |+-BeginNode      #begin..end block
  ||                 (body: Expr|nil, rescues: Array[RescueNode*], 
  ||                  else: Expr|nil, ensure: Expr|nil) 
- |+ParenedNode     #parenthesized expressions 
+ |+-ParenedNode    #parenthesized expressions 
  ||                 (body: Expr)
- |+AssignNode      #assignment (including eg +=)
+ |+-AssignNode     #assignment (including eg +=)
  ||                 (left:AssigneeList|LValue, op:String ,right:Array[Expr*]|Expr)
- |+AssigneeList    #abstract, comma-delimited list of assignables
+ |+-AssigneeList   #abstract, comma-delimited list of assignables
  |||                (Array[LValue*])
- ||+NestedAssign   #nested lhs, in parentheses
- ||+MultiAssign    #regular top-level lhs
- ||+BlockParams    #block formal parameter list
- |+CallSiteNode    #abstract, method calls
+ ||+-NestedAssign  #nested lhs, in parentheses
+ ||+-MultiAssign   #regular top-level lhs
+ ||+-BlockParams   #block formal parameter list
+ |+-CallSiteNode   #abstract, method calls
  |||                (receiver: Expr|nil, name: String, params: nil|Array[Expr+,UnaryStarNode?,UnAmpNode?], 
  |||                 blockparams: BlockParams|nil, block: Expr|nil)
- ||+CallNode       #normal method calls
- ||+KWCallNode     #keywords that look (more or less) like methods (BEGIN END yield return break continue next)
- |+ArrayLiteralNode #[..] 
- ||                 (Array[Expr*])
- |+IfNode          #if..end and unless..end
+ ||+-CallNode      #normal method calls
+ ||+-KWCallNode    #keywords that look (more or less) like methods (BEGIN END yield return break continue next)
+ |+-ArrayLiteralNode #[..] 
+ ||                   (Array[Expr*])
+ |+-IfNode         #if..end and unless..end
  ||                 (if: Expr, then: Expr|nil, elsifs: Array[ElsifNode+]|nil, else: Expr|nil)
- |+LoopNode        #while..end and until..end
+ |+-LoopNode       #while..end and until..end
  ||                 (while: Expr, do: Expr:nil)
- |+CaseNode        #case..end
+ |+-CaseNode       #case..end
  ||                 (case: Expr|nil, whens: Array[WhenNode*], else: Expr|nil)
- |+ForNode         #for..end
+ |+-ForNode        #for..end
  ||                 (for: LValue, in: Expr, do: Expr|nil)
- |+HashLiteralNode #{..}
- ||                 (Array[Expr*]) (size must be even)
- |+TernaryNode     # ? .. :
+ |+-HashLiteralNode #{..}
+ ||                  (Array[Expr*]) (size must be even)
+ |+-TernaryNode    # ? .. :
  ||                 (if: Expr, then: Expr, else: Expr)
- |+MethodNode      #def..end
+ |+-MethodNode     #def..end
  ||                 (receiver:Expr|nil, name:String, 
  ||                  params:Array[VarNode*,AssignNode*,UnaryStarNode?,UnAmpNode?]|nil, 
  ||                  body: Expr|nil, rescues: Array[RescueNode+]|nil, else: Expr|nil, ensure: Expr|nil)
- |+AliasNode       #alias foo bar
+ |+-AliasNode      #alias foo bar
  ||                 (to: String|VarNode|StringNode, from: String|VarNode|StringNode)
- |+UndefNode       #undef foo
+ |+-UndefNode      #undef foo
  ||                 (Array[String|StringNode+])
- |+NamespaceNode #abstract
- ||+ModuleNode     #module..end
+ |+-NamespaceNode  #abstract
+ ||+-ModuleNode    #module..end
  |||                (name: VarNode|ConstantNode, body: Expr|nil
  |||                 rescues: Array[RescueNode+]|nil, else: Expr|nil, ensure: Expr|nil)
- ||+ClassNode      #class..end
+ ||+-ClassNode     #class..end
  |||                (name: VarNode|ConstantNode, parent: Expr|nil, body: Expr|nil,
  |||                 rescues: Array[RescueNode+]|nil, else: Expr|nil, ensure: Expr|nil)
- ||+MetaClassNode  #class<<x..end
+ ||+-MetaClassNode #class<<x..end
  ||                 (val: Expr, body: Expr|nil,
  ||                  rescues: Array[RescueNode+]|nil, else: Expr|nil, ensure: Expr|nil)
- |+BracketsGetNode #a[b]
+ |+-BracketsGetNode #a[b]
  |                  (receiver: Expr, params: Array[Expr+,UnaryStarNode?]|nil)
  |
  ErrorNode         #mixed in to nodes with a syntax error
- +MisparsedNode    #mismatched braces or begin..end or the like
-
-
-
-== Known failing expressions
-* The following expressions are known to parse incorrectly currently:
-* def foo(a = 1)    end; def foo(a=b=c={})  end; def bar(a=b=c=1,d=2)  end
-* <<-EOS<<__LINE__
-    EOS
-* z = valueo_s rescue "?"
-* self.<=>:p8
-* return (@images = @old_imgs)
-* p ?e.<<?y
-* doc_status, err_args = Documeh_status{fcgi_state = 3; docespond do doc_response =fcgi_state =  1; end }
-* class A;def b;class <<self;@@p = false end;end;end
-* return @senders[1] =
-    2
-
-== Not exactly right, but semantically equivalent
-* These don't return exactly the same s-exp as MRI/ParseTree, but close enough:  
-* for i in (begin
-    [44,55,66,77,88] end) do p i**Math.sqrt(i) end
-* %W"is #{"Slim #{2?"W":"S"}"}#{xx}."
-* def d; return (block_given? ? begin; yield f; ensure; f.close; end : f); end
-* "#{publi}#{}>"
-* /__A#{__FILE__}tcase/n =~ i
+ +-MisparsedNode   #mismatched braces or begin..end or the like
+
+== Known problems with the parser:
+* Encoding of the input is not stored anywhere in resulting parse tree.
+* Ascii, binary, utf-8, and euc encodings are supported, but sjis is not.
+
+== Known problems with the unparser:
+* On unparse, here documents are converted into regular strings. For the most 
+  part, these are exactly equivalent to the original. However, whatever tokens
+  appeared between the here document header and body will now show up on a 
+  different line. If one of those tokens was __LINE__, it will have a 
+  different value in the unparsed code than it had originally.
+* some floating-point literals don't survive parse/unparse roundtrip intact, 
+  due to bugs in MRI 1.8's Float#to_s/String#to_f.
+* unparsing of trees whose input was in a character set other than ascii may
+  not work.
+
+== Known problems with ParseTree compatibility 
+* Major:
+  * converting non-ascii encoded parsetrees to ParseTree format doesn't work
+* Minor:
+  * :begin is not always emitted in the same places as ParseTree does:
+    * return begin;  f; end
+  * string nodes don't always come out the same way as in ParseTree... 
+    but what I emit is equivalent.
+    * %W"is #{"Slim #{2?"W":"S"}"}#{xx}."
+  * silly empty case nodes aren't always optimized to nop like in ParseTree.
 
 == Bugs in ruby
 * These expressions don't parse the same as in MRI because of bug(s) in MRI:
 * p = p m %(1)
 * p=556;p (e) /a
-* c do p (110).m end
+* c do p (110).m end #buggy in 1.8 only, fixed in 1.9
+* module Array ([Array]).first::E; end #buggy in 1.8 only, fixed in 1.9
 
 == Bugs in ParseTree
 * Unit tests see failures in these cases, but due to bugs in ParseTree:
+* doc_status, err_args = Documeh_status{fcgi_state = 3; docespond do doc_response =fcgi_state =  1; end }
 * case F;when G; else;case; when j; end;end
 * def foo(a=b=c={}) end
 * $11111111111111111111111111111111111111111111111111111111111111111111
 * proc{|&b| }
 * def sum(options = {:weights => weights = Hash.new(1)}); opt; end
-
-==Known failing files
-*  And here's a list of files which are known to parse incorrectly:
-
-
-activerdf-1.6.10/lib/active_rdf/queryengine/query2jars2.rb
-aquarium-0.4.2/spec/aquarium/aspects/aspect_spec.rb
-aspectr-0-3-5/lib/aspectr.rb
-authorails-1.0.0/lib/initializer.rb
-bibliomori-0.2.3/src/filter.rb
-cursor-0.9/cursor/circular.rb
-cursor-0.9/duck.rb
-data_objects-0.9.2/lib/data_objects/transaction.rb
-docdiff-0.3.2/docdiff/charstring.rb
-extract_curves-0.0.1/ruby_libs/pav/string/observable.rb
-extract-curves-0.1.1/ruby_libs/pav/string/observable.rb
-gecoder-with-gecode-0.8.2/lib/gecoder/interface/constraints/set/connection.rb
-gettext-1.91.0/test/test_gettext.rb
-lazytools-0.1.0/lib/dirstructure.rb
-lockfile-1.4.3/lib/lockfile-1.4.3.rb
-lockfile-1.4.3/lib/lockfile.rb
-logging-0.8.0/test/test_logger.rb
-logmerge-1.0.0/lib/logmerge/resolv.rb
-main-2.8.0/lib/main/base.rb
-muding-0.2.0/lib/initializer.rb
-net-mdns-0.4/lib/net/dns/resolv.rb
-not_naughty-0.5.1/spec/not_naughty_spec.rb
-radiant-0.6.7/vendor/rails/railties/lib/initializer.rb
-redparse/test/data/def_spec.rb
-reg/regcompiler.rb
-rgen-0.4.2/redist/xmlscan/tests/deftestcase.rb
-rq-3.4.0/lib/rq/lockfile.rb
-ruby-ajp-0.2.1/lib/net/ajp13.rb
-ruby-contract-0.1.1/lib/contract/overrides.rb
-ruby-debug-0.10.0/cli/ruby-debug/processor.rb
-rubygame-2.3.0/lib/rubygame/rect.rb
-rubygems-update-1.2.0/lib/rubygems/test_utilities.rb
-rubylexer/test/data/heremonsters_dos.rb
-ruby-nuggets-0.2.1.246/lib/nuggets/array/to_hash.rb
-sequel_core-2.2.0/lib/sequel_core/core_sql.rb
-skktools/filters/skkdictools.rb
-stick-1.3.3/lib/stick/matrix.rb
-stomp-1.0.5/lib/stomp.rb
-syntax-1.0.0/lib/syntax/lang/ruby.rb
-tournament-1.1.0/lib/tournament/bracket.rb
-typo-5.0.3.98.1/config/environment.rb
-typo-5.0.3.98.1/vendor/syntax/lib/syntax/lang/ruby.rb
-/usr/lib/ruby/1.8/docdiff/charstring.rb
-/usr/lib/ruby/1.8/puppet/provider/nameservice.rb
-/usr/lib/ruby/1.8/qwik/util-css.rb
-/usr/lib/ruby/1.8/resolv.rb
-/usr/lib/ruby/1.8/rexml/doctype.rb
-/usr/lib/ruby/1.8/tkextlib/bwidget/buttonbox.rb
-/usr/lib/ruby/1.9.0/resolv.rb
-/usr/lib/ruby/1.9.0/rexml/doctype.rb
-/usr/share/doc/libtcltk-ruby1.8/examples/tkextlib/tkHTML/ss.rb
-/usr/share/doc/libtcltk-ruby1.9/examples/tkextlib/tkHTML/ss.rb
-/usr/share/doc/libxml-ruby/tests/tc_xml_parser8.rb
-/usr/share/glark/options.rb
-/usr/share/hiki/hiki/docdiff/charstring.rb
-/usr/share/nadoka/rice/irc.rb
-/usr/share/rails/actionpack/test/controller/caching_test.rb
-/usr/share/rails/actionpack/test/controller/fragment_store_setting_test.rb
-/usr/share/rails/activesupport/lib/active_support/multibyte/handlers/utf8_handler.rb
-/usr/share/rails/railties/builtin/rails_info/rails/info.rb
-/usr/share/rails/railties/lib/initializer.rb
-/usr/share/skktools/filters/skkdictools.rb
-/usr/share/tdiary/TC_tdiary-setup.rb
-wwwsrv-0.15.3/wwwsrv/fastcgi.rb
-xmlscan-0.2.3/tests/deftestcase.rb
-ya2yaml-0.26/lib/ya2yaml.rb
-ActiveRecord-JDBC-0.5/lib/jdbc_adapter/jdbc_hsqldb.rb
-ruby-web-1.1.1/lib/webunit/parser.rb
-filemanager-0.2.0/filemanager/app/controllers/fm/filemanager_controller.rb
+* def foo(a = 1)    end; def foo(a=b=c={})  end; def bar(a=b=c=1,d=2)  end
+* yield [a_i, *p]