ripper-plus 1.1.0.pre2 → 1.2.0

data/README.md

@@ -1,8 +1,31 @@
 # ripper-plus
 
-Ripper is the Ruby parser library packaged with Ruby 1.9. While quite complete, it has some quirks that can make use frustrating. This gem intends to correct them. The most important is bareword resolution in Ruby ASTs.
+Ripper is the Ruby parser library packaged with Ruby 1.9. While quite complete, it still has bugs (2 of which I've patched alone while working on Laser, with the fixes targeted for 1.9.3: [mlhs-splats](http://redmine.ruby-lang.org/issues/4364) and [words/qwords](http://redmine.ruby-lang.org/issues/4365)), and it has higher-level quirks that can make use frustrating. ripper-plus is a gem intended to demonstrate what I believe the correct output from Ripper should be, *with the goal that these changes become the standard Ripper behavior*. I do *not* want to invent nor maintain a new AST standard. I do however believe these changes warrant separate implementation and discussion before a potential integration with Ripper, and creating a small library to demonstrate the proposed output seemed the simplest approach. Plus, [Laser](https://github.com/michaeledgar/laser/) needs all these improvements, so I had to do it anyway.
 
-## Bareword Resolution
+NB: Ripper is a SAX-style parser; one can construct an AST from the parser events however one wishes. Ripper also has a convenience method `Ripper.sexp`, which generates an Array-based AST directly from the SAX events. I personally use `Ripper.sexp` in my work, so the examples will be of `Ripper.sexp` output. All of the discussion below reflects deficiencies in the underlying SAX parser: they are unavoidable whether one uses `Ripper.sexp` or not.
+
+## tl;dr Version
+
+If you don't want to read everything, here's what you should take away:
+
+1. Ripper does not distinguish between reading a local variable and implicit-self, no-paren, no-argument method calls. Due to Ruby's semantics, this must be done by the parser. Example: `y = foo`. Is foo a method call? Ripper does not tell you.
+2. Ruby's Bison grammar rejects some invalid syntax not by restricting the grammar, but by parsing with a more permissive grammar and then validating the results. Ripper does report some of these errors in its parse tree, but not all of them, and sometimes inconveniently deep in the parse tree. Example: `proc { |a, a| }` is a syntax error, but Ripper parses it without flinching.
+
+## Still Here?
+
+If you use Ripper for anything important, I'll assume you've gotten this far. Quick info about `ripper-plus`: it is a gem I hope will correct the following issues until they are addressed in Ripper proper. To use `ripper-plus`, install it with `gem install ripper-plus`. The gem has two entry points:
+
+    require 'ripper-plus'
+    # Returns an AST from Ripper, modified as described below
+    RipperPlus.sexp(any_input_object_Ripper_accepts)
+    # Transforms a Ripper AST into a RipperPlus AST, as described below
+    RipperPlus.for_ripper_ast(some_ripper_ast)
+    # Transforms a Ripper AST into a RipperPlus AST, in-place
+    RipperPlus.for_ripper_ast(some_ripper_ast, :in_place => true)
+
+That's it - now read about why exactly ripper-plus exists.
+
+# Bareword Resolution
 
 In Ruby, a local variable is created when it is assigned to. If Ruby finds a bareword `foo` before an assignment `foo = 5`, then Ruby will call `self.foo()`.
 
@@ -10,6 +33,7 @@ Kind of.
 
 What happens when you run this code?
 
+    
     def label
       'hello'
     end
@@ -19,92 +43,162 @@ What happens when you run this code?
     end
     print_label
 
+
 If one takes the intuitive approach, one would assume it prints "Label: hello", right? Afraid not: it errors out. The intuitive approach says that `'Label: ' + label` is run before the assignment to the local variable `label`, so the `label` method will be called and appended to `"Label: "`. In reality, it won't, because local variables aren't created on assignment. Local variables are created *immediately when the Ruby parser parses a left-hand-side local variable*. Before the right-hand-side is even parsed, the local variable has been created. This also results in an infamous anomaly:
 
+[code]
     def say_what?
       x = 'hello' unless defined?(x)
       puts x
     end
-    say_what?
+[/code]
 
 `say_what?` prints a blank line! This is because as soon as the `x` on the LHS is parsed, `x` is a local variable with value `nil`. By the time `defined?(x)` is executed, `x` has long since been a local variable!
 
-## Ripper's Bareword Behavior
+This is further complicated by the many ways a local variable can be introduced:
+
+1. Single- or multiple-assignment (including sub-assignment) anywhere an expression is allowed
+2. A `for` loop's iterator variable(s)
+3. Formal arguments to a method
+4. Block arguments and block-local variables
+5. Stabby Lambda variables
+6. Rescue exception naming (`rescue StandardError => err`)
+7. Named Regexp captures in literal matches (`/name: (?<person>\w+)/ =~ 'name: Mike'` creates a local variable named "person" with value "Mike")
+
+I think that's a complete list, but I may be mistaken. In fact, I had forgotten named captures until I double-checked this list while writing this blog post. I tried to find all paths in `parse.y` to a call to `assignable()`, the C function in Ruby's parser that creates local variable in the current scope, and I think I caught them all, but I may have slipped up.
 
-Ripper doesn't carry scope information as it parses, and as such, parses any lone identifier as an AST node of type `:var_ref`. It is up to consumers of the AST to figure out the meaning of the `:var_ref` node. It can be reasonably argued that the semantics of the `:var_ref` node should not be part of the AST, as my thesis adviser pointed out when I complained about this, as the two are syntactically identical. Unfortunately, the meaning of the `var_ref` node comes from the parser itself; any attempt to determine semantics based solely on the AST is simply re-creating the work the parser ordinarily does. Indeed, when Ruby parses the code for execution, it *does* create different internal AST node types upon seeing a method-call bareword and a local variable bareword!
+## Ripper's Mistake
 
-I'd like to see this behavior rolled into Ripper proper. Until then, `ripper-plus` does it.
+Ripper doesn't carry scope information as it parses, and as such, parses any lone identifier as an AST node of type `:var_ref`. It is up to consumers of the AST to figure out the meaning of the `:var_ref` node. It can be reasonably argued that the semantics of the `:var_ref` node should not be part of the AST, as my thesis adviser pointed out when I complained about this, as the two are syntactically identical. Unfortunately, the meaning of the `:var_ref` node comes from the parser itself; any attempt to determine semantics based solely on the Ripper AST must simulate the work the parser ordinarily does. Indeed, when Ruby parses the code for execution, it *does* create different internal node types upon seeing a method-call bareword and a local variable bareword!
 
-## ripper-plus
+I'd like to see this behavior rolled into Ripper proper. Until then, `ripper-plus` is a reasonable replacement.
 
-By using our knowledge of Ruby's parser, we can recreate the scope-tracking behavior by walking the AST generated by Ripper. We simply must do normal scope tracking, observe the creation of local variables, and ensure that we walk each node of the tree in the order in which those nodes would have been parsed. Most subtrees generated by Ripper are already in this order, with the exception of the modifier nodes (`foo.bar if baz`, `x = bar() while x != nil`). Most importantly, since everything is an expression in Ruby and local variable assignments can occur just about anywhere, exact scoping semantics must be recreated. Corner cases such as this:
+## ripper-plus's Approach
 
+By using our knowledge of Ruby's parser, we can simulate the scope-tracking behavior by walking the AST generated by Ripper. We simply do normal scope tracking, observe the creation of local variables, and ensure that we walk each node of the tree in the order in which those nodes would have been parsed. Most subtrees generated by Ripper are already in this order, with the exception of the modifier nodes (`foo.bar if baz`, `x = bar() while x != nil`). Most importantly, since everything<sup>[*](http://carboni.ca/blog/p/Statements-in-Ruby)</sup> is an expression in Ruby and local variable assignments can occur just about anywhere, exact scoping semantics must be recreated. Every possible introduction of local variables (exception naming, named captures, ...) must be considered as the Ruby parser would. Corner cases such as this:
+
+[code]
     def foo(x, y = y)
     end
+[/code]
 
 Need to be properly resolved. Did you know that, unlike the `label = label` example above, the above code is equivalent to:
 
+[code]
     def foo(x, y = y())
     end
+[/code]
 
-Anyway, ripper-plus turns all method-call `:var_ref` nodes into `:zcall` nodes; the node structure is otherwise unchanged. It runs in O(N) time and O(h) space, where h is the height of the tree.
-
-## Syntax Errors
-
-Not all syntax errors in Ruby fail to parse as valid Ruby. Using `next`/`redo`/`break`/`retry` when they have no meaning will cause a "compile error (SyntaxError)" exception to be raised during parsing. Attempting to use those expressions in a value context will also fail to compile. Ripper, however, doesn't do any of this validation:
+Why doesn't y end up `nil` in the default case, as it would if you typed `y = y` into a method definition?
 
-    pp Ripper.sexp('x = 5; next x')
-    #=>
-    [:program,
-     [[:assign, [:var_field, [:@ident, "x", [1, 0]]], [:@int, "5", [1, 4]]],
-      [:next, [:args_add_block, [[:var_ref, [:@ident, "x", [1, 12]]]], false]]]]
+Anyway, ripper-plus turns all method-call `:var_ref` nodes into `:zcall` nodes; the node structure is otherwise unchanged. I believe Ripper should already make this distinction, and it can relatively simply: it simply has to re-implement the existing scope-tracking behavior to the relevant Ripper action routines. Not trivial, but `ripper-plus` does it in a couple hundred lines of Ruby.
 
-Assigning to set read-only variables is also caught by the Ruby parser, and not at runtime. An easy way to verify this is to put a simple print statement before the offending syntax error:
+# Error Handling
 
-    $ ruby -e 'p 5; $1 = "5"'
-    -e:1: Can't set variable $1
+As mentioned above, not all invalid syntaxes are excluded by Ruby's CFG production rules: additional validation happens occasionally, and when validation fails, `yyerror` is called, and parsing attempts recovery. The Ruby program will not run if such an error exists, but since parsing continues, additional such errors might be found. These are some of (but likely not all) of the errors I'm describing:
 
-Ripper catches this by wrapping the offending assignment in an `:assign_error` node. But not all such invalid assignments are caught:
+1. `class name ... end`/`module name ... end` with a non-constant name (`=~ /^[^A-Z]/`)
+2. `alias gvar $[0-9]`: you can't alias the numeric capture variables
+3. duplicate argument names to a method, block
+4. constant/instance variable/class variable/global variable as an argument name
+5. assigning a constant in a method definition
+6. class/module definition syntax in a method body
+7. defining singleton methods on literals using expression syntax (`def (expr).foo; end`)
 
-    pp Ripper.sexp('$1 = 5')
-    #=>
-    [:program,
-     [[:assign, [:assign_error, [:@backref, "$1", [1, 0]]], [:@int, "5", [1, 5]]]]]
+Whatever your tool is that is dealing with Ruby ASTs, it likely is concerned with whether the program is valid or not, and any program containing the above errors is invalid: Ruby won't even attempt to run it. Ripper, one would suppose, would also inform consumers of the parse tree that a given program is invalid in these ways.
 
-    pp Ripper.sexp('nil = self')
-    #=>
-    [:program,
-     [[:assign,
-       [:var_field, [:@kw, "nil", [1, 0]]],
-       [:var_ref, [:@kw, "self", [1, 6]]]]]]
+## Ripper's Mistake
 
-Ripper has at least one other error node type, `:class_name_error`:
+Some of these errors are noticed by Ripper, and the offending portion of the syntax tree will be wrapped in a corresponding error node:
 
+[code]
+    # Invalid alias
+    pp Ripper.sexp('alias $foo $2')
+    #=>[:program,
+        [[:alias_error,
+          [:var_alias, [:@gvar, "$foo", [1, 6]], [:@backref, "$2", [1, 11]]]]]]
+    # Invalid class name
     pp Ripper.sexp('class abc; end')
-    #=>
-    [:program,
-     [[:class,
-       [:const_ref, [:class_name_error, [:@ident, "foo", [1, 6]]]],
-       nil,
-       [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
-
-Ruby has a few dozen of these syntax errors that are caught at compile time, and most of them are things you would be hard-pressed to justify: re-using argument names (`def foo(x, x); end`), creating classes or modules in method bodies, and so on.  Yet Ripper does not provide an easy way to check if these errors are present in a given AST.
-
-I'm not convinced that wrapping offending nodes in different node types is the best way to handle this issue. Firstly, the error nodes are not always high enough in the tree (`:alias_error`, which arises when aliasing `$1, $2, ...`, is a notable exception). The position of the `:class_name_error` node forces every consumption of a `:class` node to check if the name subtree contains a `:class_name_error` node. The whole class is invalid though! If anything, the entire `:class` node should be inside an `:error` node. The same holds true for `:assign` nodes. This isn't really the Ripper designer's faults: Ripper is essentially a separate set of action routines in the same bison grammar used by YARV proper, and as an SAX-style parser, it'd be much harder to push errors upward.
-
-It does seem reasonable, however, that while we are transforming `:var_ref` into `:zcall` nodes, we can find these errors (including ones ignored by Ripper), wrap them in `:error` nodes, and provide a list of exceptions corresponding to the semantic errors. `ripper-plus` does just that. For example, the above error will transform into this:
-
-    [:program,
-     [[:error,
-       [:class,
-        [:const_ref, [:class_name_error, [:@ident, "foo", [1, 6]]]],
-        nil,
-        [:bodystmt, [[:void_stmt]], nil, nil, nil]],
-       'class/module name must be CONSTANT']]]
-
-## You Should be Using ripper-plus
-
-The truth is, everybody who is using Ripper right now *should* be doing *all* of this. Anything short, and you have bugs. [Laser](https://github.com/michaeledgar/laser/) has bugs as a result. It's a pain in the ass to get it all right. `ripper-plus` probably has bugs - I'm not gonna lie, I found one while writing this. But I'm pretty damn sure it's solid. Hopefully, in Ruby 1.9.x, this will be the default. For now, you *should* use ripper-plus.
+    #=> [:program,
+         [[:class,
+           [:const_ref, [:class_name_error, [:@ident, "abc", [1, 6]]]],
+           nil,
+           [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
+[/code]
+
+The first error, a disallowed alias, results in an `:alias_error` node wrapping the entire `:var_alias` node. This is good: anybody walking this AST would see that the entire contents of the node are semantically invalid *before* any information about the attempted alias. The second error, an invalid class name, results in a `:class_name_error` node deep in the `:class` node structure. This is bad: it puts the onus on every consumer of the AST to check all class/module definitions for `:class_name_error` nodes before assuming *anything* about the meaning of the `:class` node. This unfortunate placement also occurs when a parameter is named with a non-identifier:
+
+[code]
+    pp Ripper.sexp('def foo(a, @b); end')
+    #=> [:program,
+         [[:def,
+           [:@ident, "foo", [1, 4]],
+           [:paren,
+            [:params,
+             [[:@ident, "a", [1, 8]], [:param_error, [:@ivar, "@b", [1, 11]]]],
+             nil, nil, nil, nil]],
+           [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
+[/code]
+
+Worse, some errors are not caught at all:
+
+[code]
+    pp Ripper.sexp('def foo(a, a, a); end')
+    #=> [:program,
+         [[:def,
+           [:@ident, "foo", [1, 4]],
+           [:paren,
+            [:params,
+             [[:@ident, "a", [1, 8]],
+              [:@ident, "a", [1, 11]],
+              [:@ident, "a", [1, 14]]],
+             nil, nil, nil, nil]],
+           [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
+[/code]
+
+This is very bad. Now every consumer of the Ripper parse tree must check *all method definitions* for repeated arguments, and bail accordingly. Don't forget about subassignments in block argument declarations; you'll need to check those yourself, too:
+
+[code]
+    pp Ripper.sexp('something.each { |a, (b, *c, (d, e), a), f| }')
+    #=> [:program,
+         [[:method_add_block,
+           [:call,
+            [:var_ref, [:@ident, "something", [1, 0]]], :".",
+            [:@ident, "each", [1, 10]]],
+           [:brace_block, [:block_var,
+             [:params,
+              [[:@ident, "a", [1, 18]],
+               [:mlhs_paren,
+                [:mlhs_add_star,
+                 [[:mlhs_paren, [:@ident, "b", [1, 22]]]],
+                 [:@ident, "c", [1, 26]]]],
+               [:@ident, "f", [1, 41]]],
+              nil, nil, nil, nil], nil],
+            [[:void_stmt]]]]]]
+[/code]
+
+Can your Ripper-consuming code handle that case?
+
+## ripper-plus's Approach
+
+This is where it gets murky, and I don't have a definitive answer just yet. Right now, `ripper-plus` wraps all offending nodes it finds (which is *not* all such errors - yet) in a generic `:error` node. At least with this solution, anyone walking a `RipperPlus` AST will know when a given node is semantically invalid. However, the code walking the AST must find an `:error` node before it knows the entire AST is meaningless. For a tool like [YARD](http://yardoc.org/) or [Redcar](http://redcareditor.com/), which likely would prefer to keep extracting information from the AST, this seems preferable. The `:error` node could potentially be specified to potentially include descriptions and locations of the syntax errors discovered.
+
+Yet it also seems convenient to receive, upon attempting to parse an invalid program, a simple list of errors and nothing else. For a tool like Laser, this is far preferable, and wastes a lot less time. I think Laser may be in the minority here. So Laser will do its own thing regardless.
+
+# Minor Gripes
+
+1. Flip-flops don't have their own node type. Upon encountering a range, you must check if you are in a conditional context manually. This is annoying.
+2. \_\_END\_\_ just stops parsing: there's no getting at the actual data past it from Ripper's output.
+3. Ripper.sexp never fails. Given a program that fails to parse, Ripper will simply return the best recovery parse Bison can come up with. (`Ripper.sexp('x*^y#$a') == [:program, [:var_ref, [:@ident, "y", [1, 3]]]]`)
+4. This space reserved for future gripes.
+
+## Conclusion
+
+Overall, Ripper is very complete for a library covered in "EXPERIMENTAL" warning labels, and gives concise, traditional ASTs. What I've put forward is all I ask after nearly a year of being neck-deep in Ripper output. I think the main two points I've covered need to be addressed in the 1.9 branch of Ruby, and over the coming months, hope to work to get that done.
+
+In the meantime, I'll be using `ripper-plus` in Laser to statically analyze all kinds of stuff about Ruby, but I look forward to the day where I can add `RipperPlus = Ripper if RUBY_VERSION >= MERGED_VERSION` to ripper-plus's main file. `ripper-plus` is O(N), though it's not blazingly fast: it takes about 20-30 ms on my machine to transform the Ripper AST for [the biggest, ugliest Laser code file: the CFG compiler](https://github.com/michaeledgar/laser/blob/master/lib/laser/analysis/control_flow/cfg_builder.rb) to a RipperPlus AST. It takes 50ms for Ripper to parse it in the first place. (That benchmarked transformation is done in-place: `ripper-plus`'s default behavior is to duplicate the Ripper AST.)
+
+If you have ideas or thoughts, please discuss them on ruby-talk and/or ruby-core! I don't have comments implemented for a reason: I feel it fragments the center of discussion.
 
 ## Contributing to ripper-plus
  
@@ -119,5 +213,4 @@ The truth is, everybody who is using Ripper right now *should* be doing *all* of
 ## Copyright
 
 Copyright (c) 2011 Michael Edgar. See LICENSE.txt for
-further details.
-
+further details.

data/lib/ripper-plus.rb

@@ -2,6 +2,7 @@ if RUBY_VERSION < "1.9"
   raise 'ripper-plus requires Ruby 1.9+'
 end
 require 'ripper'
+$:.unshift(File.expand_path(File.dirname(__FILE__)))
 require 'ripper-plus/version'
 require 'ripper-plus/ripper-plus'
 require 'ripper-plus/scope_stack'

data/lib/ripper-plus/ripper-plus.rb

@@ -1,13 +1,15 @@
 # Top-level module for Ripper Plus. Provides global methods for
 # getting a RipperPlus AST for a given input program.
 module RipperPlus
+  DEFAULT_OPTS = {:in_place => false}
   # Parses the given Ruby code into a RipperPlus AST.
-  def self.sexp(text)
-    for_ripper_ast(Ripper.sexp(text))
+  def self.sexp(text, opts={})
+    for_ripper_ast(Ripper.sexp(text), opts.merge(:in_place => true))
   end
   
   # Transforms the provided Ripper AST into a RipperPlus AST.
-  def self.for_ripper_ast(tree)
-    Transformer.transform(tree)
+  def self.for_ripper_ast(tree, opts={})
+    opts = DEFAULT_OPTS.merge(opts)
+    Transformer.transform(tree, opts)
   end
 end

data/lib/ripper-plus/scope_stack.rb

@@ -25,7 +25,10 @@ module RipperPlus
       "< #{middle} >"
     end
 
-    def add_variable(var)
+    def add_variable(var, allow_duplicates=true)
+      if !allow_duplicates && @stack.last.include?(var)
+        raise DuplicateArgumentError.new("duplicated argument name (#{var})")
+      end
       @stack.last << var
     end
   

data/lib/ripper-plus/transformer.rb

@@ -1,5 +1,10 @@
 require 'set'
 module RipperPlus
+  class SyntaxError < StandardError; end
+  class LHSError < SyntaxError; end
+  class DynamicConstantError < SyntaxError; end
+  class InvalidArgumentError < SyntaxError; end
+  class DuplicateArgumentError < SyntaxError; end
   # Transforms a 1.9.2 Ripper AST into a RipperPlus AST. The only
   # change as a result of this transformation is that the nodes for
   # local variable references and zcalls (bareword calls to self)
@@ -43,14 +48,9 @@ module RipperPlus
   module Transformer
     extend self
 
-    class SyntaxError < StandardError; end
-    class LHSError < SyntaxError; end
-    class DynamicConstantError < SyntaxError; end
-    class InvalidArgumentError < SyntaxError; end
-
     # Transforms the given AST into a RipperPlus AST.
-    def transform(root)
-      new_copy = clone_sexp(root)
+    def transform(root, opts={})
+      new_copy = opts[:in_place] ? root : clone_sexp(root)
       scope_stack = ScopeStack.new
       transform_tree(new_copy, scope_stack)
       new_copy
@@ -59,135 +59,173 @@ module RipperPlus
     # Transforms the given tree into a RipperPlus AST, using a scope stack.
     # This will be recursively called through each level of the tree.
     def transform_tree(tree, scope_stack)
-      case tree[0]
-      when :assign, :massign
-        lhs, rhs = tree[1..2]
-        begin
-          add_variables_from_node(lhs, scope_stack)
-        rescue SyntaxError => err
-          wrap_node_with_error(tree)
-        else
-          transform_tree(rhs, scope_stack)
-        end
-      when :for
-        vars, iterated, body = tree[1..3]
-        add_variables_from_node(vars, scope_stack)
-        transform_tree(iterated, scope_stack)
-        transform_tree(body, scope_stack)
-      when :var_ref
-        # When we reach a :var_ref, we should know everything we need to know
-        # in order to tell if it should be transformed into a :zcall.
-        if tree[1][0] == :@ident && !scope_stack.has_variable?(tree[1][1])
-          tree[0] = :zcall
-        end
-      when :class
-        name, superclass, body = tree[1..3]
-        if name[1][0] == :class_name_error || scope_stack.in_method?
-          wrap_node_with_error(tree)
-        else
-          transform_tree(superclass, scope_stack) if superclass  # superclass node
-          scope_stack.with_closed_scope do
-            transform_tree(body, scope_stack)
-          end
-        end
-      when :module
-        name, body = tree[1..2]
-        if name[1][0] == :class_name_error || scope_stack.in_method?
-          wrap_node_with_error(tree)
-        else
-          scope_stack.with_closed_scope do
-            transform_tree(body, scope_stack)  # body
+      if Symbol === tree[0]
+        case tree[0]
+        when :assign, :massign
+          lhs, rhs = tree[1..2]
+          begin
+            add_variables_from_node(lhs, scope_stack)
+          rescue SyntaxError => err
+            wrap_node_with_error(tree)
+          else
+            transform_tree(rhs, scope_stack)
           end
-        end
-      when :sclass
-        singleton, body = tree[1..2]
-        transform_tree(singleton, scope_stack)
-        scope_stack.with_closed_scope do
+        when :for
+          vars, iterated, body = tree[1..3]
+          add_variables_from_node(vars, scope_stack)
+          transform_tree(iterated, scope_stack)
           transform_tree(body, scope_stack)
-        end
-      when :def
-        scope_stack.with_closed_scope(true) do
-          param_node = tree[2]
-          body = tree[3]
-          begin
-            transform_params(param_node, scope_stack)
-          rescue SyntaxError
+        when :var_ref
+          # When we reach a :var_ref, we should know everything we need to know
+          # in order to tell if it should be transformed into a :zcall.
+          if tree[1][0] == :@ident && !scope_stack.has_variable?(tree[1][1])
+            tree[0] = :zcall
+          end
+        when :class
+          name, superclass, body = tree[1..3]
+          if name[1][0] == :class_name_error || scope_stack.in_method?
             wrap_node_with_error(tree)
           else
-            transform_tree(body, scope_stack)
+            transform_tree(superclass, scope_stack) if superclass  # superclass node
+            scope_stack.with_closed_scope do
+              transform_tree(body, scope_stack)
+            end
           end
-        end
-      when :defs
-        transform_tree(tree[1], scope_stack)  # singleton could be a method call!
-        scope_stack.with_closed_scope(true) do
-          param_node = tree[4]
-          body = tree[5]
-          begin
-            transform_params(param_node, scope_stack)
-          rescue SyntaxError
+        when :module
+          name, body = tree[1..2]
+          if name[1][0] == :class_name_error || scope_stack.in_method?
             wrap_node_with_error(tree)
           else
+            scope_stack.with_closed_scope do
+              transform_tree(body, scope_stack)  # body
+            end
+          end
+        when :sclass
+          singleton, body = tree[1..2]
+          transform_tree(singleton, scope_stack)
+          scope_stack.with_closed_scope do
             transform_tree(body, scope_stack)
           end
-        end
-      when :rescue
-        list, name, body = tree[1..3]
-        transform_tree(list, scope_stack)
-        # Don't forget the rescue argument!
-        if name
-          add_variables_from_node(name, scope_stack)
-        end
-        transform_tree(body, scope_stack)
-      when :method_add_block
-        call, block = tree[1..2]
-        # first transform the call
-        transform_tree(call, scope_stack)
-        # then transform the block
-        block_args, block_body = block[1..2]
-        scope_stack.with_open_scope do
-          if block_args
-            transform_params(block_args[1], scope_stack)
-            if block_args[2]
-              block_args[2].each { |var| add_variables_from_node(var, scope_stack) }
+        when :def
+          scope_stack.with_closed_scope(true) do
+            param_node = tree[2]
+            body = tree[3]
+            transform_params_then_body(tree, param_node, body, scope_stack)
+          end
+        when :defs
+          transform_tree(tree[1], scope_stack)  # singleton could be a method call!
+          scope_stack.with_closed_scope(true) do
+            param_node = tree[4]
+            body = tree[5]
+            transform_params_then_body(tree, param_node, body, scope_stack)
+          end
+        when :lambda
+          param_node, body = tree[1..2]
+          scope_stack.with_open_scope do
+            transform_params_then_body(tree, param_node, body, scope_stack)
+          end
+        when :rescue
+          list, name, body = tree[1..3]
+          transform_tree(list, scope_stack)
+          # Don't forget the rescue argument!
+          if name
+            add_variables_from_node(name, scope_stack)
+          end
+          transform_tree(body, scope_stack)
+        when :method_add_block
+          call, block = tree[1..2]
+          # first transform the call
+          transform_tree(call, scope_stack)
+          # then transform the block
+          param_node, body = block[1..2]
+          scope_stack.with_open_scope do
+            begin
+              if param_node
+                transform_params(param_node[1], scope_stack)
+                if param_node[2]
+                  add_variable_list(param_node[2], scope_stack, false)
+                end
+              end
+            rescue SyntaxError
+              wrap_node_with_error(tree)
+            else
+              transform_tree(body, scope_stack)
+            end
+          end
+        when :binary
+          # must check for named groups in a literal match. wowzerz.
+          lhs, op, rhs = tree[1..3]
+          if op == :=~
+            if lhs[0] == :regexp_literal
+              add_locals_from_regexp(lhs, scope_stack)
+              transform_tree(rhs, scope_stack)
+            elsif lhs[0] == :paren && !lhs[1].empty? && lhs[1] != [[:void_stmt]] && lhs[1].last[0] == :regexp_literal
+              lhs[1][0..-2].each { |node| transform_tree(node, scope_stack) }
+              add_locals_from_regexp(lhs[1].last, scope_stack)
+              transform_tree(rhs, scope_stack)
+            else
+              transform_in_order(tree, scope_stack)
             end
+          else
+            transform_in_order(tree, scope_stack)
           end
-          transform_tree(block_body, scope_stack)
+        when :if_mod, :unless_mod, :while_mod, :until_mod, :rescue_mod
+          # The AST is the reverse of the parse order for these nodes.
+          transform_tree(tree[2], scope_stack)
+          transform_tree(tree[1], scope_stack)
+        when :alias_error, :assign_error  # error already top-level! wrap it again.
+          wrap_node_with_error(tree)
+        else
+          transform_in_order(tree, scope_stack)
         end
-      when :if_mod, :unless_mod, :while_mod, :until_mod, :rescue_mod
-        # The AST is the reverse of the parse order for these nodes.
-        transform_tree(tree[2], scope_stack)
-        transform_tree(tree[1], scope_stack)
-      when :alias_error, :assign_error  # error already top-level! wrap it again.
-        wrap_node_with_error(tree)
       else
         transform_in_order(tree, scope_stack)
       end
     end
 
+    def transform_params_then_body(tree, params, body, scope_stack)
+      transform_params(params, scope_stack)
+    rescue SyntaxError
+      wrap_node_with_error(tree)
+    else
+      transform_tree(body, scope_stack)
+    end
+
+    def add_locals_from_regexp(regexp, scope_stack)
+      regexp_parts = regexp[1]
+      if regexp_parts.one? && regexp_parts[0][0] == :@tstring_content
+        regexp_text = regexp_parts[0][1]
+        captures = Regexp.new(regexp_text).names
+        captures.each { |var_name| scope_stack.add_variable(var_name) }
+      end
+    end
+
+    def add_variable_list(list, scope_stack, allow_duplicates=true)
+      list.each { |var| add_variables_from_node(var, scope_stack, allow_duplicates) }
+    end
+
     # Adds variables to the given scope stack from the given node. Allows
     # nodes from parameter lists, left-hand-sides, block argument lists, and
     # so on.
-    def add_variables_from_node(lhs, scope_stack)
+    def add_variables_from_node(lhs, scope_stack, allow_duplicates=true)
       case lhs[0]
       when :@ident
-        scope_stack.add_variable(lhs[1])
+        scope_stack.add_variable(lhs[1], allow_duplicates)
       when :const_path_field, :@const, :top_const_field
         if scope_stack.in_method?
           raise DynamicConstantError.new
         end
       when Array
-        lhs.each { |var| add_variables_from_node(var, scope_stack) }
+        add_variable_list(lhs, scope_stack, allow_duplicates)
       when :mlhs_paren, :var_field, :rest_param, :blockarg
-        add_variables_from_node(lhs[1], scope_stack)
+        add_variables_from_node(lhs[1], scope_stack, allow_duplicates)
       when :mlhs_add_star
         pre_star, star, post_star = lhs[1..3]
-        pre_star.each { |var| add_variables_from_node(var, scope_stack) }
+        add_variable_list(pre_star, scope_stack, allow_duplicates)
         if star
-          add_variables_from_node(star, scope_stack)
-        end
-        if post_star
-          post_star.each { |var| add_variables_from_node(var, scope_stack) }
+          add_variables_from_node(star, scope_stack, allow_duplicates)
         end
+        add_variable_list(post_star, scope_stack, allow_duplicates) if post_star
       when :param_error
         raise InvalidArgumentError.new
       when :assign_error
@@ -214,25 +252,19 @@ module RipperPlus
       param_node = param_node[1] if param_node[0] == :paren
       if param_node
         positional_1, optional, rest, positional_2, block = param_node[1..5]
-        if positional_1
-          positional_1.each { |var| add_variables_from_node(var, scope_stack) }
-        end
+        add_variable_list(positional_1, scope_stack, false) if positional_1
         if optional
           optional.each do |var, value|
             # MUST walk value first. (def foo(y=y); end) == (def foo(y=y()); end)
             transform_tree(value, scope_stack)
-            add_variables_from_node(var, scope_stack)
+            add_variables_from_node(var, scope_stack, false)
           end
         end
         if rest && rest[1]
-          add_variables_from_node(rest, scope_stack)
-        end
-        if positional_2
-          positional_2.each { |var| add_variables_from_node(var, scope_stack) }
-        end
-        if block
-          add_variables_from_node(block, scope_stack)
+          add_variables_from_node(rest, scope_stack, false)
         end
+        add_variable_list(positional_2, scope_stack, false) if positional_2
+        add_variables_from_node(block, scope_stack, false) if block
       end
     end
 

data/lib/ripper-plus/version.rb

@@ -1,10 +1,14 @@
 module RipperPlus
   module Version
     MAJOR = 1
-    MINOR = 1
+    MINOR = 2
     PATCH = 0
-    BUILD = 'pre2'
+    BUILD = ''
 
-    STRING = [MAJOR, MINOR, PATCH, BUILD].compact.join('.')
+    if BUILD.empty?
+      STRING = [MAJOR, MINOR, PATCH].compact.join('.')
+    else
+      STRING = [MAJOR, MINOR, PATCH, BUILD].compact.join('.')
+    end
   end
 end

data/spec/spec_helper.rb

@@ -28,5 +28,6 @@ def dfs_for_node_type(tree, type)
 end
 
 RSpec.configure do |config|
-  
+  config.filter_run focus: true
+  config.run_all_when_everything_filtered = true
 end

data/spec/transformer_spec.rb

@@ -248,6 +248,66 @@ describe RipperPlus::Transformer do
       input_tree.should transform_to(output_tree)
     end
   
+    it 'should transform respecting stabby-lambda arguments' do
+      input_tree =
+        [:program,
+         [[:assign,
+           [:var_field, [:@ident, "y", [1, 0]]],
+           [:lambda,
+            [:paren,
+             [:params,
+              [[:@ident, "a", [1, 7]]],
+              [[[:@ident, "b", [1, 10]], [:var_ref, [:@ident, "a", [1, 12]]]],
+               [[:@ident, "c", [1, 15]], [:var_ref, [:@ident, "d", [1, 17]]]]],
+              [:rest_param, [:@ident, "rest", [1, 21]]],
+              nil,
+              nil]],
+            [[:binary,
+              [:array,
+               [[:binary,
+                 [:binary,
+                  [:binary,
+                   [:var_ref, [:@ident, "a", [1, 30]]],
+                   :*,
+                   [:var_ref, [:@ident, "b", [1, 34]]]],
+                  :*,
+                  [:var_ref, [:@ident, "c", [1, 38]]]],
+                 :*,
+                 [:var_ref, [:@ident, "d", [1, 42]]]]]],
+              :+,
+              [:var_ref, [:@ident, "rest", [1, 47]]]]]]],
+          [:var_ref, [:@ident, "a", [1, 55]]]]]
+      output_tree =
+        [:program,
+         [[:assign,
+           [:var_field, [:@ident, "y", [1, 0]]],
+           [:lambda,
+            [:paren,
+             [:params,
+              [[:@ident, "a", [1, 7]]],
+              [[[:@ident, "b", [1, 10]], [:var_ref, [:@ident, "a", [1, 12]]]],
+               [[:@ident, "c", [1, 15]], [:zcall, [:@ident, "d", [1, 17]]]]],
+              [:rest_param, [:@ident, "rest", [1, 21]]],
+              nil,
+              nil]],
+            [[:binary,
+              [:array,
+               [[:binary,
+                 [:binary,
+                  [:binary,
+                   [:var_ref, [:@ident, "a", [1, 30]]],
+                   :*,
+                   [:var_ref, [:@ident, "b", [1, 34]]]],
+                  :*,
+                  [:var_ref, [:@ident, "c", [1, 38]]]],
+                 :*,
+                 [:zcall, [:@ident, "d", [1, 42]]]]]],
+              :+,
+              [:var_ref, [:@ident, "rest", [1, 47]]]]]]],
+          [:zcall, [:@ident, "a", [1, 55]]]]]
+      input_tree.should transform_to output_tree
+    end
+  
     it 'should transform respecting subassignments in block arguments' do
       input_tree =
         [:program,
@@ -563,6 +623,98 @@ describe RipperPlus::Transformer do
                [:args_add_block, [[:zcall, [:@ident, "err", [1, 78]]]], false]]]]]]]]
       input_tree.should transform_to output_tree
     end
+    
+    it 'finds local variables created by named captures' do
+      input_tree =
+        [:program,
+         [[:def,
+           [:@ident, "foo", [1, 4]],
+           [:paren, [:params, [[:@ident, "a", [1, 8]]], nil, nil, nil, nil]],
+           [:bodystmt,
+            [[:void_stmt],
+             [:binary,
+              [:regexp_literal,
+               [[:@tstring_content, "name: (?<name>w+)", [1, 13]]],
+               [:@regexp_end, "/", [1, 30]]],
+              :=~,
+              [:var_ref, [:@ident, "a", [1, 35]]]],
+             [:command,
+              [:@ident, "p", [1, 38]],
+              [:args_add_block, [[:var_ref, [:@ident, "name", [1, 40]]]], false]],
+             [:var_ref, [:@ident, "named", [1, 46]]]],
+            nil,
+            nil,
+            nil]]]]
+      output_tree =
+        [:program,
+         [[:def,
+           [:@ident, "foo", [1, 4]],
+           [:paren, [:params, [[:@ident, "a", [1, 8]]], nil, nil, nil, nil]],
+           [:bodystmt,
+            [[:void_stmt],
+             [:binary,
+              [:regexp_literal,
+               [[:@tstring_content, "name: (?<name>w+)", [1, 13]]],
+               [:@regexp_end, "/", [1, 30]]],
+              :=~,
+              [:var_ref, [:@ident, "a", [1, 35]]]],
+             [:command,
+              [:@ident, "p", [1, 38]],
+              [:args_add_block, [[:var_ref, [:@ident, "name", [1, 40]]]], false]],
+             [:zcall, [:@ident, "named", [1, 46]]]],
+            nil,
+            nil,
+            nil]]]]
+      input_tree.should transform_to output_tree
+    end
+    
+    it 'finds local variables created by named captures in a paren LHS' do
+      input_tree =
+        [:program,
+         [[:def,
+           [:@ident, "abc", [1, 4]],
+           [:paren, [:params, [[:@ident, "a", [1, 8]]], nil, nil, nil, nil]],
+           [:bodystmt,
+            [[:void_stmt],
+             [:binary,
+              [:paren,
+               [[:var_ref, [:@ident, "foo", [1, 13]]],
+                [:var_ref, [:@ident, "bar", [1, 18]]],
+                [:regexp_literal,
+                 [[:@tstring_content, "name: (?<name>w+) (?<numba>d+)", [1, 24]]],
+                 [:@regexp_end, "/", [1, 54]]]]],
+              :=~,
+              [:var_ref, [:@ident, "a", [1, 60]]]],
+             [:var_ref, [:@ident, "name", [1, 63]]],
+             [:var_ref, [:@ident, "numba", [1, 69]]],
+             [:var_ref, [:@ident, "number", [1, 76]]]],
+            nil,
+            nil,
+            nil]]]]
+      output_tree =
+        [:program,
+         [[:def,
+           [:@ident, "abc", [1, 4]],
+           [:paren, [:params, [[:@ident, "a", [1, 8]]], nil, nil, nil, nil]],
+           [:bodystmt,
+            [[:void_stmt],
+             [:binary,
+              [:paren,
+               [[:zcall, [:@ident, "foo", [1, 13]]],
+                [:zcall, [:@ident, "bar", [1, 18]]],
+                [:regexp_literal,
+                 [[:@tstring_content, "name: (?<name>w+) (?<numba>d+)", [1, 24]]],
+                 [:@regexp_end, "/", [1, 54]]]]],
+              :=~,
+              [:var_ref, [:@ident, "a", [1, 60]]]],
+             [:var_ref, [:@ident, "name", [1, 63]]],
+             [:var_ref, [:@ident, "numba", [1, 69]]],
+             [:zcall, [:@ident, "number", [1, 76]]]],
+            nil,
+            nil,
+            nil]]]]
+      input_tree.should transform_to output_tree
+    end
   end
   describe 'error transformation' do
     it 'should push up module name errors' do
@@ -821,5 +973,183 @@ describe RipperPlus::Transformer do
       end
     end
     
+    it 'should wrap method definitions with duplicated arguments in error nodes' do
+      input_tree =
+        [:program,
+         [[:def,
+           [:@ident, "foo", [1, 4]],
+           [:paren,
+            [:params,
+             [[:@ident, "a", [1, 8]], [:@ident, "a", [1, 11]]],
+             nil,
+             nil,
+             nil,
+             nil]],
+           [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
+      output_tree =
+        [:program,
+         [[:error,
+           [:def,
+            [:@ident, "foo", [1, 4]],
+            [:paren,
+             [:params,
+              [[:@ident, "a", [1, 8]], [:@ident, "a", [1, 11]]],
+              nil,
+              nil,
+              nil,
+              nil]],
+            [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]]
+      input_tree.should transform_to output_tree
+    end
+    
+    it 'should wrap singleton method definitions with duplicated arguments in error nodes' do
+      input_tree =
+        [:program,
+         [[:defs,
+           [:var_ref, [:@kw, "self", [1, 4]]],
+           [:@period, ".", [1, 8]],
+           [:@ident, "foo", [1, 4]],
+           [:paren,
+            [:params,
+             [[:@ident, "a", [1, 8]], [:@ident, "a", [1, 11]]],
+             nil,
+             nil,
+             nil,
+             nil]],
+           [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]
+      output_tree =
+        [:program,
+         [[:error,
+           [:defs,
+            [:var_ref, [:@kw, "self", [1, 4]]],
+            [:@period, ".", [1, 8]],
+            [:@ident, "foo", [1, 4]],
+            [:paren,
+             [:params,
+              [[:@ident, "a", [1, 8]], [:@ident, "a", [1, 11]]],
+              nil,
+              nil,
+              nil,
+              nil]],
+            [:bodystmt, [[:void_stmt]], nil, nil, nil]]]]]
+      input_tree.should transform_to output_tree
+    end
+    
+    it 'should wrap stabby lambda definitions with duplicated arguments in error nodes' do
+      input_tree =
+        [:program,
+         [[:lambda,
+           [:paren,
+            [:params,
+             [[:@ident, "a", [1, 3]], [:@ident, "b", [1, 6]], [:@ident, "a", [1, 9]]],
+             nil,
+             nil,
+             nil,
+             nil]],
+           [[:void_stmt]]]]]
+      output_tree =
+        [:program,
+         [[:error,
+           [:lambda,
+            [:paren,
+             [:params,
+              [[:@ident, "a", [1, 3]], [:@ident, "b", [1, 6]], [:@ident, "a", [1, 9]]],
+              nil,
+              nil,
+              nil,
+              nil]],
+            [[:void_stmt]]]]]]
+      input_tree.should transform_to output_tree
+    end
+    
+    it 'should wrap block calls with duplicated block args in error nodes' do
+      input_tree =
+        [:program,
+         [[:method_add_block,
+           [:call,
+            [:var_ref, [:@kw, "self", [1, 0]]],
+            :".",
+            [:@ident, "foo!", [1, 5]]],
+           [:brace_block,
+            [:block_var,
+             [:params,
+              [[:@ident, "a", [1, 13]]],
+              [[[:@ident, "b", [1, 16]], [:@int, "3", [1, 18]]]],
+              nil,
+              [[:mlhs_paren,
+                [:mlhs_add_star,
+                 [[:mlhs_paren, [:@ident, "c", [1, 22]]]],
+                 [:@ident, "a", [1, 26]]]]],
+              nil],
+             nil],
+            [[:void_stmt]]]]]]
+      output_tree =
+        [:program,
+         [[:error,
+           [:method_add_block,
+            [:call,
+             [:var_ref, [:@kw, "self", [1, 0]]],
+             :".",
+             [:@ident, "foo!", [1, 5]]],
+            [:brace_block,
+             [:block_var,
+              [:params,
+               [[:@ident, "a", [1, 13]]],
+               [[[:@ident, "b", [1, 16]], [:@int, "3", [1, 18]]]],
+               nil,
+               [[:mlhs_paren,
+                 [:mlhs_add_star,
+                  [[:mlhs_paren, [:@ident, "c", [1, 22]]]],
+                  [:@ident, "a", [1, 26]]]]],
+               nil],
+              nil],
+             [[:void_stmt]]]]]]]
+      input_tree.should transform_to output_tree
+    end
+    
+    it 'should wrap block calls with duplicated block-local args in error nodes' do
+      input_tree =
+        [:program,
+         [[:method_add_block,
+           [:call,
+            [:var_ref, [:@kw, "self", [1, 0]]],
+            :".",
+            [:@ident, "foo!", [1, 5]]],
+           [:brace_block,
+            [:block_var,
+             [:params,
+              [[:@ident, "a", [1, 13]]],
+              [[[:@ident, "b", [1, 16]], [:@int, "3", [1, 18]]]],
+              nil,
+              [[:mlhs_paren,
+                [:mlhs_add_star,
+                 [[:mlhs_paren, [:@ident, "c", [1, 22]]]],
+                 [:@ident, "d", [1, 26]]]]],
+              nil],
+             [[:@ident, "e", [1, 30]], [:@ident, "a", [1, 33]]]],
+            [[:void_stmt]]]]]]
+      output_tree =
+        [:program,
+         [[:error,
+           [:method_add_block,
+            [:call,
+             [:var_ref, [:@kw, "self", [1, 0]]],
+             :".",
+             [:@ident, "foo!", [1, 5]]],
+            [:brace_block,
+             [:block_var,
+              [:params,
+               [[:@ident, "a", [1, 13]]],
+               [[[:@ident, "b", [1, 16]], [:@int, "3", [1, 18]]]],
+               nil,
+               [[:mlhs_paren,
+                 [:mlhs_add_star,
+                  [[:mlhs_paren, [:@ident, "c", [1, 22]]]],
+                  [:@ident, "d", [1, 26]]]]],
+               nil],
+              [[:@ident, "e", [1, 30]], [:@ident, "a", [1, 33]]]],
+             [[:void_stmt]]]]]]]
+      input_tree.should transform_to output_tree
+    end
   end
 end

metadata

@@ -1,8 +1,8 @@
 --- !ruby/object:Gem::Specification 
 name: ripper-plus
 version: !ruby/object:Gem::Version 
-  prerelease: 6
-  version: 1.1.0.pre2
+  prerelease: 
+  version: 1.2.0
 platform: ruby
 authors: 
 - Michael Edgar
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-05-02 00:00:00 Z
+date: 2011-05-04 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: rspec
@@ -98,16 +98,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      hash: 633908631425792624
+      hash: -1365054536111206632
       segments: 
       - 0
       version: "0"
 required_rubygems_version: !ruby/object:Gem::Requirement 
   none: false
   requirements: 
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version 
-      version: 1.3.1
+      version: "0"
 requirements: []
 
 rubyforge_project: