sourcify 0.5.0 → 0.6.0.rc1

data/.gitignore

@@ -20,4 +20,5 @@ pkg
 Gemfile.lock
 
 ## PROJECT::SPECIFIC
+.rvmrc
 tmp

data/HISTORY.txt

@@ -1,3 +1,8 @@
+=== Current (pre 0.6.0)
+
+* adds Method#to_source, Method#to_sexp & Method#to_raw_source to MRI-1.9.2 (issue#3)
+* extensive refactoring to existing proc support to support the above
+
 === 0.5.0 (May 2, 2011)
 
 * adds Proc#to_raw_source that supports extracting of raw code, as it is written in

data/README.rdoc

@@ -20,6 +20,7 @@ uses a home-baked ragel-generated scanner to extract the proc code. Further proc
 with RubyParser & Ruby2Ruby to ensure 100% with ParseTree (yup, there is no denying that
 i really like ParseTree).
 
+
 == Installing It
 
 The religiously standard way:
@@ -30,9 +31,8 @@ Or on 1.9.* or JRuby:
 
   $ gem install ruby_parser file-tail sourcify
 
-== Using It
 
-Sourcify adds 4 methods to Proc:
+== Sourcify adds 4 methods to Proc
 
 === 1. Proc#to_source
 
@@ -101,6 +101,83 @@ consistency under 1.8.*:
   lambda { x + y }.source_location
   # >> ["/tmp/test.rb", 5]
 
+
+== Sourcify adds 3 methods to Method
+
+*IMPORTANT*: These only work for MRI-1.9.2, as currently, only it supports
+(1) discovering of the original source location with Method#source_location,
+and (2) reliably determinig a method's parameters with Method#parameters.
+Attempting to use these methods on other rubies will raise
+Sourcify::PlatformNotSupportedError.
+
+*NOTE*: The following works for methods defined using both def .. end &
+Module#define_method. However, when a method is defined using the later
+approach, sourcify uses Proc#to_source to handle the processing, thus, the
+usual gotchas related to proc source extraction apply.
+
+=== 1. Method#to_source
+
+Returns the code representation of the method:
+
+  require 'sourcify'
+
+  class MyMath
+    def self.sum(x, y)
+      x + y # (blah)
+    end
+  end
+
+  MyMath.method(:sum).to_source
+  # >> "def sum(x, y)
+  # >>   (x + y)
+  # >> end"
+
+Just like the Proc#to_source equivalent, u can set :strip_enclosure => true
+to extract only the body within.
+
+=== 2. Method#to_sexp
+
+Returns the S-expression of the method:
+
+  require 'sourcify'
+
+  class MyMath
+    def self.sum(x, y)
+      x + y # (blah)
+    end
+  end
+
+  MyMath.method(:sum).to_sexp
+  >> s(:defn,
+  >>  :sum,
+  >>  s(:args, :x, :y),
+  >>  s(:scope, s(:block, s(:call, s(:lvar, :x), :+, s(:arglist, s(:lvar, :y))))))
+
+Just like the Proc#to_sexp equivalent, u can set :strip_enclosure => true
+to extract only the body within.
+
+=== 3. Method#to_raw_source
+
+Unlike Method#to_source, which returns code that retains only functional aspects,
+fetching of raw source returns the method's raw code, including fluff like comments:
+
+  require 'sourcify'
+
+  class MyMath
+    def self.sum(x, y)
+      x + y # (blah)
+    end
+  end
+
+  MyMath.method(:sum).to_source
+  # >> "def sum(x, y)
+  # >>   x + y # (blah)
+  # >> end"
+
+Just like the Proc#to_raw_source equivalent, u can set :strip_enclosure => true
+to extract only the body within.
+
+
 == Performance
 
 Performance is embarassing for now, benchmarking results for processing 500 procs
@@ -117,6 +194,7 @@ probably become better & faster as my knowlegde & skills with ragel improve. Als
 instead of generating a pure ruby scanner, we can generate native code (eg. C or java, or
 whatever) instead. As i'm a C & java noob, this will probably take some time to realize.
 
+
 == Gotchas
 
 Nothing beats ParseTree's ability to access the runtime AST, it is a very powerful feature.
@@ -210,6 +288,7 @@ Under the hood, sourcify relies on RubyParser to yield s-expression, and since R
 does not yet fully handle 1.8.7 & 1.9.* syntax, you will get a nasty Racc::ParseError when
 you have any code that is not compatible with 1.8.6.
 
+
 == Is it really working ??
 
 Sourcify spec suite currently passes in the following rubies:
@@ -226,6 +305,7 @@ For projects:
 
 (TODO: the more the merrier)
 
+
 == Projects using it
 
 Projects using sourcify include:
@@ -233,6 +313,7 @@ Projects using sourcify include:
 * ruote[http://ruote.rubyforge.org/]
 * dm-ambition[https://github.com/dkubb/dm-ambition]
 
+
 == Additional Resources
 
 Sourcify is heavily inspired by many ideas gathered from the ruby community:
@@ -243,8 +324,9 @@ Sourcify is heavily inspired by many ideas gathered from the ruby community:
 The sad fact that Proc#to_source wouldn't be available in the near future:
 * http://redmine.ruby-lang.org/issues/show/2080
 
+
 == Note on Patches/Pull Requests
- 
+
 * Fork the project.
 * Make your feature addition or bug fix.
 * Add tests for it. This is important so I don't break it in a
@@ -253,6 +335,7 @@ The sad fact that Proc#to_source wouldn't be available in the near future:
   (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
+
 == Copyright
 
 Copyright (c) 2010 NgTzeYang. See LICENSE for details.

data/Rakefile

@@ -8,20 +8,38 @@ task :default => :spec
 
 RUBIES = {
   :parsetree => [
-    'ruby-1.8.6-p420@sourcify-parsetree',
-    'ruby-1.8.7-p334@sourcify-parsetree',
-    'ree-1.8.7-2011.03@sourcify-parsetree'
+    [nil, 'ruby-1.8.6-p420@sourcify-parsetree'],
+    [nil, 'ruby-1.8.7-p334@sourcify-parsetree'],
+    [nil, 'ree-1.8.7-2011.03@sourcify-parsetree']
   ],
   :static => [
-    'ruby-1.8.6-p420@sourcify',
-    'ruby-1.8.7-p334@sourcify',
-    'ree-1.8.7-2011.03@sourcify',
-    'jruby-1.6.1@sourcify',
-    'ruby-1.9.1-p378@sourcify',
-    'ruby-1.9.2-p180@sourcify',
+    [nil, 'ruby-1.8.6-p420@sourcify'],
+    [nil, 'ruby-1.8.7-p334@sourcify'],
+    [nil, 'ree-1.8.7-2011.03@sourcify'],
+    [nil, 'jruby-1.6.3@sourcify'],
+    [nil, 'ruby-1.9.1-p378@sourcify'],
+    ['METHOD_TO_SOURCE=true', 'ruby-1.9.2-p290@sourcify'],
+
+    # NOTE: This doesn't support Method#to_source (& friends) yet yet due to
+    # jruby's Method#parameters bug, see http://jira.codehaus.org/browse/JRUBY-5954
+    ['JRUBY_OPTS="--1.9"', 'jruby-1.6.3@sourcify'],
+
   ]
 }
 
+def run_spec_script_for(envs_and_rubies)
+  envs_and_rubies.group_by{|arry| arry[0..-2] }.each do |envs, arry|
+    rubies = arry.map(&:last)
+    declared_envs = ['export MUTE_BACON=true']
+    declared_envs << envs.map{|env| env ? "export #{env}" : nil }.compact
+
+    system [
+      declared_envs.flatten.join('; '),
+      "rvm #{rubies.join(',')} exec #{SPEC_SCRIPT}"
+    ].join('; ')
+  end
+end
+
 # ///////////////////////////////////////////////////////////
 # Running Specs
 # ///////////////////////////////////////////////////////////
@@ -32,20 +50,34 @@ end
 
 desc "Run specs in all rubies (both ParseTree & static scanner modes)"
 task :'spec:all' do
-  system 'export MUTE_BACON=true; rvm ' +
-    RUBIES.values.flatten.join(',') + ' exec ' + SPEC_SCRIPT
+  run_spec_script_for RUBIES.values.flatten(1)
 end
 
 desc "Run specs in rubies supporting ParseTree mode"
 task :'spec:parsetree' do
-  system 'export MUTE_BACON=true; rvm ' +
-    RUBIES[:parsetree].join(',') + ' exec ' + SPEC_SCRIPT
+  run_spec_script_for RUBIES[:parsetree]
 end
 
 desc "Run specs in rubies supporting static scanner mode"
 task :'spec:static' do
-  system 'export MUTE_BACON=true; rvm ' +
-    RUBIES[:static].join(',') + ' exec ' + SPEC_SCRIPT
+  run_spec_script_for RUBIES[:static]
+end
+
+# ///////////////////////////////////////////////////////////
+# Build ruby files from ragel definitions
+# ///////////////////////////////////////////////////////////
+desc "Run ragel to generate ruby scanner files from ragel definitions"
+task :ragel do
+  %w{method proc}.each do |m|
+    common_dir = File.expand_path('../lib/sourcify/common/ragel', __FILE__)
+    ragel_dir = File.expand_path("../lib/sourcify/#{m}/parser", __FILE__)
+    ragel_file = File.join(ragel_dir, 'raw_scanner.rl')
+    ruby_file = File.join(ragel_dir, 'raw_scanner.rb')
+    File.delete(ruby_file) rescue nil
+    puts 'Processing %s -> %s' %
+      [ragel_file, ruby_file].map{|f| f.sub(File.expand_path('../', __FILE__) + '/', '')}
+    system("ragel -I #{common_dir} -R #{ragel_file}")
+  end
 end
 
 # ///////////////////////////////////////////////////////////

data/lib/sourcify.rb

@@ -12,6 +12,17 @@ rescue LoadError
 end
 
 module Sourcify #:nodoc:
+
+  IS_19x = RUBY_VERSION.include?('1.9.')
+
+  HAS_RIPPER =
+    begin
+      require 'ripper'
+      true
+    rescue LoadError
+      false
+    end
+
   class << self
 
     def require_rb(*args)
@@ -27,5 +38,7 @@ module Sourcify #:nodoc:
 end
 
 Sourcify.require_rb('facets')
+Sourcify.require_rb('errors')
 Sourcify.require_rb('version')
 Sourcify.require_rb('proc')
+Sourcify.require_rb('method')

data/lib/sourcify/common/parser/converter.rb

@@ -0,0 +1,29 @@
+module Sourcify
+  module Common
+    class Parser #:nodoc:all
+      class Converter
+        class << self
+
+          RUBY_PARSER = RubyParser.new
+          RUBY_2_RUBY = Ruby2Ruby.new
+
+          def to_sexp(code, file = nil)
+            retried = false
+            begin
+              RUBY_PARSER.reset
+              (retried ? RubyParser.new : RUBY_PARSER).parse(*[code, file].compact)
+            rescue Racc::ParseError, SyntaxError
+              return nil if retried
+              retried = true; retry
+            end
+          end
+
+          def to_code(sexp)
+            RUBY_2_RUBY.process(Sexp.from_array(sexp.to_a))
+          end
+
+        end
+      end
+    end
+  end
+end

data/lib/sourcify/common/parser/raw_scanner/comment.rb

@@ -0,0 +1,23 @@
+module Sourcify
+  module Common
+    class Parser
+      module RawScanner #:nodoc:all
+        class Comment
+
+          def <<(content)
+            (@contents ||= []) << content
+          end
+
+          def to_s
+            @contents.join
+          end
+
+          def closed?
+            @contents[-1].split("\n")[-1].strip == '=end'
+          end
+
+        end
+      end
+    end
+  end
+end

data/lib/sourcify/common/parser/raw_scanner/counter.rb

@@ -0,0 +1,43 @@
+module Sourcify
+  module Common
+    class Parser
+      module RawScanner #:nodoc:all
+        class Counter
+
+          attr_reader :counts
+
+          def initialize
+            @counts = [0,0]
+          end
+
+          def started?
+            @counts.any?(&:nonzero?)
+          end
+
+          def just_started?
+            @counts.any?{|count| count == 1 }
+          end
+
+          def balanced?
+            @counts.any?(&:zero?)
+          end
+
+          def decrement
+            (0..1).each{|i| @counts[i] -= 1 unless @counts[i].zero? }
+          end
+
+          def increment(val = 1)
+            if val.is_a?(Range)
+              @counts[0] += val.first
+              @counts[1] += val.last
+            else
+              (0..1).each{|i| @counts[i] += 1 }
+            end
+          end
+
+        end
+
+      end
+    end
+  end
+end

data/lib/sourcify/common/parser/raw_scanner/dstring.rb

@@ -0,0 +1,60 @@
+module Sourcify
+  module Common
+    class Parser
+      module RawScanner #:nodoc:all
+        class DString < Struct.new(:tag)
+
+          # To suppress 'warning: Object#type is deprecated; use Object#class' when
+          # evaluating string
+          attr_reader :type
+
+          def <<(content)
+            (@contents ||= []) << content
+          end
+
+          def to_s
+            @contents.join
+          end
+
+          def closed?
+            evaluable? && parsable?
+          end
+
+          private
+
+            CLOSING_TAGS = {'(' => ')', '[' => ']', '<' => '>', '{' => '}'}
+
+            def evaluable?
+              @contents[-1][-1].chr == end_tag
+            end
+
+            def parsable?
+              begin
+                RubyParser.new.parse(safe_contents)
+                true
+              rescue Exception
+                false
+              end
+            end
+
+            def safe_contents
+              # NOTE: %x & ` strings are dangerous to eval cos they execute shell commands,
+              # thus we convert them to normal strings 1st
+              to_s.gsub(/(%x)(\W|\_)/, '%Q\2').gsub(/.{0,2}(`)/) do |s|
+                s =~ /^(%Q|%W|%r|%x|.?%|.?\\)/ ? s : s.sub(/`$/,'%Q`')
+              end
+            end
+
+            def start_tag
+              @start_tag ||= tag[-1].chr
+            end
+
+            def end_tag
+              @end_tag ||= (CLOSING_TAGS[start_tag] || start_tag)
+            end
+
+        end
+      end
+    end
+  end
+end

data/lib/sourcify/common/parser/raw_scanner/extensions.rb

@@ -0,0 +1,121 @@
+%w{heredoc comment dstring counter}.each do |f|
+  Sourcify.require_rb('common', 'parser', 'raw_scanner', f)
+end
+
+module Sourcify
+  module Common
+    class Parser
+      module RawScanner #:nodoc:all
+        module Extensions
+
+          class Escape < Exception; end
+
+          def process(data, opts={})
+            begin
+              @start_pattern = opts[:start_pattern] || /.*/
+              @body_matcher = opts[:body_matcher] || lambda{|_| true }
+              @stop_on_newline = opts[:stop_on_newline]
+              @results, @data = [], data.unpack("c*")
+              reset_attributes
+              execute!
+            rescue Escape
+              @results
+            end
+          end
+
+          def data_frag(range)
+            @data[range].pack('c*')
+          end
+
+          def push(key, ts, te)
+            @tokens << [key, data_frag(ts .. te.pred)]
+          end
+
+          def push_dstring(ts, te)
+            data = data_frag(ts .. te.pred)
+            @dstring ||= DString.new(data[%r{^("|`|/|%(?:Q|W|r|x|)(?:\W|_))},1])
+            @dstring << data
+            return true unless @dstring.closed?
+            @tokens << [:dstring, @dstring.to_s]
+            @dstring = nil
+          end
+
+          def push_comment(ts, te)
+            data = data_frag(ts .. te.pred)
+            @comment ||= Comment.new
+            @comment << data
+            return true unless @comment.closed?
+            @tokens << [:comment, @comment.to_s]
+            @comment = nil
+          end
+
+          def push_heredoc(ts, te)
+            data = data_frag(ts .. te.pred)
+            unless @heredoc
+              indented, tag = data.match(/\<\<(\-?)['"]?(\w+)['"]?$/)[1..3]
+              @heredoc = Heredoc.new(tag, !indented.empty?)
+            end
+            @heredoc << data
+            return true unless @heredoc.closed?(data_frag(te .. te))
+            @tokens << [:heredoc, @heredoc.to_s]
+            @heredoc = nil
+          end
+
+          def push_label(ts, te)
+            # NOTE: 1.9.* supports label key, which RubyParser cannot handle, thus
+            # conversion is needed.
+            @tokens << [:symbol, ':' + data_frag(ts .. te - 2)]
+            @tokens << [:space, ' ']
+            @tokens << [:assoc, '=>']
+          end
+
+          def preceded_with?(*args)
+            prev_token = @tokens[-1][0] == :space ? @tokens[-2] : @tokens[-1]
+            !([args].flatten & prev_token).empty? rescue nil
+          end
+
+          def increment_lineno
+            @lineno += 1
+            raise Escape \
+              if @stop_on_newline || !@results.empty? || (@results.empty? && @rejecting_block)
+          end
+
+          def codified_tokens
+            @tokens.map(&:last).join
+          end
+
+          def reset_attributes
+            @tokens, @lineno = [], 1
+            @heredoc, @dstring, @comment, @rejecting_block = nil
+          end
+
+          def offset_attributes
+            @lineno = 1 # Fixing JRuby's lineno bug (see http://jira.codehaus.org/browse/JRUBY-5014)
+            @tokens = [@tokens[-1]] unless @tokens.empty?
+          end
+
+          if HAS_RIPPER
+
+            def valid?(snippet, validate_as = nil)
+              sexp = Ripper.sexp(snippet)
+
+              case validate_as
+              when :hash then sexp[-1][0][0] == :hash
+              when nil then sexp && true
+              else raise ArgumentError
+              end
+            end
+
+          else
+
+            def valid?(snippet, _ = nil)
+              RubyParser.new.parse("#{snippet}") rescue false
+            end
+
+          end
+
+        end
+      end
+    end
+  end
+end