serializable_proc 0.1.0

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,21 @@
+## MAC OS
+.DS_Store
+
+## TEXTMATE
+*.tmproj
+tmtags
+
+## EMACS
+*~
+\#*
+.\#*
+
+## VIM
+*.swp
+
+## PROJECT::GENERAL
+coverage
+rdoc
+pkg
+
+## PROJECT::SPECIFIC

data/HISTORY.txt

@@ -0,0 +1,4 @@
+=== 0.1.0 (Apr 14, 2010)
+
+= 1st gem release !! [#ngty]
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 NgTzeYang
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,132 @@
+= serializable_proc
+
+As the name suggests, SerializableProc is a proc that can be serialized (marshalled).
+A proc is a closure, which consists of the code block defining it, and binding of
+local variables. SerializableProc's approach to serializability is to extract:
+
+1. the code from the proc (using ParseTree or RubyParser), and
+2. the local, instance, class & global variables reference within the proc from the
+   proc's binding, using deep copy via Marshal.load(Marshal.dump(var))
+
+A SerializableProc differs from the vanilla Proc in the following 2 ways:
+
+=== 1. Isolated variables
+
+Upon initializing, all variables (local, instance, class & global) within its context
+are extracted from the proc's binding, and are isolated from changes outside the proc's
+scope, thus, achieving a snapshot effect.
+
+  require 'rubygems'
+  require 'serializable_proc'
+
+  x, @x, @@x, $x = 'lx', 'ix', 'cx', 'gx'
+
+  s_proc = SerializableProc.new { [x, @x, @@x, $x].join(', ') }
+  v_proc = Proc.new { [x, @x, @@x, $x].join(', ') }
+
+  x, @x, @@x, $x = 'ly', 'iy', 'cy', 'gy'
+
+  s_proc.call # >> "lx, ix, cx, gx"
+  v_proc.call # >> "ly, iy, cy, gy"
+
+=== 2. Marshallable
+
+No throwing of TypeError when marshalling a SerializableProc:
+
+  Marshal.load(Marshal.dump(s_proc)).call # >> "lx, ix, cx, gx"
+  Marshal.load(Marshal.dump(v_proc)).call # >> TypeError (cannot dump Proc)
+
+== Installing It
+
+The religiously standard way:
+
+  $ gem install ParseTree serializable_proc
+
+Or on 1.9.* or JRuby:
+
+  $ gem install ruby_parser serializable_proc
+
+By default, SerializableProc attempts to load ParseTree, which supports better
+performance & offers many dynamic goodness. If ParseTree cannot be found,
+SerializableProc falls back to the RubyParser-based which suffers some gotchas due
+to its static analysis nature (see 'Gotchas' section).
+
+== Performance
+
+SerializableProc relies on ParseTree or RubyParser to do code extraction. While running
+in ParseTree mode, thanks to the goodness of dynamic code analysis, SerializableProc
+performs faster by a magnitude of abt 7.5 times for the same ruby, as illustrated with
+the following benchmark results (obtained from running the specs suite):
+
+  MRI & implementation    user      system    total     real
+  1.8.7p299 (ParseTree)   0.000000  0.000000  1.310000  1.312676
+  1.8.7p299 (RubyParser)  0.000000  0.010000  9.560000  9.706455
+  1.9.1p376 (RubyParser)  0.010000  0.010000  8.240000  8.288799
+
+(the above is run on my x86_64-linux):
+
+== Gotchas
+
+As RubyParser does only static code analysis, quite a bit of regexp matchings are needed
+to get SerializableProc to work in RubyParser mode. However, as our regexp kungfu is not
+perfect (yet), pls take note of the following:
+
+=== 1. Cannot have multiple initializing code block per line
+
+The following initializations throw SerializableProc::CannotAnalyseCodeError:
+
+  # Multiple SerializableProc.new per line
+  SerializableProc.new { x } ; SerializableProc.new { y }
+
+  # Multiple lambda per line (the same applies to proc & Proc.new)
+  x_proc = lambda { x } ; y_proc = lambda { y }
+  SerializableProc.new(&x_proc)
+
+  # Mixed lambda, proc & Proc.new per line
+  x_proc = proc { x } ; y_proc = lambda { y }
+  SerializableProc.new(&x_proc)
+
+=== 2. Limited ways to initialize code blocks
+
+Code block must be initialized with lambda, proc, Proc.new & SerializableProc.new,
+the following will throw SerializableProc::CannotAnalyseCodeError:
+
+  def create_serializable_proc(&block)
+    SerializableProc.new(&block)
+  end
+
+  create_serializable_proc { x }
+
+But the following will work as expected:
+
+  x_proc = lambda { x }
+  create_serializable_proc(&x_proc)
+
+== TODO (just brain-dumping)
+
+1. Provide flexibility for user to specify whether global variables should be isolated,
+   because globals are globals, it may sometimes be useful to use/manipulate the globals
+   within the context that the SerializableProc is called, instead of when it is
+   initialized
+2. The RubyParser-based implementation probably need alot more optimization to catch up
+   on ParseTree-based one
+3. Implementing alternative means of extracting the code block without requiring help
+   of ParseTree or RubyParser
+4. Implement workaround to tackle line-numbering bug in JRuby, which causes the
+   RubyParser-based implementation to fail, for more info abt JRuby's line-numbering
+   bug, see http://stackoverflow.com/questions/3454838/jruby-line-numbering-problem
+
+== Note on Patches/Pull Requests
+
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by
+  itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+== Copyright
+
+Copyright (c) 2010 NgTzeYang. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,136 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+    gem.name = "serializable_proc"
+    gem.summary = %Q{Proc that can be serialized (as the name suggests)}
+    gem.description = %Q{
+      Give & take, serializing a ruby proc is possible, though not a perfect one.
+      Requires either ParseTree (faster) or RubyParser (& Ruby2Ruby).
+    }
+    gem.email = "ngty77@gmail.com"
+    gem.homepage = "http://github.com/ngty/serializable_proc"
+    gem.authors = ["NgTzeYang"]
+    gem.add_dependency "ruby2ruby", ">= 1.2.4"
+    gem.add_development_dependency "bacon", ">= 0"
+    # Plus one of the following groups:
+    #
+    # 1). ParseTree (better performance + dynamic goodness, but not supported on java & 1.9.*)
+    # >> gem.add_dependency "ParseTree", ">= 3.0.5"
+    #
+    # 2). RubyParser (supported for all)
+    # >> gem.add_dependency "ruby_parser", ">= 2.0.4"
+
+    unless RUBY_PLATFORM =~ /java/i or RUBY_VERSION.include?('1.9.')
+      gem.post_install_message = %Q{
+ /////////////////////////////////////////////////////////////////////////////////
+
+  ** SerializableProc **
+
+  You are installing SerializableProc on a ruby platform & version that supports
+  ParseTree. With ParseTree, u can enjoy better performance of SerializableProc,
+  as well as other dynamic code analysis goodness, as compared to the default
+  implementation using RubyParser's less flexible static code analysis.
+
+  Anyway, u have been informed, SerializableProc will fallback on its default
+  implementation using RubyParser.
+
+ /////////////////////////////////////////////////////////////////////////////////
+      }
+    end
+  end
+
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:spec) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |spec|
+    spec.libs << 'spec'
+    spec.pattern = 'spec/**/*_spec.rb'
+    spec.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :spec => :check_dependencies
+
+begin
+  require 'reek/adapters/rake_task'
+  Reek::RakeTask.new do |t|
+    t.fail_on_error = true
+    t.verbose = false
+    t.source_files = 'lib/**/*.rb'
+  end
+rescue LoadError
+  task :reek do
+    abort "Reek is not available. In order to run reek, you must: sudo gem install reek"
+  end
+end
+
+begin
+  require 'roodi'
+  require 'roodi_task'
+  RoodiTask.new do |t|
+    t.verbose = false
+  end
+rescue LoadError
+  task :roodi do
+    abort "Roodi is not available. In order to run roodi, you must: sudo gem install roodi"
+  end
+end
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "serializable_proc #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+# Benchmarking
+task :benchmark, :task, :times do |t, args|
+  times, task = (args.times || 5).to_i.method(:times), args.task
+  title = " ~ Benchmark Results for Task :#{task} ~ "
+  results = [%w{nth}, %w{user}, %w{system}, %w{total}, %w{real}]
+
+  # Running benchmarking & collecting results
+  require 'benchmark'
+  times.call do |i|
+    result = Benchmark.measure{ Rake::Task[task].execute }.to_s
+    user, system, total, real =
+      result.match(/^\s*(\d+\.\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)\s+\(\s*(\d+\.\d+)\)$/)[1..-1]
+    ["##{i.succ}", user, system, total, real].each_with_index{|val, j| results[j] << val }
+  end
+
+  # Formatting benchmarking results
+  formatted_results = results.map do |rs|
+    width = rs.map(&:to_s).map(&:size).max
+    rs.map{|r| '  ' + r.ljust(width, ' ') }
+  end.transpose.map{|row| row.join }
+
+  # Showdown .. printout
+  line = '=' * ([title.size, formatted_results.map(&:size).max].max + 2)
+  puts [line, title, formatted_results.join("\n"), line].join("\n\n")
+
+end
+

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/serializable_proc/binding.rb

@@ -0,0 +1,53 @@
+class SerializableProc
+
+  class CannotSerializeVariableError < Exception ; end
+
+  class Binding
+
+    include Marshalable
+    marshal_attr :vars
+
+    def initialize(binding, sexp)
+      @vars, sexp_str = {}, sexp.inspect
+      while m = sexp_str.match(/^(.*?s\(:(?:l|g|c|i)var, :([^\)]+)\))/)
+        ignore, var = m[1..2]
+        sexp_str.sub!(ignore,'')
+        begin
+          val = binding.eval(var) rescue nil
+          @vars.update(Sandboxer.fvar(var) => mclone(val))
+        rescue TypeError
+          raise CannotSerializeVariableError.new("Variable #{var} cannot be serialized !!")
+        end
+      end
+    end
+
+    def eval!
+      @binding ||= (
+        set_vars = @vars.map{|(k,v)| "#{k} = Marshal.load(%|#{mdump(v)}|)" } * '; '
+        (binding = Kernel.binding).eval(set_vars)
+        binding.extend(Extensions)
+      )
+    end
+
+    module Extensions
+      def self.extended(base)
+        class << base
+
+          alias_method :orig_eval, :eval
+
+          def eval(str)
+            begin
+              @fvar = Sandboxer.fvar(str).to_s
+              orig_eval(@fvar)
+            rescue NameError => e
+              msg = e.message.sub(@fvar, str)
+              raise NameError.new(msg)
+            end
+          end
+
+        end
+      end
+    end
+
+  end
+end

data/lib/serializable_proc/marshalable.rb

@@ -0,0 +1,51 @@
+class SerializableProc
+  module Marshalable
+
+    def self.included(base)
+      base.class_eval do
+        extend ClassMethods
+        include InstanceMethods
+      end
+    end
+
+    module ClassMethods
+
+      protected
+
+        def marshal_attrs(*attrs)
+          attrs = attrs.map{|attr| :"@#{attr}" }
+          self.class_eval do
+            define_method(:marshalable_attrs) { attrs }
+          end
+        end
+
+        alias_method :marshal_attr, :marshal_attrs
+
+    end
+
+    module InstanceMethods
+
+      def marshal_dump
+        marshalable_attrs.map{|attr| instance_variable_get(attr) }
+      end
+
+      def marshal_load(data)
+        [data].flatten.each_with_index do |val, i|
+          instance_variable_set(marshalable_attrs[i], val)
+        end
+      end
+
+      protected
+
+        def mdump(val)
+          Marshal.dump(val).gsub('|','\|')
+        end
+
+        def mclone(val)
+          Marshal.load(mdump(val))
+        end
+
+    end
+
+  end
+end

data/lib/serializable_proc/parsers/pt.rb

@@ -0,0 +1,16 @@
+class SerializableProc
+  module Parsers
+    module PT
+      class << self
+        def process(block)
+          if Object.const_defined?(:ParseTree)
+            sexp = block.to_sexp
+            runnable_code = RUBY_2_RUBY.process(Sandboxer.fsexp(sexp))
+            extracted_code = RUBY_2_RUBY.process(eval(sexp.inspect))
+            [{:runnable => runnable_code, :extracted => extracted_code}, sexp]
+          end
+        end
+      end
+    end
+  end
+end

data/lib/serializable_proc/parsers/rp.rb

@@ -0,0 +1,93 @@
+class SerializableProc
+
+  class CannotAnalyseCodeError < Exception ; end
+
+  module Parsers
+    module RP
+      class << self
+
+        def process(klass, file, line)
+          const_set(:RUBY_PARSER, RubyParser.new) unless const_defined?(:RUBY_PARSER)
+          @klass, @file, @line = klass, file, line
+          extract_code_and_sexp
+        end
+
+        private
+
+          def extract_code_and_sexp
+            sexp_str, remaining, type, marker = extract_sexp_args
+            while frag = remaining[/^([^\)]*\))/,1]
+              begin
+                sexp = eval(sexp_str += frag) # this throws SyntaxError if sexp is invalid
+                runnable_code, extracted_code = [
+                  RUBY_2_RUBY.process(Sandboxer.fsexp(sexp)),
+                  RUBY_2_RUBY.process(eval(sexp.inspect))
+                ].map do |code|
+                  unescape_magic_vars(code).sub(/#{marker}\s*;?/m,'').sub(type,'lambda')
+                end
+                return [{:runnable => runnable_code, :extracted => extracted_code}, sexp]
+              rescue SyntaxError
+                remaining.sub!(frag,'')
+              end
+            end
+          end
+
+          def extract_sexp_args
+            raw, type, marker = raw_sexp_and_marker
+            rq = lambda{|s| Regexp.quote(s) }
+            regexp = Regexp.new([
+              '^(.*(', (
+                case type
+                when /(#{@klass}|Proc)/ then rq["s(:iter, s(:call, s(:const, :#{$1}), :new, s(:arglist)),"]
+                else rq['s(:iter, s(:call, nil, :'] + '(?:proc|lambda)' + rq[', s(:arglist']
+                end
+              ),
+              '.*?',
+              rq["s(:call, nil, :#{marker}, s(:arglist))"],
+              '))(.*)$'
+            ].join, Regexp::MULTILINE)
+            [raw.match(regexp)[2..3], type, marker].flatten
+          end
+
+
+          def raw_sexp_and_marker
+            %W{#{@klass}\.new lambda|proc|Proc\.new}.each do |declarative|
+              regexp = /^(.*?(#{declarative})\s*(do|\{)\s*(\|([^\|]*)\|\s*)?)/
+              raw = raw_code
+              lines1, lines2 = [(0 .. (@line - 2)), (@line.pred .. -1)].map{|r| raw[r] }
+              match, type = lines2[0].match(regexp)[1..2] rescue next
+
+              if lines2[0] =~ /^(.*?\W)?(#{declarative})(\W.*?\W(#{declarative}))+(\W.*)?$/
+                msg = "Static code analysis can only handle single occurrence of '%s' per line !!" %
+                  declarative.split('|').join("'/'")
+                raise CannotAnalyseCodeError.new(msg)
+              elsif lines2[0] =~ /^(.*?\W)?(#{declarative})(\W.*)?$/
+                marker = "__serializable_proc_marker_#{@line}__"
+                lines = lines1.join + escape_magic_vars(lines2[0].sub(match, match+marker+';') + lines2[1..-1].join)
+                return [RUBY_PARSER.parse(lines).inspect, type, marker]
+              end
+            end
+            raise CannotAnalyseCodeError.new('Cannot find specified initializer !!')
+          end
+
+          def escape_magic_vars(s)
+            %w{__FILE__ __LINE__}.inject(s) do |s, var|
+              s.gsub(var, "__serializable_proc_#{var.downcase}__")
+            end
+          end
+
+          def unescape_magic_vars(s)
+            %w{__FILE__ __LINE__}.inject(s) do |s, var|
+              s.gsub("__serializable_proc_#{var.downcase}__", var)
+            end
+          end
+
+          def raw_code
+            File.readlines(@file)
+          end
+
+      end
+    end
+  end
+end
+

data/lib/serializable_proc/parsers.rb

@@ -0,0 +1,8 @@
+class SerializableProc
+  module Parsers
+    RUBY_2_RUBY = Ruby2Ruby.new
+  end
+end
+
+require 'serializable_proc/parsers/pt'
+require 'serializable_proc/parsers/rp'

data/lib/serializable_proc/sandboxer.rb

@@ -0,0 +1,24 @@
+class SerializableProc
+  module Sandboxer
+    class << self
+
+      def fsexp(sexp)
+        n_sexp, t_sexp = nil, sexp.inspect
+        while m = t_sexp.match(/^(.*?s\(:)((i|l|c|g)(asgn|var|vdecl))(,\ :)((|@|@@|\$)([\w]+))(\)|,)/)
+          orig, prepend, _, type, declare, join, _, _, name, append = m[0..-1]
+          declare.sub!('vdecl','asgn')
+          n_sexp = "#{n_sexp}#{prepend}l#{declare}#{join}#{type}var_#{name}#{append}"
+          t_sexp.sub!(orig,'')
+        end
+        eval(n_sexp ? "#{n_sexp}#{t_sexp}" : sexp.inspect)
+      end
+
+      def fvar(var)
+        @translate_var_maps ||= {'@' => 'ivar_', '@@' => 'cvar_', '$' => 'gvar_', '' => 'lvar_'}
+        m = var.to_s.match(/^(|@|@@|\$)(\w+)$/)
+        var.to_s.sub(m[1], @translate_var_maps[m[1]]).to_sym
+      end
+
+    end
+  end
+end