crystalizer 0.2.2

data/Changelog

@@ -0,0 +1,27 @@
+Changelog
+
+-- 0.2.0
+
+* New features:
+  * New plugin architecture for optimizations and other things
+  * Optimization plugins: const_cache, builtin_methods, inline_methods,
+    case_optimize
+  * Other plugins: require_include, warnings
+  * Eval2C for compilation at runtime
+  * Available as gem now
+
+* Improvements:
+  * Implemented control flow "through" rescue/ensure clauses
+  * Better vmode handling
+  * Support for Ruby 1.8.5 and 1.8.6
+  * Improved performance for global variables
+
+* Fixes:
+  * Workaround for an issue with ALLOCA_N in while loops
+  * Fixed a problem with case/when (missing newline nodes)
+  * Removed libraries from compile command
+  * Various other small fixes
+
+-- 0.1.0
+
+* Initial release

data/README

@@ -0,0 +1,79 @@
+----------- JIT -------------
+
+The crystalizer is a loose wrapper for a ruby2c converter called ruby2cext.  It takes your ruby code, creates its rubyC equivalent, and re-loadsthe C version over the top of it, thus effectively JIT-ing your existing Ruby code.  It is called crystalizer because it effectively should be called *after* setting up existing classes and methods, so that when it can "crystalize" the calls to existing methods, by cacheing their location.  Thus, it is not 100% ruby compatible, and currently loses some backtrace information.  It has proven quite a bit faster than 1.9 in some benchmarks, though it is currently only 1.8 compatible [presumably if it were 1.9 compatible, it might be even faster].
+
+Synopsis:
+
+require 'concretizer'
+
+klass.concretize! # concretizes all interior methods of this class
+
+of, if you have several classes and don't want to have to remember exactly which ones to concretize:
+
+Ruby2CExtension::Concretize.crystalize_after_first_time_through {
+    # do some stuff here
+    # after completing this block, it will concretize any and all classes it ran into in the course of the block
+    # so the second and third times through this block should be faster [or any other code that subsequently calls those classes]
+}
+
+And for the daring:
+
+Ruby2CExtension::Conretize.concretize_all!  # optimizes all known classes in the current Ruby VM -- takes forever, might discover bugs in edge cases that aren't covered.
+
+Also an option is the rb2cx executable
+$ rb2cx ruby_filename.rb
+
+creates ruby_filename.{c, so} based on ruby_filename.rb.
+Thus you could run the ruby script's c equivalent thus:
+$ ruby -rruby_filename.so -e ''
+
+
+Much thanks to the original author of Ruby2CExtension, since this is only a wrapper to his work.
+
+Note that crystalize won't crystalize existing procs in the system--they stay as pure Ruby--it crystalizes methods and future procs created from/within those methods, however. 
+
+Ruby2CExtension
+===============
+
+Ruby2CExtension is a Ruby to C extension translator/compiler. It takes any Ruby
+source file, parses it using Ruby's builtin parser and then translates the
+abstract syntax tree into "equivalent" C extension code.
+
+
+Requirements
+------------
+
+* Ruby 1.8.4, 1.8.5 or 1.8.6 (possibly 1.8.7--haven't tried)
+* RubyNode (available at http://rubynode.rubyforge.org/)
+
+
+Installation
+------------
+
+Just run (as root): TODO
+
+  gem install ruby2cext
+
+Or if you do not use the gem:
+
+  ruby setup.rb
+
+This will install Ruby2CExtension to the default locations. Optionally you can
+supply some options to setup.rb to customize the installation (see "ruby
+setup.rb --help").
+
+
+Documentation
+-------------
+
+Please see doc/index.html for more documentation.
+
+
+License
+-------
+
+Copyright 2006-2007 Dominik Bathon <dbatml@gmx.de>
+
+Ruby2CExtension is licensed under the same terms as Ruby.
+
+http://github.com/rdp/crystalizer

data/Rakefile

@@ -0,0 +1,24 @@
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "crystalizer"
+    # 0.2.0.
+    #       2 actually works with concretize now
+    s.author = "Dominik Bathon, rogerdpack"
+    s.email = "dbatml@gmx.de"
+#    s.homepage = "http://ruby2cext.rubyforge.org/"
+    s.summary = "Ruby2CExtension is a Ruby to C extension translator/compiler/concretizer."
+    s.add_dependency("rubynode", ">= 0.1.1")
+    s.add_dependency("sane")
+    s.add_dependency("event_hook")
+    s.files.exclude '**/temp*'
+    s.add_dependency("backports")
+    s.add_dependency("ruby2ruby")
+    s.add_dependency("ParseTree")
+    s.add_development_dependency("assert2")
+    s.add_development_dependency("backtracer")
+  end
+
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end

data/TODO

@@ -0,0 +1,14 @@
+benchmark it against 1.9, etc.
+
+make sure it works in linux [should pass tests]
+
+can it run rails? do I need to avoid binding/eval methods etc? 
+
+benchmarks on rails?
+
+fix inline_builtin [meh]
+fix the other bwokded on
+
+make it compile faster [possibilities: fork, can methods compare to those we know are C-y?]
+
+is it faster to compile object's methods down into the class itself? [benchmarks]

data/VERSION

@@ -0,0 +1 @@
+0.2.2

data/benchmarks/bench.rb

@@ -0,0 +1,129 @@
+# some benchmarks from ruby2cext that it used against itself
+
+require "benchmark"
+
+def fib(n)
+    if n < 2
+        n
+    else
+        fib(n - 1) + fib(n - 2)
+    end
+end
+
+def mkmatrix(rows, cols)
+    count = 1
+    mx = Array.new(rows)
+    (0 .. (rows - 1)).each do |bi|
+        row = Array.new(cols, 0)
+        (0 .. (cols - 1)).each do |j|
+            row[j] = count
+            count += 1
+        end
+        mx[bi] = row
+    end
+    mx
+end
+
+def mmult(rows, cols, m1, m2)
+    m3 = Array.new(rows)
+    (0 .. (rows - 1)).each do |bi|
+        row = Array.new(cols, 0)
+        (0 .. (cols - 1)).each do |j|
+            val = 0
+            (0 .. (cols - 1)).each do |k|
+                val += m1.at(bi).at(k) * m2.at(k).at(j)
+            end
+            row[j] = val
+        end
+        m3[bi] = row
+    end
+    m3
+end
+
+def sqrt(x)
+    x = x.to_f
+    average = lambda { |a, b| (a+b)/2 }
+    impr    = lambda { |g| average[g, x/g] }
+    good_en = lambda { |g| (g*g - x).abs < 0.001 }
+    try     = lambda { |g| good_en[g] ? g : try[impr[g]] }
+    try[1.0]
+end
+
+
+Benchmark.bm(12) { |bmx|
+    bmx.report("times") { 3000000.times{|e| e + e } }
+    bmx.report("times") { 3000000.times{|e| e + e } }
+    bmx.report("fib") { fib 30 }
+    bmx.report("array") {
+        n = 2000
+        x = Array.new(n)
+        y = Array.new(n, 0)
+        n.times{|bi| x[bi] = bi + 1 }
+        (0 .. 999).each do |e|
+            (n-1).step(0,-1) do |bi|
+                y[bi] += x.at(bi)
+            end
+        end
+    }
+    bmx.report("matrix") {
+        n = 40
+        size = 30
+        m1 = mkmatrix(size, size)
+        m2 = mkmatrix(size, size)
+        mm = Array.new
+        n.times do
+            mm = mmult(size, size, m1, m2)
+        end
+    }
+    bmx.report("while") {
+        i = 3000000
+        while i > 0
+            break if i == 5
+            i -= 1
+            next if i % 100 == 0
+            redo if (i-=1) % 100 == 1
+        end
+    }
+    bmx.report("sqrt") {
+        i = 40000
+        while (i-=1) > 0
+            sqrt 2
+        end
+    }
+    bmx.report("3 [] alloc") {
+        i = 1000000
+        while (i-=1) > 0
+            [[], []]
+        end
+    }
+    bmx.report("const cache") {
+        1000000.times {
+            a = Array
+            a = Object::Array
+            a = ::Array
+        }
+    }
+    bmx.report("method calls") {
+        max = 2000
+        z = x = 0
+        while (x+=1) <= max
+            y = 0
+            while (y+=1) <= max
+                z = (x+y-z) % 32000
+            end
+        end
+    }
+    bmx.report("ivar") {
+        Class.new {
+            def run
+                @i = 4000000
+                while 0 < (@i = -1 + @i)
+                    a = @i
+                    b = @i
+                end
+            end
+        }.new.run
+    }
+}
+
+

data/benchmarks/concretize_test.rb

@@ -0,0 +1,26 @@
+require 'rubygems'
+require 'benchmark'
+require '../lib/concretizer'
+
+
+at_exit {
+  if false #$!
+    puts "==== "
+    puts $!.backtrace.join("\n")
+    puts "===="
+  end
+}
+
+
+class A
+ def go
+ end
+ def go2
+  go
+ end
+end
+
+a = A.new
+3.times {puts Benchmark.realtime { 4000000.times { a.go }} }
+puts Benchmark.realtime { Ruby2CExtension::Concretize.concretize_all! }
+3.times { puts Benchmark.realtime { 4000000.times { a.go }} }

data/benchmarks/extconf.rb

@@ -0,0 +1,10 @@
+require 'mkmf'
+
+# Give it a name
+extension_name = 'tak_source'
+
+# The destination
+dir_config(extension_name)
+
+# Do the work
+create_makefile(extension_name)

data/benchmarks/tak_rb.rb

@@ -0,0 +1,7 @@
+require './tak_source.rb'
+
+require 'benchmark'
+
+[7, 8, 9].each do |n|
+  print n, " took ", Benchmark.realtime {   tak(18, n, 0) }, "\n"
+end

data/benchmarks/tak_so.rb

@@ -0,0 +1,7 @@
+require './tak_source.so'
+
+require 'benchmark'
+
+[7, 8, 9].each do |n|
+  print n, " took ", Benchmark.realtime {   tak(18, n, 0) }, "\n"
+end

data/benchmarks/tak_source.rb

@@ -0,0 +1,16 @@
+public
+def tak(x, y, z)
+  unless (y < x)
+    z
+  else
+    tak(tak(x-1, y, z),
+         tak(y-1, z, x),
+         tak(z-1, x, y))
+  end
+end
+
+require 'benchmark'
+
+[7, 8, 9].each do |n|
+  print n, " took ", Benchmark.realtime {   tak(18, n, 0) }, "\n"
+end

data/bin/rb2cx

@@ -0,0 +1,162 @@
+#!/usr/bin/env ruby
+if $0 == __FILE__
+  require 'rubygems'  
+  $:.unshift '../lib' # in case you use it during development
+end
+
+require "ruby2cext/compiler"
+require "ruby2cext/version"
+require "getoptlong"
+require "logger"
+
+include Ruby2CExtension
+
+def usage(logger)
+	logger.warn(<<EOS.strip)
+Usage: rb2cx [options] file.rb ...
+
+Translates the given Ruby file into an equivalent C extension. The result is
+stored in file.c. It will then be compiled into a shared object file, unless
+the option --only-c is given.
+
+If multiple files are given, each file will be handled separately.
+
+=== General Options:
+
+-h / --help	 print this help
+-c / --only-c	 only translate to C
+-v / --verbose	 print status messages
+-w / --warnings	 print warnings for things that might not work as expected
+-V / --version	 print the Ruby2CExtension version
+
+=== Include Option:
+
+-I / --include path
+
+If a Ruby file "require"s another Ruby file and that file can be found in the
+given path, then it will be included in the C extension. This option can be
+used multiple times with different paths, the paths are then searched in the
+given order. Use --verbose to see which files were included.
+
+=== Optimization Options:
+
+-O / --optimization <optimization>
+
+Where <optimization> is one of the following:
+
+const_cache
+  enables local constant lookup caching
+
+builtin_methods
+  optimizes calls to many methods of builtin types
+
+inline_methods
+  inlines the methods nil?, equal? and __send__
+
+case_optimize
+  optimizes case statments with nil, true, false or Fixnums
+
+inline_builtin
+  inlines methods calls to builtin types
+
+cache_call
+  caches method lookup for each call
+  
+direct_self_call
+  directly calls private or recursive methods defined in same scope
+  
+all
+  enables all of the above optimizations
+
+=== Examples:
+
+rb2cx -wv file.rb
+rb2cx -I . -O all file.rb
+rb2cx -I . -I ../libs -O const_cache -O builtin_methods -w file.rb
+EOS
+end
+
+def main
+	opts = GetoptLong.new(
+		["--help",	   "-h", GetoptLong::NO_ARGUMENT],
+		["--only-c",	   "-c", GetoptLong::NO_ARGUMENT],
+		["--verbose",	   "-v", GetoptLong::NO_ARGUMENT],
+		["--version",	   "-V", GetoptLong::NO_ARGUMENT],
+		["--debug",		 GetoptLong::NO_ARGUMENT], # undocumented
+		["--warnings",	   "-w", GetoptLong::NO_ARGUMENT],
+		["--include",	   "-I", GetoptLong::REQUIRED_ARGUMENT],
+		["--optimization", "-O", GetoptLong::REQUIRED_ARGUMENT]
+	)
+
+	logger = Logger.new(STDERR)
+	logger.formatter = proc { |severity, time, progname, msg| "#{msg}\n" }
+	logger.level = Logger::INFO
+
+	only_c = false
+	include_paths = []
+	optimizations = {}
+	all_optimizations = false
+	plugins = {}
+	version_printed = false
+	begin
+		opts.each do |opt, arg|
+			case opt
+			when "--help"
+				usage(logger)
+				exit
+			when "--only-c"
+				only_c = true
+			when "--verbose"
+				logger.level = Logger::INFO
+			when "--version"
+				unless version_printed
+					logger.warn(Ruby2CExtension::FULL_VERSION_STRING)
+					version_printed = true
+				end
+			when "--debug"
+				logger.level = Logger::DEBUG
+			when "--warnings"
+				plugins[:warnings] = true
+			when "--include"
+				unless File.directory?(arg)
+					raise "'#{arg}' is no directory"
+				end
+				include_paths << arg
+			when "--optimization"
+				case (arg = arg.to_sym)
+				when :const_cache,
+				     :case_optimize,
+				     :inline_builtin,
+				     :inline_methods,
+				     :cache_call,
+				     :builtin_methods,
+				     :direct_self_call
+					optimizations[arg] = true
+				when :all
+					all_optimizations = true
+				else
+					raise "unknown optimization: #{arg}"
+				end
+			end
+		end
+		if ARGV.empty? && !version_printed
+			raise "No files given"
+		end
+	rescue => e
+		logger.error("#{e} ('rb2cx --help' for help)")
+		exit 1
+	end
+
+	plugins[:optimizations] = all_optimizations ? :all : optimizations
+
+	begin
+		ARGV.each { |fn|
+			Compiler.compile_file(fn, plugins, include_paths, only_c, logger)
+		}
+	rescue RuntimeError, SyntaxError => e
+		logger.error(e)
+		exit 1
+	end
+end
+
+main