ruby2cext 0.2.0

data/Changelog

@@ -0,0 +1,27 @@
+Changelog
+
+-- 0.2.0
+
+* New features:
+  * New plugin architecture for optimizations and other things
+  * Optimization plugins: const_cache, builtin_methods, inline_methods,
+    case_optimize
+  * Other plugins: require_include, warnings
+  * Eval2C for compilation at runtime
+  * Available as gem now
+
+* Improvements:
+  * Implemented control flow "through" rescue/ensure clauses
+  * Better vmode handling
+  * Support for Ruby 1.8.5 and 1.8.6
+  * Improved performance for global variables
+
+* Fixes:
+  * Workaround for an issue with ALLOCA_N in while loops
+  * Fixed a problem with case/when (missing newline nodes)
+  * Removed libraries from compile command
+  * Various other small fixes
+
+-- 0.1.0
+
+* Initial release

data/README

@@ -0,0 +1,44 @@
+
+Ruby2CExtension
+===============
+
+Ruby2CExtension is a Ruby to C extension translator/compiler. It takes any Ruby
+source file, parses it using Ruby's builtin parser and then translates the
+abstract syntax tree into "equivalent" C extension code.
+
+
+Requirements
+------------
+
+* Ruby 1.8.4, 1.8.5 or 1.8.6
+* RubyNode (available at http://rubynode.rubyforge.org/)
+
+
+Installation
+------------
+
+Just run (as root):
+
+  gem install ruby2cext
+
+Or if you do not use the gem:
+
+  ruby setup.rb
+
+This will install Ruby2CExtension to the default locations. Optionally you can
+supply some options to setup.rb to customize the installation (see "ruby
+setup.rb --help").
+
+
+Documentation
+-------------
+
+Please see doc/index.html for more documentation.
+
+
+License
+-------
+
+Copyright 2006-2007 Dominik Bathon <dbatml@gmx.de>
+
+Ruby2CExtension is licensed under the same terms as Ruby.

data/bin/rb2cx

@@ -0,0 +1,178 @@
+#!/usr/bin/env ruby
+
+require "ruby2cext/compiler"
+require "ruby2cext/version"
+require "getoptlong"
+require "logger"
+
+include Ruby2CExtension
+
+def usage(logger)
+	logger.warn(<<EOS.strip)
+Usage: rb2cx [options] file.rb ...
+
+Translates the given Ruby file into an equivalent C extension. The result is
+stored in file.c. It will then be compiled into a shared object file, unless
+the option --only-c is given.
+
+If multiple files are given, each file will be handled separately.
+
+=== General Options:
+
+-h / --help      print this help
+-c / --only-c    only translate to C
+-v / --verbose   print status messages
+-w / --warnings  print warnings for things that might not work as expected
+-V / --version   print the Ruby2CExtension version
+
+=== Include Option:
+
+-I / --include path
+
+If a Ruby file "require"s another Ruby file and that file can be found in the
+given path, then it will be included in the C extension. This option can be
+used multiple times with different paths, the paths are then searched in the
+given order. Use --verbose to see which files were included.
+
+=== Optimization Options:
+
+-O / --optimization <optimization>
+
+Where <optimization> is one of the following:
+
+const_cache
+  enables local constant lookup caching
+
+builtin_methods
+  optimizes calls to many methods of builtin types
+
+inline_methods
+  inlines the methods nil?, equal? and __send__
+
+case_optimize
+  optimizes case statments with nil, true, false or Fixnums
+
+all
+  enables all of the above optimizations
+
+=== Examples:
+
+rb2cx -wv file.rb
+rb2cx -I . -O all file.rb
+rb2cx -I . -I ../libs -O const_cache -O builtin_methods -w file.rb
+EOS
+end
+
+def compile_file(file_name, plugins, include_paths, only_c, logger)
+	bn = File.basename(file_name)
+	unless bn =~ /\A(.*)\.rb\w?\z/
+		raise "#{file_name} is no ruby file"
+	end
+	name = $1;
+	unless name =~ /\A\w+\z/
+		raise "'#{name}' is not a valid extension name"
+	end
+	file_name = File.join(File.dirname(file_name), bn)
+
+	logger.info("reading #{file_name}")
+	source_str = IO.read(file_name)
+
+	logger.info("translating #{file_name} to C")
+	c = Compiler.new(name, logger)
+	unless include_paths.empty?
+		plugins = plugins.merge({:require_include => [include_paths, [file_name]]})
+	end
+	logger.debug("plugins = #{plugins.inspect}")
+	c.add_plugins(plugins)
+	logger.debug("plugins used: #{c.plugins.map { |pi| pi.class }.inspect}")
+	c.add_rb_file(source_str, file_name)
+	c_code = c.to_c_code
+
+	c_file_name = File.join(File.dirname(file_name), "#{name}.c")
+	logger.info("writing #{c_file_name}")
+	File.open(c_file_name, "w") { |f| f.puts(c_code) }
+
+	unless only_c
+		logger.info("compiling #{c_file_name}")
+		Compiler.compile_c_file_to_dllib(c_file_name, logger)
+	end
+end
+
+def main
+	opts = GetoptLong.new(
+		["--help",         "-h", GetoptLong::NO_ARGUMENT],
+		["--only-c",       "-c", GetoptLong::NO_ARGUMENT],
+		["--verbose",      "-v", GetoptLong::NO_ARGUMENT],
+		["--version",      "-V", GetoptLong::NO_ARGUMENT],
+		["--debug",              GetoptLong::NO_ARGUMENT], # undocumented
+		["--warnings",     "-w", GetoptLong::NO_ARGUMENT],
+		["--include",      "-I", GetoptLong::REQUIRED_ARGUMENT],
+		["--optimization", "-O", GetoptLong::REQUIRED_ARGUMENT]
+	)
+
+	logger = Logger.new(STDERR)
+	logger.formatter = proc { |severity, time, progname, msg| "#{msg}\n" }
+	logger.level = Logger::WARN
+
+	only_c = false
+	include_paths = []
+	optimizations = {}
+	all_optimizations = false
+	plugins = {}
+	version_printed = false
+	begin
+		opts.each do |opt, arg|
+			case opt
+			when "--help"
+				usage(logger)
+				exit
+			when "--only-c"
+				only_c = true
+			when "--verbose"
+				logger.level = Logger::INFO
+			when "--version"
+				unless version_printed
+					logger.warn(Ruby2CExtension::FULL_VERSION_STRING)
+					version_printed = true
+				end
+			when "--debug"
+				logger.level = Logger::DEBUG
+			when "--warnings"
+				plugins[:warnings] = true
+			when "--include"
+				unless File.directory?(arg)
+					raise "'#{arg}' is no directory"
+				end
+				include_paths << arg
+			when "--optimization"
+				case (arg = arg.to_sym)
+				when :const_cache, :case_optimize, :builtin_methods, :inline_methods
+					optimizations[arg] = true
+				when :all
+					all_optimizations = true
+				else
+					raise "unknown optimization: #{arg}"
+				end
+			end
+		end
+		if ARGV.empty? && !version_printed
+			raise "No files given"
+		end
+	rescue => e
+		logger.error("#{e} ('rb2cx --help' for help)")
+		exit 1
+	end
+
+	plugins[:optimizations] = all_optimizations ? :all : optimizations
+
+	begin
+		ARGV.each { |fn|
+			compile_file(fn, plugins, include_paths, only_c, logger)
+		}
+	rescue RuntimeError, SyntaxError => e
+		logger.error(e)
+		exit 1
+	end
+end
+
+main

data/doc/eval2c.html

@@ -0,0 +1,281 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
+<html>
+<head>
+	<title>Eval2C</title>
+	<link href="style.css" media="all" rel="Stylesheet" type="text/css">
+</head>
+<body>
+<h1>Eval2C</h1>
+
+
+	<p>Eval2C is a class that allows the compilation of Ruby code to a C extension at
+runtime. The compiled C extension will be <code>require</code>d automatically
+and the compiled code will be available as a Proc.</p>
+
+
+	<p>It is easy to integrate Eval2C into existing scripts or libraries to improve
+performance. It is also pretty simple to provide fallback code for when
+Ruby2CExtension is not available.</p>
+
+
+	<p>Sections: <a href="#section1">Basic Usage</a>, <a href="#section2">Options</a>, <a href="#section3">Implementation Details</a>, <a href="#section4">Using Eval2C Only When It Is Available</a>.</p>
+
+
+	<h2 id="section1">Basic Usage</h2>
+
+
+	<p>Here is an example:</p>
+
+
+<pre><code>require "ruby2cext/eval2c" 
+
+$e2c = Ruby2CExtension::Eval2C.new
+
+$e2c.toplevel_eval("puts 'hello'") # prints hello
+</code></pre>
+
+	<p>First you need to create an Eval2C instance, this instance basically stores
+the options/configuration. In the above example no options were given, so the
+defaults are used. The available options are explained below.</p>
+
+
+	<p>The last line of the example does what the method name suggests: the string of
+Ruby code is evaluated at the toplevel (i.e. not inside any class scope). To
+be more precise, that last line is equivalent to this Ruby code:</p>
+
+
+<pre><code>$some_global_var = proc { puts 'hello' }
+$some_global_var.call
+</code></pre>
+
+The proc is compiled to a C extension, that C expension is then
+<code>require</code>d and finally the proc is called.
+
+	<p>The implementation of <code>toplevel_eval</code> is actually pretty simple:</p>
+
+
+<pre><code>def toplevel_eval(code_str)
+  compile_to_proc(code_str).call
+end
+</code></pre>
+
+	<p>So the main work is done by <code>compile_to_proc</code>, which can also be used
+directly:</p>
+
+
+<pre><code>$e2c.compile_to_proc("|a,b| a+b").call(2, 3) # =&gt; 5
+</code></pre>
+
+	<p>There are some things to note: The proc won&#8217;t have access to local variables
+and it will be defined at toplevel, which is important for constant lookup:</p>
+
+
+<pre><code>SOME_CONST = :toplevel
+
+class A
+  SOME_CONST = :A
+  $e2c.compile_to_proc("SOME_CONST").call # =&gt; :toplevel, not :A!
+end
+</code></pre>
+
+	<p>Eval2C also offers equivalents to Ruby&#8217;s <code>module_eval</code>/<code>class_eval</code> and
+<code>instance_eval</code>:</p>
+
+
+<pre><code>class A
+  $e2c.module_eval(self, %{
+    def initialize(x)
+      @x = x
+    end
+    def foo(y)
+      @x + y
+    end
+  })
+end
+
+A.new(5).foo(6) # =&gt; 11
+
+$e2c.instance_eval("4321", "reverse") # =&gt; "1234" 
+</code></pre>
+
+	<p>Their implementations also use <code>compile_to_proc</code> and they are very similar to
+the implementation of <code>toplevel_eval</code>:</p>
+
+
+<pre><code>def module_eval(mod, code_str)
+  mod.module_eval(&#38;compile_to_proc(code_str))
+end
+alias :class_eval :module_eval
+
+def instance_eval(object, code_str)
+  object.instance_eval(&#38;compile_to_proc(code_str))
+end
+</code></pre>
+
+	<p>With <code>module_eval</code>/<code>class_eval</code> it is possible to selectively compile some
+performance critical methods to C and leave the rest in Ruby.</p>
+
+
+	<p>But there is one more thing: <code>compile_methods</code>.</p>
+
+
+	<p>Defining the methods using <code>module_eval</code> as in the last example, has at least
+one downside: it won&#8217;t play nice with syntax-highlighting text editors,
+because the code of the methods is actually in a string. It would be much
+nicer to <code>def</code> the methods the normal way and then just compile them
+afterwards.</p>
+
+
+	<p>Thanks to <code>compile_methods</code> this is possible:</p>
+
+
+<pre><code>class A
+  def initialize(x)
+    @x = x
+  end
+  def foo(y)
+    @x + y
+  end
+
+  $e2c.compile_methods(self, :initialize, :foo)
+end
+
+A.new(5).foo(6) # =&gt; 11
+</code></pre>
+
+	<p><code>compile_methods</code> will grab the methods&#8217; node trees using RubyNode, compile
+them to C and then replace the old methods with the C versions. The visibility
+of the old methods will be preserved.</p>
+
+
+	<p>There are some limitations: <code>compile_methods</code> only works with methods defined
+using <code>def</code> and the methods must not need a cref (for more informations on
+what crefs are please see the respective section in
+<a href="limitations.html">limitations</a>). This mainly means that constants must be
+prefixed with a double colon: <code>Array</code> needs a cref, while <code>::Array</code> does not.</p>
+
+
+	<h2 id="section2">Options</h2>
+
+
+	<p>Eval2C is configured with an option hash, the available options are:</p>
+
+
+	<ul>
+	<li><code>:path</code>: this is the path where the compiled extensions are stored, the
+  default is the directory <code>.ruby2cext</code> in the current users home dir.</li>
+		<li><code>:prefix</code>: this is the prefix for the names of the generated extensions, the
+  names of the extensions are generated from this prefix and a <span class="caps">SHA1</span> digest of
+  the code and options. The default is <code>"eval2c"</code> and it is generally not
+  necessary to change it.</li>
+		<li><code>:plugins</code>: this is used to configure the options for the compilation
+  (compare <a href="rb2cx.html">rb2cx</a>) and it is another option hash. The default is
+  <code>{:optimizations =&gt; :all}</code>, but all of the following keys are
+  possible:
+	<ul>
+	<li><code>:warnings</code>: if set to <code>true</code>, then warnings about possible problems will
+   be sent to the logger (see below).</li>
+		<li><code>:require_include</code>: can be set to a single search path or an array of
+   search paths (equivalent to the <code>-I</code> option of <code>rb2cx</code>).</li>
+		<li><code>:optimizations</code>: yet another option hash to configure the
+   <a href="optimizations.html">optimizations</a>. The valid keys are <code>:const_cache</code>,
+   <code>:builtin_methods</code>, <code>:inline_methods</code> and <code>:case_optimize</code>, each with
+   <code>true</code> or <code>false</code> as value. To just use all optimizations it is also
+   possible to just use the symbol <code>:all</code> instead of the hash.</li>
+	</ul>
+	</li>
+		<li><code>:logger</code>: used for log messages, can be either <code>nil</code> or an instance of
+  <code>Logger</code> (<code>require "logger"</code>). The default is <code>nil</code>, which means
+  that no output is generated.</li>
+		<li><code>:force_recompile</code>: if this is set to <code>true</code>, then each generated extension
+  is (re)compiled, even if it is already available from a previous run. The
+  default is <code>false</code>.</li>
+	</ul>
+
+
+	<p>Some examples:</p>
+
+
+<pre><code>Eval2C.new(:path =&gt; ".", :plugins =&gt; {})
+Eval2C.new(:plugins =&gt; {:require_include=&gt;[".", "../lib"], :optimizations=&gt;:all})
+Eval2C.new(:plugins =&gt; {:warnings=&gt;true, :optimizations=&gt;{:builtin_methods=&gt;true}})
+Eval2C.new(:prefix =&gt; "my_lib_name", :logger =&gt; Logger.new(STDOUT))
+</code></pre>
+
+	<h2 id="section3">Implementation Details</h2>
+
+
+	<p>The <code>compile_to_proc</code> method works as follows:</p>
+
+
+	<ol>
+	<li>The name of the extension is determined from the given prefix combined with
+  the <span class="caps">SHA1</span> digest of the Ruby code and some extra data (Ruby version,
+  Ruby2CExtension version and plugin options). So the code determines the
+  extension name and if the same code is compiled with the same version and
+  settings, the name will be the same. This allows very simple and efficient
+  caching of the compiled extensions.</li>
+		<li>From the extension name a global variable name is constructed. The compiled
+  proc will be stored in that global variable by the generated C extension.</li>
+		<li>If the extension with the generated name already exists, then we are
+  basically done and can skip this step. Otherwise the Ruby proc code is
+  translated to C and compiled to a C extension in the specified path.</li>
+		<li>The generated or already existing C extension is <code>require</code>d.</li>
+		<li>The global variable that now contains the compiled proc is read and the
+  result is returned.</li>
+	</ol>
+
+
+	<p>The <code>compile_methods</code> method works similar.</p>
+
+
+	<p>As mentioned above, this scheme allows very efficient caching of the C
+extensions. Basically each extension is only compiled once when the
+program/library is executed for the first time. In later runs the extension
+will already exist and just be <code>require</code>d.</p>
+
+
+	<p>Adding the Ruby version and the Ruby2CExtension version to the digest avoids
+problems if multiple versions of Ruby or Ruby2CExtension are installed and
+used in parallel and it also guarantees that the extensions are automatically
+recompiled if Ruby or Ruby2CExtension are updated.</p>
+
+
+	<h2 id="section4">Using Eval2C Only When It Is Available</h2>
+
+
+	<p>Having Ruby2CExtension as a hard dependency just for some speedup might not
+be acceptable for many projects, in particular since Ruby2CExtension requires
+that a C compiler is available at runtime.</p>
+
+
+	<p>An alternative that avoids the hard dependency is to use Ruby2CExtension only
+if it is installed anyway and otherwise just execute the Ruby code as usual.
+This works works very well if Eval2C and <code>compile_methods</code> is used, for
+example:</p>
+
+
+<pre><code>begin
+  require "ruby2cext/eval2c" 
+  $e2c = Ruby2CExtension::Eval2C.new
+rescue LoadError
+  $e2c = nil
+end
+
+class A
+  def initialize(x)
+    @x = x
+  end
+  def foo(y)
+    @x + y
+  end
+
+  $e2c &#38;&#38; $e2c.compile_methods(self, :initialize, :foo)
+end
+</code></pre>
+
+	<p>So if Ruby2CExtension is available, then the methods will be compiled and
+fast, otherwise the Ruby code will just work as usual (but it might be a bit
+slower of course).</p>
+</body>
+</html>