posix-spawn 0.3.0

data/.gitignore

@@ -0,0 +1,5 @@
+Gemfile.lock
+ext/Makefile
+lib/posix_spawn_ext.*
+tmp
+pkg

data/COPYING

@@ -0,0 +1,28 @@
+Copyright (c) 2011 by Ryan Tomayko <r@tomayko.com>
+                  and Aman Gupta <aman@tmm1.net>
+
+Permission  is  hereby granted, free of charge, to any person ob-
+taining a copy of  this  software  and  associated  documentation
+files  (the "Software"), to deal in the Software without restric-
+tion, including without limitation the rights to use, copy, modi-
+fy, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is  fur-
+nished to do so, subject to the following conditions:
+
+The  above  copyright  notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF  ANY  KIND,
+EXPRESS  OR  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE  AND  NONIN-
+FRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER  IN  AN
+ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN  THE
+SOFTWARE.
+
+A small portion of the environ dup'ing code in ext/posix-spawn.c
+was taken from glibc <http://www.gnu.org/s/libc/> and is maybe
+Copyright (c) 2011 by The Free Software Foundation or maybe
+by others mentioned in the glibc LICENSES file. glibc is
+distributed under the terms of the LGPL license.

data/Gemfile

@@ -0,0 +1,2 @@
+source :rubygems
+gemspec

data/HACKING

@@ -0,0 +1,26 @@
+Clone the project:
+
+   git clone http://github.com/rtomayko/posix-spawn.git
+   cd posix-spawn
+   bundle install
+
+Rake tasks can be run without further setup:
+
+   rake build
+   rake test
+   rake benchmark
+
+Just `rake' builds the extension and runs the tests.
+
+If you want to run the benchmark scripts or tests directly out of a
+working copy, first setup your PATH and RUBYLIB environment:
+
+    PATH="$(pwd)/bin:$PATH"
+    RUBYLIB="$(pwd)/lib:$(pwd)/ext:$RUBYLIB"
+    export RUBYLIB
+
+Or, use the following rbdev script to quickly setup your PATH and
+RUBYLIB environment for this and other projects adhering to the
+Ruby Packaging Standard:
+
+    https://github.com/rtomayko/dotfiles/blob/rtomayko/bin/rbdev

data/README.md

@@ -0,0 +1,228 @@
+# posix-spawn
+
+`fork(2)` calls slow down as the parent process uses more memory due to the need
+to copy page tables. In many common uses of fork(), where it is followed by one
+of the exec family of functions to spawn child processes (`Kernel#system`,
+`IO::popen`, `Process::spawn`, etc.), it's possible to remove this overhead by using
+the use of special process spawning interfaces (`posix_spawn()`, `vfork()`, etc.)
+
+The posix-spawn library aims to implement a subset of the Ruby 1.9 `Process::spawn`
+interface in a way that takes advantage of fast process spawning interfaces when
+available and provides sane fallbacks on systems that do not.
+
+### FEATURES
+
+ - Fast, constant-time spawn times across a variety of platforms.
+ - A largish compatible subset of Ruby 1.9's `Process::spawn` interface and
+   enhanced versions of `Kernel#system`, <code>Kernel#`</code>, etc. under
+   Ruby >= 1.8.7 (currently MRI only).
+ - High level `POSIX::Spawn::Child` class for quick (but correct!)
+   non-streaming IPC scenarios.
+
+## BENCHMARKS
+
+The following benchmarks illustrate time needed to fork/exec a child process at
+increasing resident memory sizes on Linux 2.6 and MacOS X. Tests were run using
+the [`posix-spawn-benchmark`][pb] program included with the package.
+
+[pb]: https://github.com/rtomayko/posix-spawn/tree/master/bin
+
+### Linux
+
+![](https://chart.googleapis.com/chart?chbh=a,5,25&chxr=1,0,36,7&chd=t:5.77,10.37,15.72,18.31,19.73,25.13,26.70,29.31,31.44,35.49|0.86,0.82,1.06,0.99,0.79,1.06,0.84,0.79,0.93,0.94&chxs=1N**%20secs&chs=900x200&chds=0,36&chxl=0:|50%20MB|100%20MB|150%20MB|200%20MB|250%20MB|300%20MB|350%20MB|400%20MB|450%20MB|500%20MB&cht=bvg&chdl=fspawn%20%28fork%2Bexec%29|pspawn%20%28posix_spawn%29&chtt=posix-spawn-benchmark%20--graph%20--count%20500%20--mem-size%20500%20%28x86_64-linux%29&chco=1f77b4,ff7f0e&chf=bg,s,f8f8f8&chxt=x,y#.png)
+
+`posix_spawn` is faster than `fork+exec`, and executes in constant time when
+used with `POSIX_SPAWN_USEVFORK`.
+
+`fork+exec` is extremely slow for large parent processes.
+
+### OSX
+
+![](https://chart.googleapis.com/chart?chxl=0:|50%20MB|100%20MB|150%20MB|200%20MB|250%20MB|300%20MB|350%20MB|400%20MB|450%20MB|500%20MB&cht=bvg&chdl=fspawn%20%28fork%2Bexec%29|pspawn%20%28posix_spawn%29&chtt=posix-spawn-benchmark%20--graph%20--count%20500%20--mem-size%20500%20%28i686-darwin10.5.0%29&chco=1f77b4,ff7f0e&chf=bg,s,f8f8f8&chxt=x,y&chbh=a,5,25&chxr=1,0,3,0&chd=t:1.95,2.07,2.56,2.29,2.21,2.32,2.15,2.25,1.96,2.02|0.84,0.97,0.89,0.82,1.13,0.89,0.93,0.81,0.83,0.81&chxs=1N**%20secs&chs=900x200&chds=0,3#.png)
+
+`posix_spawn` is faster than `fork+exec`, but neither is affected by the size of
+the parent process.
+
+## USAGE
+
+This library includes two distinct interfaces: `POSIX::Spawn::spawn`, a lower
+level process spawning interface based on the new Ruby 1.9 `Process::spawn`
+method, and `POSIX::Spawn::Child`, a higher level class geared toward easy
+spawning of processes with simple string based standard input/output/error
+stream handling. The former is much more versatile, the latter requires much
+less code for certain common scenarios.
+
+### POSIX::Spawn::spawn
+
+The `POSIX::Spawn` module (with help from the accompanying C extension)
+implements a subset of the [Ruby 1.9 Process::spawn][ps] interface, largely
+through the use of the [IEEE Std 1003.1 `posix_spawn(2)` systems interfaces][po].
+These are widely supported by various UNIX operating systems.
+
+[ps]: http://www.ruby-doc.org/core-1.9/classes/Process.html#M002230
+[po]: http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html
+
+In its simplest form, the `POSIX::Spawn::spawn` method can be used to execute a
+child process similar to `Kernel#system`:
+
+    require 'posix/spawn'
+    pid  = POSIX::Spawn::spawn('echo', 'hello world')
+    stat = Process::waitpid(pid)
+
+The first line executes `echo` with a single argument and immediately returns
+the new process's `pid`. The second line waits for the process to complete and
+returns a `Process::Status` object. Note that `spawn` *does not* wait for the
+process to finish execution like `system` and does not reap the child's exit
+status -- you must call `Process::waitpid` (or equivalent) or the process will
+become a zombie.
+
+The `spawn` method is capable of performing a large number of additional
+operations, from setting up the new process's environment, to changing the
+child's working directory, to redirecting arbitrary file descriptors.
+
+See the Ruby 1.9 [`Process::spawn` documentation][ps] for details and the
+`STATUS` section below for a full account of the various `Process::spawn`
+features supported by `POSIX::Spawn::spawn`.
+
+### `system`, `popen4`, and <code>`</code>
+
+In addition to the `spawn` method, Ruby 1.9 compatible implementations of
+`Kernel#system` and <code>Kernel#\`</code> are provided in the `POSIX::Spawn`
+module. The `popen4` method can be used to spawn a process with redirected
+stdin, stdout, and stderr objects.
+
+### POSIX::Spawn as a Mixin
+
+The `POSIX::Spawn` module can also be mixed in to classes and modules to include
+`spawn` and all utility methods in that namespace:
+
+    require 'posix/spawn'
+
+    class YourGreatClass
+      include POSIX::Spawn
+
+      def speak(message)
+        pid = spawn('echo', message)
+        Process::waitpid(pid)
+      end
+
+      def calculate(expression)
+        pid, in, out, err = popen4('bc')
+        in.write(expression)
+        in.close
+        out.read
+      ensure
+        [in, out, err].each { |io| io.close if !io.closed? }
+        Process::waitpid(pid)
+      end
+    end
+
+### POSIX::Spawn::Child
+
+The `POSIX::Spawn::Child` class includes logic for executing child processes and
+reading/writing from their standard input, output, and error streams. It's
+designed to take all input in a single string and provides all output as single
+strings and is therefore not well-suited to streaming large quantities of data
+in and out of commands. That said, it has some benefits:
+
+ - **Simple** - requires little code for simple stream input and capture.
+ - **Internally non-blocking** (using `select(2)`) - handles all pipe hang cases
+   due to exceeding `PIPE_BUF` limits on one or more streams.
+ - **Potentially portable** - abstracts lower-level process and stream
+   management APIs so the class can be made to work on platforms like Java and
+   Windows where UNIX process spawning and stream APIs are not supported.
+
+`POSIX::Spawn::Child` takes the standard `spawn` arguments when instantiated,
+and runs the process to completion after writing all input and reading all
+output:
+
+    >> require 'posix/spawn'
+    >> child = POSIX::Spawn::Child.new('git', '--help')
+
+Retrieve process output written to stdout / stderr, or inspect the process's
+exit status:
+
+    >> child.out
+    => "usage: git [--version] [--exec-path[=GIT_EXEC_PATH]]\n ..."
+    >> child.err
+    => ""
+    >> child.status
+    => #<Process::Status: pid=80718,exited(0)>
+
+Use the `:input` option to write data on the new process's stdin immediately
+after spawning:
+
+    >> child = POSIX::Spawn::Child.new('bc', :input => '40 + 2')
+    >> child.out
+    "42\n"
+
+Additional options can be used to specify the maximum output size and time of
+execution before the child process is aborted. See the `POSIX::Spawn::Child`
+docs for more info.
+
+## STATUS
+
+The `POSIX::Spawn::spawn` method is designed to be as compatible with Ruby 1.9's
+`Process::spawn` as possible. Right now, it is a compatible subset.
+
+These `Process::spawn` arguments are currently supported to any of
+`Spawn::spawn`, `Spawn::system`, `Spawn::popen4`, and `Spawn::Child.new`:
+
+    env: hash
+      name => val : set the environment variable
+      name => nil : unset the environment variable
+    command...:
+      commandline                 : command line string which is passed to a shell
+      cmdname, arg1, ...          : command name and one or more arguments (no shell)
+      [cmdname, argv0], arg1, ... : command name, argv[0] and zero or more arguments (no shell)
+    options: hash
+      clearing environment variables:
+        :unsetenv_others => true   : clear environment variables except specified by env
+        :unsetenv_others => false  : don't clear (default)
+      redirection:
+        key:
+          FD              : single file descriptor in child process
+          [FD, FD, ...]   : multiple file descriptor in child process
+        value:
+          FD                        : redirect to the file descriptor in parent process
+          :close                    : close the file descriptor in child process
+          string                    : redirect to file with open(string, "r" or "w")
+          [string]                  : redirect to file with open(string, File::RDONLY)
+          [string, open_mode]       : redirect to file with open(string, open_mode, 0644)
+          [string, open_mode, perm] : redirect to file with open(string, open_mode, perm)
+        FD is one of follows
+          :in     : the file descriptor 0 which is the standard input
+          :out    : the file descriptor 1 which is the standard output
+          :err    : the file descriptor 2 which is the standard error
+          integer : the file descriptor of specified the integer
+          io      : the file descriptor specified as io.fileno
+      current directory:
+        :chdir => str
+
+These options are currently NOT supported:
+
+    options: hash
+      process group:
+        :pgroup => true or 0 : make a new process group
+        :pgroup => pgid      : join to specified process group
+        :pgroup => nil       : don't change the process group (default)
+      resource limit: resourcename is core, cpu, data, etc.  See Process.setrlimit.
+        :rlimit_resourcename => limit
+        :rlimit_resourcename => [cur_limit, max_limit]
+      umask:
+        :umask => int
+      redirection:
+        value:
+          [:child, FD]              : redirect to the redirected file descriptor
+      file descriptor inheritance: close non-redirected non-standard fds (3, 4, 5, ...) or not
+        :close_others => false : inherit fds (default for system and exec)
+        :close_others => true  : don't inherit (default for spawn and IO.popen)
+
+## ACKNOWLEDGEMENTS
+
+Copyright (c) by
+[Ryan Tomayko](http://tomayko.com/about)
+and
+[Aman Gupta](https://github.com/tmm1).
+
+See the `COPYING` file for more information on license and redistribution.

data/Rakefile

@@ -0,0 +1,36 @@
+task :default => :test
+
+# ==========================================================
+# Packaging
+# ==========================================================
+
+GEMSPEC = eval(File.read('posix-spawn.gemspec'))
+
+require 'rake/gempackagetask'
+Rake::GemPackageTask.new(GEMSPEC) do |pkg|
+end
+
+# ==========================================================
+# Ruby Extension
+# ==========================================================
+
+require 'rake/extensiontask'
+Rake::ExtensionTask.new('posix_spawn_ext', GEMSPEC) do |ext|
+  ext.ext_dir = 'ext'
+end
+task :build => :compile
+
+# ==========================================================
+# Testing
+# ==========================================================
+
+require 'rake/testtask'
+Rake::TestTask.new 'test' do |t|
+  t.test_files = FileList['test/test_*.rb']
+end
+task :test => :build
+
+desc 'Run some benchmarks'
+task :benchmark => :build do
+  ruby '-Ilib', 'bin/posix-spawn-benchmark'
+end

data/TODO

@@ -0,0 +1,23 @@
+[x] license (LGPL)
+[x] fucking name this thing
+[x] fastspawn-bm should take iterations and memsize arguments
+[x] high level Grit::Process like class (string based) (tmm1)
+[x] add FD => '/path/to/file' (all variations) (rtomayko)
+[x] raise exception on unhandled options
+[x] benchmarks in README (tmm1)
+[x] POSIX::Spawn::spawn usage examples in README
+[x] POSIX::Spawn#pspawn should be just #spawn
+[x] :err => :out case -- currently closing out after dup2'ing
+[x] POSIX::Spawn::Process.new should have same method signature as Process::spawn
+[x] POSIX::Spawn::Process renamed to POSIX::Spawn::Child
+[x] Better POSIX::Spawn#spawn comment docs
+[x] POSIX::Spawn::Child usage examples in README
+
+
+[ ] popen* interfaces
+[x] system interface
+[x] ` interface
+[ ] jruby Grit::Process stuff
+[ ] make :vfork an option to Spawn#spawn
+[ ] check all posix_spawn_* function call return values
+[ ] POSIX::Spawn as ::Spawn? (maybe, we'll see)

data/bin/posix-spawn-benchmark

@@ -0,0 +1,117 @@
+#!/usr/bin/env ruby
+#/ Usage: posix-spawn-benchmark [-n <count>] [-m <mem-size>]
+#/ Run posix-spawn (Ruby extension) benchmarks and report to standard output.
+#/
+#/ Options:
+#/   -n, --count=NUM         total number of processes to spawn.
+#/   -m, --mem-size=MB       RES size to bloat to before performing benchmarks.
+#/   -g, --graph             benchmark at 10MB itervals up to RES and graph results.
+#/
+#/ Benchmarks run with -n 1000 -m 100 by default.
+require 'optparse'
+require 'posix-spawn'
+require 'benchmark'
+include Benchmark
+
+allocate   = 100 * (1024 ** 2)
+iterations = 1_000
+graphmode  = false
+ARGV.options do |o|
+  o.set_summary_indent('  ')
+  o.on("-n", "--count=num")   { |val| iterations = val.to_i }
+  o.on("-m", "--mem-size=MB") { |val| allocate   = val.to_i * (1024 ** 2) }
+  o.on("-g", "--graph")       { graphmode = true }
+  o.on_tail("-h", "--help")   { exec "grep ^#/ <'#{__FILE__}' |cut -c4-" }
+  o.parse!
+end
+
+if graphmode
+  bloat = []
+  data  = {}
+  chunk = allocate / 10
+  max   = 0
+
+  10.times do
+    puts "allocating #{chunk / (1024 ** 2)}MB (#{(bloat.size+1) * chunk / (1024 ** 2)}MB total)"
+    bloat << ('x' * chunk)
+    # size = bloat.size / (1024 ** 2)
+
+    %w[ fspawn pspawn ].each do |type|
+      print " - benchmarking #{type}... "
+      time = Benchmark.realtime do
+        iterations.times do
+          pid = POSIX::Spawn.send(type, 'true')
+          Process.wait(pid)
+        end
+      end
+      puts "done (#{time})"
+
+      data[type] ||= []
+      data[type] << time
+      max = time if time > max
+    end
+  end
+
+  max = max < 0.5 ? (max * 10).round / 10.0 : max.ceil
+  minmb, maxmb = chunk/(1024**2), allocate/(1024**2)
+  series = %w[ fspawn pspawn ].map{|name| data[name].map{|d| "%.2f" % d }.join(',') }
+
+  chart = {
+    :chs  => '900x200',
+    :cht  => 'bvg', # grouped vertical bar chart
+    :chtt => "posix-spawn-benchmark --graph --count #{iterations} --mem-size #{maxmb} (#{RUBY_PLATFORM})",
+
+    :chf  => 'bg,s,f8f8f8', # background
+    :chbh => 'a,5,25', # 25px between bar groups
+
+    :chd  => "t:#{series.join('|')}", # data
+    :chds => "0,#{max}", # scale
+    :chdl => 'fspawn (fork+exec)|pspawn (posix_spawn)', # legend
+    :chco => '1f77b4,ff7f0e', # colors
+
+    :chxt => 'x,y',
+    :chxr => "1,0,#{max},#{max/5}", # y labels up to max time
+    :chxs => '1N** secs', # y labels are +=' secs'
+    :chxl => "0:|#{minmb.step(maxmb, maxmb/10).map{ |mb| "#{mb} MB"}.join('|')}", # x bucket labels
+  }
+
+  url  = "https://chart.googleapis.com/chart?"
+  url += chart.map do |key, val|
+    "#{key}=#{val.gsub(' ','%20').gsub('(','%28').gsub(')','%29').gsub('+','%2B')}"
+  end.join('&')
+  url += '#.png'
+
+  puts url
+
+  exit!
+end
+
+puts "benchmarking fork/exec vs. posix_spawn over #{iterations} runs" +
+     " at #{allocate / (1024 ** 2)}M res"
+
+# bloat the process
+bloat = 'x' * allocate
+
+# run the benchmarks
+bm 40 do |x|
+  x.report("fspawn (fork/exec):") do
+    iterations.times do
+      pid = POSIX::Spawn.fspawn('true')
+      Process.wait(pid)
+    end
+  end
+  x.report("pspawn (posix_spawn):") do
+    iterations.times do
+      pid = POSIX::Spawn.pspawn('true')
+      Process.wait(pid)
+    end
+  end
+  if Process.respond_to?(:spawn)
+    x.report("spawn (native):") do
+      iterations.times do
+        pid = Process.spawn('true')
+        Process.wait(pid)
+      end
+    end
+  end
+end