tlconnor-spawn 1.0.0

data/.gitignore

@@ -0,0 +1,2 @@
+*~
+*.gem

data/CHANGELOG

@@ -0,0 +1,55 @@
+v0.1 - 2007/09/13
+
+initial version
+
+--------------------------------------------------
+v0.2 - 2007/09/28
+
+* return PID of the child process
+* added ":detach => false" option
+
+--------------------------------------------------
+v0.3 - 2007/10/15
+
+* added ':method => :thread' for threaded spawns
+* removed ':detach => false' option in favor of more generic implementation
+* added ability to set configuration of the form 'Spawn::method :thread'
+* added patch to ActiveRecord::Base to allow for more efficient reconnect in child processes
+* added monkey patch for http://dev.rubyonrails.org/ticket/7579
+* added wait() method to wait for spawned code blocks
+* don't allow threading if allow_concurrency=false
+
+--------------------------------------------------
+v0.4 - 2008/1/26
+
+* default to :thread on windows, still :fork on all other platforms
+* raise exception when used with :method=>:true and allow_concurrency != true
+
+--------------------------------------------------
+v0.5 - 2008/3/1
+* also default to :thread on JRuby (java)
+* added new :method => :yield which doesn't fork or thread, this is useful for testing
+* fixed problem with connections piling up on PostgreSQL
+
+--------------------------------------------------
+v0.6 - 2008/04/21
+* only apply clear_reloadable_connections patch on Rails 1.x (7579 fixed in Rails 2.x)
+* made it more responsive in more environments by disconnecting from the listener socket in the forked process
+
+--------------------------------------------------
+v0.7 - 2008/04/24
+* more generic mechanism for closing resources after fork
+* check for existence of Mongrel before patching it
+
+--------------------------------------------------
+v0.8 - 2008/05/02
+* call exit! within the ensure block so that at_exit handlers aren't called on exceptions
+* set logger from RAILS_DEFAULT_LOGGER if available, else STDERR
+
+--------------------------------------------------
+v0.9 - 2008/05/11
+* added ability to set nice level for child process
+
+--------------------------------------------------
+v1.0 - 2010/10/09
+* merged edged to master, let's call this version 1.0

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2007 Tom Anderson (tom@squeat.com)
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,193 @@
+# Spawn
+
+This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
+code so that your application can return results to your users more quickly.
+This plugin works by creating new database connections in ActiveRecord::Base for the
+spawned block.
+
+The plugin also patches ActiveRecord::Base to handle some known bugs when using 
+threads (see lib/patches.rb).
+
+## Installation
+
+To install the plugin from the master branch (recommended).
+
+    script/plugin install git://github.com/tra/spawn.git
+
+If you want to install the plugin from the 'edge' branch (latest development):
+
+    script/plugin install git://github.com/tra/spawn.git -r edge
+
+If you are unfortunate enough to be stuck on Rails 1.x, then it is recommended you
+stick with v1.0 of this plugin (Rails 1.x won't be supported in future versions but
+it might still work if you're lucky).   To install this version:
+
+    script/plugin install git://github.com/tra/spawn.git -r master:v1.0
+
+## Usage
+
+Here's a simple example of how to demonstrate the spawn plugin.
+In one of your controllers, insert this code (after installing the plugin of course):
+
+    spawn do
+       logger.info("I feel sleepy...")
+       sleep 11
+       logger.info("Time to wake up!")
+     end
+
+If everything is working correctly, your controller should finish quickly then you'll see
+the last log message several seconds later.
+
+If you need to wait for the spawned processes/threads, then pass the objects returned by
+spawn to Spawn::wait(), like this:
+
+    N.times do |i|
+      # spawn N blocks of code
+      spawn_ids[i] = spawn do
+        something(i)
+      end
+    end
+    # wait for all N blocks of code to finish running
+    wait(spawn_ids)
+
+## Options
+
+The options you can pass to spawn are:
+
+<table>
+  <tr><th>Option</th><th>Values</th></tr>
+  <tr><td>:method</td><td>:fork, :thread, :yield</td></tr>
+  <tr><td>:nice</td><td>integer value 0-19, 19 = really nice</td></tr>
+  <tr><td>:kill</td><td>boolean value indicating whether the parent should kill the spawned process
+   when it exits (only valid when :method => :fork)</td></tr>
+  <tr><td>:argv</td><td>string to override the process name</td></tr>
+</table>
+
+Any option to spawn can be set as a default so that you don't have to pass them in
+to every call of spawn.   To configure the spawn default options, add a line to
+your configuration file(s) like this: 
+
+    Spawn::default_options {:method => :thread}
+
+If you don't set any default options, the :method will default to :fork.  To
+specify different values for different environments, add the default_options call to
+he appropriate environment file (development.rb, test.rb).   For testing you can set
+the default :method to :yield so that the code is run inline.
+
+    # in environment.rb
+    Spawn::method :method => :fork, :nice => 7
+    # in test.rb, will override the environment.rb setting
+    Spawn::method :method => :yield
+
+This allows you to set your production and development environments to use different
+methods according to your needs.
+
+### be nice
+
+If you want your forked child to run at a lower priority than the parent process, pass in
+the :nice option like this:
+
+    spawn(:nice => 7) do
+      do_something_nicely
+    end
+
+### fork me
+
+By default, spawn will use the fork to spawn child processes.  You can configure it to
+do threading either by telling the spawn method when you call it or by configuring your
+environment.
+For example, this is how you can tell spawn to use threading on the call,
+
+    spawn(:method => :thread) do
+      something
+    end
+  
+For older versions of Rails (1.x), when using the :thread setting, spawn will check to
+make sure that you have set allow_concurrency=true in your configuration.   If you
+want this setting then put this line in one of your environment config files: 
+
+    config.active_record.allow_concurrency = true
+
+If it is not set, then spawn will raise an exception.
+
+### kill or be killed
+
+Depending on your application, you may want the children processes to go away when
+the parent  process exits.   By default spawn lets the children live after the
+parent dies.   But you can tell it to kill the children by setting the :kill option
+to true.
+
+### a process by any other name
+
+If you'd like to be able to identify which processes are spawned by looking at the
+output of ps then set the :argv option with a string of your choice.
+You should then be able to see this string as the process name when
+listing the running processes (ps).
+
+For example, if you do something like this,
+
+    3.times do |i|
+      spawn(:argv => "spawn -#{i}-") do
+        something(i)
+      end
+    end
+
+then in the shell,
+
+    $ ps -ef | grep spawn
+    502  2645  2642   0   0:00.01 ttys002    0:00.02 spawn -0-
+    502  2646  2642   0   0:00.02 ttys002    0:00.02 spawn -1-
+    502  2647  2642   0   0:00.02 ttys002    0:00.03 spawn -2-
+
+The length of the process name may be limited by your OS so you might want to experiment
+to see how long it can be (it may be limited by the length of the original process name).
+
+## Forking vs. Threading
+
+There are several tradeoffs for using threading vs. forking.   Forking was chosen as the
+default primarily because it requires no configuration to get it working out of the box.
+
+Forking advantages:
+
+- more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
+  Even though spawn attempts to patch known problems with the threaded implementation,
+  there are no guarantees.  Forking is heavier but should be fairly reliable.
+- keep running - this could also be a disadvantage, but you may find you want to fork
+  off a process that could have a life longer than its parent.  For example, maybe you
+  want to restart your server without killing the spawned processes.
+  We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
+- easier - forking works out of the box with spawn, threading requires you set
+  allow_concurrency=true (for older versions of Rails).
+  Also, beware of automatic reloading of classes in development
+  mode (config.cache_classes = false).
+
+Threading advantages:
+- less filling - threads take less resources... how much less?  it depends.   Some 
+  flavors of Unix are pretty efficient at forking so the threading advantage may not
+  be as big as you think... but then again, maybe it's more than you think.  ;-)
+- debugging - you can set breakpoints in your threads
+
+## Acknowledgements
+
+This plugin was initially inspired by Scott Persinger's blog post on how to use fork
+in rails for background processing.
+    http://geekblog.vodpod.com/?p=26
+
+Further inspiration for the threading implementation came from Jonathon Rochkind's
+blog post on threading in rails.
+    http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
+    
+Also thanks to all who have helped debug problems and suggest improvements
+including:
+
+-  Ahmed Adam, Tristan Schneiter, Scott Haug, Andrew Garfield, Eugene Otto, Dan Sharp,
+  Olivier Ruffin, Adrian Duyzer, Cyrille Labesse
+
+-  Garry Tan, Matt Jankowski (Rails 2.2.x fixes), Mina Naguib (Rails 2.3.6 fix)
+
+-  Tim Kadom, Mauricio Marcon Zaffari, Danial Pearce, Hongli Lai, Scott Wadden
+  (passenger fixes)
+
+-  &lt;your name here&gt;
+
+Copyright (c) 2007-present Tom Anderson (tom@squeat.com), see LICENSE

data/lib/spawn.rb

@@ -0,0 +1,6 @@
+require File.join(File.dirname(__FILE__), 'spawn', 'spawn')
+require File.join(File.dirname(__FILE__), 'spawn', 'patches')
+
+ActiveRecord::Base.send :include, Spawn
+ActionController::Base.send :include, Spawn
+ActiveRecord::Observer.send :include, Spawn

data/lib/spawn/patches.rb

@@ -0,0 +1,109 @@
+# see activerecord/lib/active_record/connection_adaptors/abstract/connection_specification.rb
+class ActiveRecord::Base
+  # reconnect without disconnecting
+  if Spawn::RAILS_2_2
+    def self.spawn_reconnect(klass=self)
+      # keep ancestors' connection_handlers around to avoid them being garbage collected in the forked child
+      @@ancestor_connection_handlers ||= []
+      @@ancestor_connection_handlers << self.connection_handler
+      self.connection_handler = ActiveRecord::ConnectionAdapters::ConnectionHandler.new
+
+      establish_connection
+    end
+  else
+    def self.spawn_reconnect(klass=self)
+      spec = @@defined_connections[klass.name]
+      konn = active_connections[klass.name]
+      # remove from internal arrays before calling establish_connection so that
+      # the connection isn't disconnected when it calls AR::Base.remove_connection
+      @@defined_connections.delete_if { |key, value| value == spec }
+      active_connections.delete_if { |key, value| value == konn }
+      establish_connection(spec ? spec.config : nil)
+    end
+  end
+
+  # this patch not needed on Rails 2.x and later
+  if Spawn::RAILS_1_x
+    # monkey patch to fix threading problems,
+    # see: http://dev.rubyonrails.org/ticket/7579
+    def self.clear_reloadable_connections!
+      if @@allow_concurrency
+        # Hash keyed by thread_id in @@active_connections. Hash of hashes.
+        @@active_connections.each do |thread_id, conns|
+          conns.each do |name, conn|
+            if conn.requires_reloading?
+              conn.disconnect!
+              @@active_connections[thread_id].delete(name)
+            end
+          end
+        end
+      else
+        # Just one level hash, no concurrency.
+        @@active_connections.each do |name, conn|
+          if conn.requires_reloading?
+            conn.disconnect!
+            @@active_connections.delete(name)
+          end
+        end
+      end
+    end
+  end
+end
+
+# see mongrel/lib/mongrel.rb
+# it's possible that this is not defined if you're running outside of mongrel
+# examples: ./script/runner or ./script/console
+if defined? Mongrel::HttpServer
+  class Mongrel::HttpServer
+    # redefine Montrel::HttpServer::process_client so that we can intercept
+    # the socket that is being used so Spawn can close it upon forking
+    alias_method :orig_process_client, :process_client
+    def process_client(client)
+      Spawn.resources_to_close(client, @socket)
+      orig_process_client(client)
+    end
+  end
+end
+
+need_passenger_patch = true
+if defined? PhusionPassenger::VERSION_STRING
+  # The VERSION_STRING variable was defined sometime after 2.1.0.
+  # We don't need passenger patch for 2.2.2 or later.
+  pv = PhusionPassenger::VERSION_STRING.split('.').collect{|s| s.to_i}
+  need_passenger_patch = pv[0] < 2 || (pv[0] == 2 && (pv[1] < 2 || (pv[1] == 2 && pv[2] < 2)))
+end
+
+if need_passenger_patch
+  # Patch for work with passenger < 2.1.0
+  if defined? Passenger::Railz::RequestHandler
+    class Passenger::Railz::RequestHandler
+      alias_method :orig_process_request, :process_request
+      def process_request(headers, input, output)
+        Spawn.resources_to_close(input, output)
+        orig_process_request(headers, input, output)
+      end
+    end
+  end
+
+  # Patch for work with passenger >= 2.1.0
+  if defined? PhusionPassenger::Railz::RequestHandler
+    class PhusionPassenger::Railz::RequestHandler
+      alias_method :orig_process_request, :process_request
+      def process_request(headers, input, output)
+        Spawn.resources_to_close(input, output)
+        orig_process_request(headers, input, output)
+      end
+    end
+  end
+
+  # Patch for passenger with Rails >= 2.3.0 (uses rack)
+  if defined? PhusionPassenger::Rack::RequestHandler
+    class PhusionPassenger::Rack::RequestHandler
+      alias_method :orig_process_request, :process_request
+      def process_request(headers, input, output)
+        Spawn.resources_to_close(input, output)
+        orig_process_request(headers, input, output)
+      end
+    end
+  end
+end

data/lib/spawn/spawn.rb

@@ -0,0 +1,211 @@
+module Spawn
+  RAILS_1_x = (::Rails::VERSION::MAJOR == 1) unless defined?(RAILS_1_x)
+  RAILS_2_2 = (::Rails::VERSION::MAJOR > 2 || (::Rails::VERSION::MAJOR == 2 && ::Rails::VERSION::MINOR >= 2)) unless defined?(RAILS_2_2)
+
+  @@default_options = {
+    # default to forking (unless windows or jruby)
+    :method => ((RUBY_PLATFORM =~ /(win32|java)/) ? :thread : :fork),
+    :nice   => nil,
+    :kill   => false,
+    :argv   => nil
+  }
+  
+  # things to close in child process
+  @@resources = []
+  # in some environments, logger isn't defined
+  @@logger = defined?(RAILS_DEFAULT_LOGGER) ? RAILS_DEFAULT_LOGGER : Logger.new(STDERR)
+  # forked children to kill on exit
+  @@punks = []
+
+  # Set the options to use every time spawn is called unless specified
+  # otherwise.  For example, in your environment, do something like
+  # this:
+  #   Spawn::default_options = {:nice => 5}
+  # to default to using the :nice option with a value of 5 on every call.
+  # Valid options are:
+  #   :method => (:thread | :fork | :yield)
+  #   :nice   => nice value of the forked process
+  #   :kill   => whether or not the parent process will kill the
+  #              spawned child process when the parent exits
+  #   :argv   => changes name of the spawned process as seen in ps
+  def self.default_options(options = {})
+    @@default_options.merge!(options)
+    @@logger.info "spawn> default options = #{options.inspect}"
+  end
+
+  # @deprecated - please use Spawn::default_options(:method => ) instead
+  # add calls to this in your environment.rb to set your configuration, for example,
+  # to use forking everywhere except your 'development' environment:
+  #   Spawn::method :fork
+  #   Spawn::method :thread, 'development'
+  def self.method(method, env = nil)
+    @@logger.warn "spawn> please use Spawn::default_options(:method => #{method}) instead of Spawn::method"
+    if !env || env == RAILS_ENV
+      default_options :method => method
+    end
+  end
+
+  # set the resources to disconnect from in the child process (when forking)
+  def self.resources_to_close(*resources)
+    @@resources = resources
+  end
+
+  # close all the resources added by calls to resource_to_close
+  def self.close_resources
+    @@resources.each do |resource|
+      resource.close if resource && resource.respond_to?(:close) && !resource.closed?
+    end
+    # in case somebody spawns recursively
+    @@resources.clear
+  end
+  
+  def self.alive?(pid)
+    begin
+      Process::kill 0, pid
+      # if the process is alive then kill won't throw an exception
+      true
+    rescue Errno::ESRCH
+      false
+    end
+  end
+  
+  def self.kill_punks
+    @@punks.each do |punk|
+      if alive?(punk)
+        @@logger.info "spawn> parent(#{Process.pid}) killing child(#{punk})"
+        begin
+          Process.kill("TERM", punk)
+        rescue
+        end
+      end
+    end
+    @@punks = []
+  end
+  # register to kill marked children when parent exits
+  at_exit {kill_punks}
+
+  # Spawns a long-running section of code and returns the ID of the spawned process.
+  # By default the process will be a forked process.   To use threading, pass
+  # :method => :thread or override the default behavior in the environment by setting
+  # 'Spawn::method :thread'.
+  def spawn(opts = {})
+    options = @@default_options.merge(opts.symbolize_keys)
+    # setting options[:method] will override configured value in default_options[:method]
+    if options[:method] == :yield
+      yield
+    elsif options[:method] == :thread
+      # for versions before 2.2, check for allow_concurrency
+      if RAILS_2_2 || ActiveRecord::Base.allow_concurrency
+        thread_it(options) { yield }
+      else
+        @@logger.error("spawn(:method=>:thread) only allowed when allow_concurrency=true")
+        raise "spawn requires config.active_record.allow_concurrency=true when used with :method=>:thread"
+      end
+    else
+      fork_it(options) { yield }
+    end
+  end
+  
+  def wait(sids = [])
+    # wait for all threads and/or forks (if a single sid passed in, convert to array first)
+    Array(sids).each do |sid|
+      if sid.type == :thread
+        sid.handle.join()
+      else
+        begin
+          Process.wait(sid.handle)
+        rescue
+          # if the process is already done, ignore the error
+        end
+      end
+    end
+    # clean up connections from expired threads
+    ActiveRecord::Base.verify_active_connections!()
+  end
+  
+  class SpawnId
+    attr_accessor :type
+    attr_accessor :handle
+    def initialize(t, h)
+      self.type = t
+      self.handle = h
+    end
+  end
+
+  protected
+  def fork_it(options)
+    # The problem with rails is that it only has one connection (per class),
+    # so when we fork a new process, we need to reconnect.
+    @@logger.debug "spawn> parent PID = #{Process.pid}"
+    child = fork do
+      begin
+        start = Time.now
+        @@logger.debug "spawn> child PID = #{Process.pid}"
+
+        # this child has no children of it's own to kill (yet)
+        @@punks = []
+
+        # set the nice priority if needed
+        Process.setpriority(Process::PRIO_PROCESS, 0, options[:nice]) if options[:nice]
+
+        # disconnect from the listening socket, et al
+        Spawn.close_resources
+        # get a new connection so the parent can keep the original one
+        ActiveRecord::Base.spawn_reconnect
+        
+        # set the process name
+        $0 = options[:argv] if options[:argv]
+
+        # run the block of code that takes so long
+        yield
+
+      rescue => ex
+        @@logger.error "spawn> Exception in child[#{Process.pid}] - #{ex.class}: #{ex.message}"
+      ensure
+        begin
+          # to be safe, catch errors on closing the connnections too
+          if RAILS_2_2
+            ActiveRecord::Base.connection_handler.clear_all_connections!
+          else
+            ActiveRecord::Base.connection.disconnect!
+            ActiveRecord::Base.remove_connection
+          end
+        ensure
+          @@logger.info "spawn> child[#{Process.pid}] took #{Time.now - start} sec"
+          # ensure log is flushed since we are using exit!
+          @@logger.flush if @@logger.respond_to?(:flush)
+          # this child might also have children to kill if it called spawn
+          Spawn::kill_punks
+          # this form of exit doesn't call at_exit handlers
+          exit!(0)
+        end
+      end
+    end
+
+    # detach from child process (parent may still wait for detached process if they wish)
+    Process.detach(child)
+
+    # remove dead children from the target list to avoid memory leaks
+    @@punks.delete_if {|punk| !Spawn::alive?(punk)}
+
+    # mark this child for death when this process dies
+    if options[:kill]
+      @@punks << child
+      @@logger.debug "spawn> death row = #{@@punks.inspect}"
+    end
+
+    return SpawnId.new(:fork, child)
+  end
+
+  def thread_it(options)
+    # clean up stale connections from previous threads
+    ActiveRecord::Base.verify_active_connections!()
+    thr = Thread.new do
+      # run the long-running code block
+      yield
+    end
+    thr.priority = -options[:nice] if options[:nice]
+    return SpawnId.new(:thread, thr)
+  end
+
+end

data/tlconnor-spawn.gemspec

@@ -0,0 +1,15 @@
+$:.push File.expand_path("../lib", __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name          = 'tlconnor-spawn'
+  gem.version       = '1.0.0'
+  gem.platform      = Gem::Platform::RUBY
+  gem.authors       = ['Tim Connor']
+  gem.email         = 'tlconnor@gmail.com'
+  gem.summary       = 'Spawn'
+  gem.description   = gem.summary
+
+  gem.files         = `git ls-files`.split("\n")
+  gem.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  gem.require_paths = ["lib"]
+end

metadata

@@ -0,0 +1,74 @@
+--- !ruby/object:Gem::Specification 
+name: tlconnor-spawn
+version: !ruby/object:Gem::Version 
+  hash: 23
+  prerelease: 
+  segments: 
+  - 1
+  - 0
+  - 0
+  version: 1.0.0
+platform: ruby
+authors: 
+- Tim Connor
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2012-03-09 00:00:00 +13:00
+default_executable: 
+dependencies: []
+
+description: Spawn
+email: tlconnor@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- .gitignore
+- CHANGELOG
+- LICENSE
+- README.markdown
+- lib/spawn.rb
+- lib/spawn/patches.rb
+- lib/spawn/spawn.rb
+- tlconnor-spawn.gemspec
+has_rdoc: true
+homepage: 
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.6.2
+signing_key: 
+specification_version: 3
+summary: Spawn
+test_files: []
+