tastebook-spawn 1.0.1

data/CHANGELOG

@@ -0,0 +1,51 @@
+v0.1 - 2007/09/13
+
+initial version
+
+--------------------------------------------------
+v0.2 - 2007/09/28
+
+* return PID of the child process
+* added ":detach => false" option
+
+--------------------------------------------------
+v0.3 - 2007/10/15
+
+* added ':method => :thread' for threaded spawns
+* removed ':detach => false' option in favor of more generic implementation
+* added ability to set configuration of the form 'Spawn::method :thread'
+* added patch to ActiveRecord::Base to allow for more efficient reconnect in child processes
+* added monkey patch for http://dev.rubyonrails.org/ticket/7579
+* added wait() method to wait for spawned code blocks
+* don't allow threading if allow_concurrency=false
+
+--------------------------------------------------
+v0.4 - 2008/1/26
+
+* default to :thread on windows, still :fork on all other platforms
+* raise exception when used with :method=>:true and allow_concurrency != true
+
+--------------------------------------------------
+v0.5 - 2008/3/1
+* also default to :thread on JRuby (java)
+* added new :method => :yield which doesn't fork or thread, this is useful for testing
+* fixed problem with connections piling up on PostgreSQL
+
+--------------------------------------------------
+v0.6 - 2008/04/21
+* only apply clear_reloadable_connections patch on Rails 1.x (7579 fixed in Rails 2.x)
+* made it more responsive in more environments by disconnecting from the listener socket in the forked process
+
+--------------------------------------------------
+v0.7 - 2008/04/24
+* more generic mechanism for closing resources after fork
+* check for existence of Mongrel before patching it
+
+--------------------------------------------------
+v0.8 - 2008/05/02
+* call exit! within the ensure block so that at_exit handlers aren't called on exceptions
+* set logger from RAILS_DEFAULT_LOGGER if available, else STDERR
+
+--------------------------------------------------
+v0.9 - 2008/05/11
+* added ability to set nice level for child process

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2007 Tom Anderson (tom@squeat.com)
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,194 @@
+# Spawn
+
+This plugin provides a 'spawn' method to easily fork OR thread long-running sections of
+code so that your application can return results to your users more quickly.
+This plugin works by creating new database connections in ActiveRecord::Base for the
+spawned block.
+
+The plugin also patches ActiveRecord::Base to handle some known bugs when using 
+threads (see lib/patches.rb).
+
+## Installation
+
+Use
+
+    gem "spawn", :git => 'git://github.com/rfc2822/spawn'
+
+in your Gemfile and use bundler to manage it (bundle install, bundle update).
+
+Make sure that ActiveRecord reconnects to your database automatically when needed,
+for instance put
+
+    production/development:
+      ...
+      reconnect: true
+
+into your config/database.yml.
+
+## Usage
+
+Here's a simple example of how to demonstrate the spawn plugin.
+In one of your controllers, insert this code (after installing the plugin of course):
+
+    spawn_block do
+       logger.info("I feel sleepy...")
+       sleep 11
+       logger.info("Time to wake up!")
+     end
+
+If everything is working correctly, your controller should finish quickly then you'll see
+the last log message several seconds later.
+
+If you need to wait for the spawned processes/threads, then pass the objects returned by
+spawn to Spawn::wait(), like this:
+
+    N.times do |i|
+      # spawn N blocks of code
+      spawn_ids[i] = spawn_block do
+        something(i)
+      end
+    end
+    # wait for all N blocks of code to finish running
+    wait(spawn_ids)
+
+## Options
+
+The options you can pass to spawn_block are:
+
+<table>
+  <tr><th>Option</th><th>Values</th></tr>
+  <tr><td>:method</td><td>:fork, :thread, :yield</td></tr>
+  <tr><td>:nice</td><td>integer value 0-19, 19 = really nice</td></tr>
+  <tr><td>:kill</td><td>boolean value indicating whether the parent should kill the spawned process
+   when it exits (only valid when :method => :fork)</td></tr>
+  <tr><td>:argv</td><td>string to override the process name</td></tr>
+</table>
+
+Any option to spawn_block can be set as a default so that you don't have to pass them in
+to every call of spawn_block.   To configure the spawn default options, add a line to
+your configuration file(s) like this: 
+
+    Spawn::default_options {:method => :thread}
+
+If you don't set any default options, the :method will default to :fork.  To
+specify different values for different environments, add the default_options call to
+he appropriate environment file (development.rb, test.rb).   For testing you can set
+the default :method to :yield so that the code is run inline.
+
+    # in environment.rb
+    Spawn::method :method => :fork, :nice => 7
+    # in test.rb, will override the environment.rb setting
+    Spawn::method :method => :yield
+
+This allows you to set your production and development environments to use different
+methods according to your needs.
+
+### be nice
+
+If you want your forked child to run at a lower priority than the parent process, pass in
+the :nice option like this:
+
+    spawn_block(:nice => 7) do
+      do_something_nicely
+    end
+
+### fork me
+
+By default, spawn will use the fork to spawn child processes.  You can configure it to
+do threading either by telling the spawn method when you call it or by configuring your
+environment.
+For example, this is how you can tell spawn to use threading on the call,
+
+    spawn_block(:method => :thread) do
+      something
+    end
+
+When you use threaded spawning, make sure that your application is thread-safe. Rails
+can be switched to thread-safe mode with
+
+    # Enable threaded mode
+    config.threadsafe!
+
+in environments/your_environment.rb
+
+### kill or be killed
+
+Depending on your application, you may want the children processes to go away when
+the parent  process exits.   By default spawn lets the children live after the
+parent dies.   But you can tell it to kill the children by setting the :kill option
+to true.
+
+### a process by any other name
+
+If you'd like to be able to identify which processes are spawned by looking at the
+output of ps then set the :argv option with a string of your choice.
+You should then be able to see this string as the process name when
+listing the running processes (ps).
+
+For example, if you do something like this,
+
+    3.times do |i|
+      spawn_block(:argv => "spawn -#{i}-") do
+        something(i)
+      end
+    end
+
+then in the shell,
+
+    $ ps -ef | grep spawn
+    502  2645  2642   0   0:00.01 ttys002    0:00.02 spawn -0-
+    502  2646  2642   0   0:00.02 ttys002    0:00.02 spawn -1-
+    502  2647  2642   0   0:00.02 ttys002    0:00.03 spawn -2-
+
+The length of the process name may be limited by your OS so you might want to experiment
+to see how long it can be (it may be limited by the length of the original process name).
+
+## Forking vs. Threading
+
+There are several tradeoffs for using threading vs. forking.   Forking was chosen as the
+default primarily because it requires no configuration to get it working out of the box.
+
+Forking advantages:
+
+- more reliable? - the ActiveRecord code is generally not deemed to be thread-safe.
+  Even though spawn attempts to patch known problems with the threaded implementation,
+  there are no guarantees.  Forking is heavier but should be fairly reliable.
+- keep running - this could also be a disadvantage, but you may find you want to fork
+  off a process that could have a life longer than its parent.  For example, maybe you
+  want to restart your server without killing the spawned processes.
+  We don't necessarily condone this (i.e. haven't tried it) but it's technically possible.
+- easier - forking works out of the box with spawn, threading requires you set
+  allow_concurrency=true (for older versions of Rails).
+  Also, beware of automatic reloading of classes in development
+  mode (config.cache_classes = false).
+
+Threading advantages:
+- less filling - threads take less resources... how much less?  it depends.   Some 
+  flavors of Unix are pretty efficient at forking so the threading advantage may not
+  be as big as you think... but then again, maybe it's more than you think.  ;-)
+- debugging - you can set breakpoints in your threads
+
+## Acknowledgements
+
+This plugin was initially inspired by Scott Persinger's blog post on how to use fork
+in rails for background processing.
+    http://geekblog.vodpod.com/?p=26
+
+Further inspiration for the threading implementation came from Jonathon Rochkind's
+blog post on threading in rails.
+    http://bibwild.wordpress.com/2007/08/28/threading-in-rails/
+    
+Also thanks to all who have helped debug problems and suggest improvements
+including:
+
+-  Ahmed Adam, Tristan Schneiter, Scott Haug, Andrew Garfield, Eugene Otto, Dan Sharp,
+  Olivier Ruffin, Adrian Duyzer, Cyrille Labesse
+
+-  Garry Tan, Matt Jankowski (Rails 2.2.x fixes), Mina Naguib (Rails 2.3.6 fix)
+
+-  Tim Kadom, Mauricio Marcon Zaffari, Danial Pearce, Hongli Lai, Scott Wadden
+  (passenger fixes)
+
+-  &lt;your name here&gt;
+
+Copyright (c) 2007-present Tom Anderson (tom@squeat.com), see LICENSE

data/lib/spawn.rb

@@ -0,0 +1,191 @@
+module Spawn
+  @@default_options = {
+    # default to forking (unless windows or jruby)
+    :method => ((RUBY_PLATFORM =~ /(win32|java|mingw32)/) ? :thread : :fork),
+    :nice   => nil,
+    :kill   => false,
+    :argv   => nil
+  }
+  
+  # things to close in child process
+  @@resources = []
+  # in some environments, logger isn't defined
+  @@logger = Rails.logger || Logger.new(STDERR)
+  # forked children to kill on exit
+  @@punks = []
+
+  # Set the options to use every time spawn is called unless specified
+  # otherwise.  For example, in your environment, do something like
+  # this:
+  #   Spawn::default_options = {:nice => 5}
+  # to default to using the :nice option with a value of 5 on every call.
+  # Valid options are:
+  #   :method => (:thread | :fork | :yield)
+  #   :nice   => nice value of the forked process
+  #   :kill   => whether or not the parent process will kill the
+  #              spawned child process when the parent exits
+  #   :argv   => changes name of the spawned process as seen in ps
+  def self.default_options(options = {})
+    @@default_options.merge!(options)
+    @@logger.info "spawn> default options = #{options.inspect}"
+  end
+
+  # set the resources to disconnect from in the child process (when forking)
+  def self.resources_to_close(*resources)
+    @@resources = resources
+  end
+
+  # close all the resources added by calls to resource_to_close
+  def self.close_resources
+    @@resources.each do |resource|
+      resource.close if resource && resource.respond_to?(:close) && !resource.closed?
+    end
+    # in case somebody spawns recursively
+    @@resources.clear
+  end
+  
+  def self.alive?(pid)
+    begin
+      Process::kill 0, pid
+      # if the process is alive then kill won't throw an exception
+      true
+    rescue Errno::ESRCH
+      false
+    end
+  end
+  
+  def self.kill_punks
+    @@punks.each do |punk|
+      if alive?(punk)
+        @@logger.info "spawn> parent(#{Process.pid}) killing child(#{punk})"
+        begin
+          Process.kill("TERM", punk)
+        rescue
+        end
+      end
+    end
+    @@punks = []
+  end
+  # register to kill marked children when parent exits
+  at_exit {kill_punks}
+
+  # Spawns a long-running section of code and returns the ID of the spawned process.
+  # By default the process will be a forked process.   To use threading, pass
+  # :method => :thread or override the default behavior in the environment by setting
+  # 'Spawn::method :thread'.
+  def spawn_block(opts = {})
+    options = @@default_options.merge(opts.symbolize_keys)
+    # setting options[:method] will override configured value in default_options[:method]
+    if options[:method] == :yield
+      yield
+    elsif options[:method] == :thread || (options[:method] == nil && @@method == :thread)
+      thread_it(options) { yield }
+    else
+      fork_it(options) { yield }
+    end
+  end
+  
+  def wait(sids = [])
+    # wait for all threads and/or forks (if a single sid passed in, convert to array first)
+    Array(sids).each do |sid|
+      if sid.type == :thread
+        sid.handle.join()
+      else
+        begin
+          Process.wait(sid.handle)
+        rescue
+          # if the process is already done, ignore the error
+        end
+      end
+    end
+    # clean up connections from expired threads
+    ActiveRecord::Base.verify_active_connections!() if defined? ActiveRecord
+  end
+
+  class SpawnId
+    attr_accessor :type
+    attr_accessor :handle
+    def initialize(t, h)
+      self.type = t
+      self.handle = h
+    end
+  end
+
+  protected
+  def fork_it(options)
+    # The problem with rails is that it only has one connection (per class),
+    # so when we fork a new process, we need to reconnect.
+    @@logger.debug "spawn> parent PID = #{Process.pid}"
+    child = fork do
+      begin
+        start = Time.now
+        @@logger.debug "spawn> child PID = #{Process.pid}"
+
+        # this child has no children of it's own to kill (yet)
+        @@punks = []
+
+        # set the nice priority if needed
+        Process.setpriority(Process::PRIO_PROCESS, 0, options[:nice]) if options[:nice]
+
+        # disconnect from the listening socket, et al
+        Spawn.close_resources
+        # get a new connection so the parent can keep the original one
+        # Old spawn did a bunch of hacks inside activerecord here. There is
+        # most likely a reason that this won't work, but I am dumb.
+        ActiveRecord::Base.connection.reconnect! if defined? ActiveRecord
+
+        # set the process name
+        $0 = options[:argv] if options[:argv]
+
+        # run the block of code that takes so long
+        yield
+
+      rescue => ex
+        @@logger.error "spawn> Exception in child[#{Process.pid}] - #{ex.class}: #{ex.message}"
+      ensure
+        begin
+          # to be safe, catch errors on closing the connnections too
+          ActiveRecord::Base.connection_handler.clear_all_connections! if defined? ActiveRecord
+        ensure
+          @@logger.info "spawn> child[#{Process.pid}] took #{Time.now - start} sec"
+          # ensure log is flushed since we are using exit!
+          @@logger.flush if @@logger.respond_to?(:flush)
+          # this child might also have children to kill if it called spawn
+          Spawn::kill_punks
+          # this form of exit doesn't call at_exit handlers
+          exit!(0)
+        end
+      end
+    end
+
+    # detach from child process (parent may still wait for detached process if they wish)
+    Process.detach(child)
+
+    # remove dead children from the target list to avoid memory leaks
+    @@punks.delete_if {|punk| !Spawn::alive?(punk)}
+
+    # mark this child for death when this process dies
+    if options[:kill]
+      @@punks << child
+      @@logger.debug "spawn> death row = #{@@punks.inspect}"
+    end
+
+    return SpawnId.new(:fork, child)
+  end
+
+  def thread_it(options)
+    # clean up stale connections from previous threads
+    ActiveRecord::Base.verify_active_connections!() if defined? ActiveRecord
+    thr = Thread.new do
+      # run the long-running code block
+      yield
+    end
+    thr.priority = -options[:nice] if options[:nice]
+    return SpawnId.new(:thread, thr)
+  end
+end
+
+
+ActiveRecord::Base.send :include, Spawn if defined? ActiveRecord
+ActionController::Base.send :include, Spawn
+ActiveRecord::Observer.send :include, Spawn if defined? ActiveRecord

metadata

@@ -0,0 +1,60 @@
+--- !ruby/object:Gem::Specification
+name: tastebook-spawn
+version: !ruby/object:Gem::Version
+  version: 1.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Tom Anderson
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2010-08-08 00:00:00.000000000 Z
+dependencies: []
+description: ! 'This plugin provides a ''spawn'' method to easily fork OR
+
+  thread long-running sections of code so that your application can return
+
+  results to your users more quickly.  This plugin works by creating new database
+
+  connections in ActiveRecord::Base for the spawned block.
+
+
+  The plugin also patches ActiveRecord::Base to handle some known bugs when using
+
+  threads (see lib/patches.rb).'
+email:
+- tom@squeat.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/spawn.rb
+- CHANGELOG
+- LICENSE
+- README.markdown
+homepage: http://github.com/tra/spawn
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.3.6
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.17
+signing_key: 
+specification_version: 3
+summary: Easily fork OR thread long-running sections of code in Ruby
+test_files: []