rainbows 0.1.0

data/SIGNALS

@@ -0,0 +1,94 @@
+== Signal handling
+
+In general, signals need only be sent to the master process.  However,
+the signals Rainbows! uses internally to communicate with the worker
+processes are documented here as well.
+
+=== Master Process
+
+* HUP - reload config file, app, and gracefully restart all workers
+
+* INT/TERM - quick shutdown, kills all workers immediately
+
+* QUIT - graceful shutdown, waits for workers to finish their
+  current request before finishing.
+
+* USR1 - reopen all logs owned by the master and all workers
+  See Unicorn::Util.reopen_logs for what is considered a log.
+
+* USR2 - reexecute the running binary.  A separate QUIT
+  should be sent to the original process once the child is verified to
+  be up and running.
+
+* WINCH - gracefully stops workers but keep the master running.
+  This will only work for daemonized processes.
+
+* TTIN - increment the number of worker processes by one
+
+* TTOU - decrement the number of worker processes by one
+
+=== Worker Processes
+
+Sending signals directly to the worker processes should not normally be
+needed.  If the master process is running, any exited worker will be
+automatically respawned.
+
+* INT/TERM - Quick shutdown, immediately exit.
+  Unless WINCH has been sent to the master (or the master is killed),
+  the master process will respawn a worker to replace this one.
+
+* QUIT - Gracefully exit after finishing the current request.
+  Unless WINCH has been sent to the master (or the master is killed),
+  the master process will respawn a worker to replace this one.
+
+* USR1 - Reopen all logs owned by the worker process.
+  See Unicorn::Util.reopen_logs for what is considered a log.
+  Log files are not reopened until it is done processing
+  the current request, so multiple log lines for one request
+  (as done by Rails) will not be split across multiple logs.
+
+=== Procedure to replace a running rainbows executable
+
+You may replace a running instance of unicorn with a new one without
+losing any incoming connections.  Doing so will reload all of your
+application code, Unicorn config, Ruby executable, and all libraries.
+The only things that will not change (due to OS limitations) are:
+
+1. The path to the rainbows executable script.  If you want to change to
+   a different installation of Ruby, you can modify the shebang
+   line to point to your alternative interpreter.
+
+The procedure is exactly like that of nginx:
+
+1. Send USR2 to the master process
+
+2. Check your process manager or pid files to see if a new master spawned
+   successfully.  If you're using a pid file, the old process will have
+   ".oldbin" appended to its path.  You should have two master instances
+   of rainbows running now, both of which will have workers servicing
+   requests.  Your process tree should look something like this:
+
+     rainbows master (old)
+     \_ rainbows worker[0]
+     \_ rainbows worker[1]
+     \_ rainbows worker[2]
+     \_ rainbows worker[3]
+     \_ rainbows master
+        \_ rainbows worker[0]
+        \_ rainbows worker[1]
+        \_ rainbows worker[2]
+        \_ rainbows worker[3]
+
+3. You can now send WINCH to the old master process so only the new workers
+   serve requests.  If your rainbows process is bound to an
+   interactive terminal, you can skip this step.  Step 5 will be more
+   difficult but you can also skip it if your process is not daemonized.
+
+4. You should now ensure that everything is running correctly with the
+   new workers as the old workers die off.
+
+5. If everything seems ok, then send QUIT to the old master.  You're done!
+
+   If something is broken, then send HUP to the old master to reload
+   the config and restart its workers.  Then send QUIT to the new master
+   process.

data/TODO

@@ -0,0 +1,20 @@
+= TODO items for Rainbows!
+
+We're lazy and pick the easy items to do first, then the ones people
+care about.
+
+* Rev (without Revactor) - since we already use Revactor we might as
+  well support this one so 1.8 users won't be left out.  Doing TeeInput
+  is probably going to get ugly, though...
+
+* EventMachine - much like Rev, but we haven't looked at this one much
+  (our benevolent dictator doesn't like C++).  If we can figure out how
+  to do Rev without Revactor, then this should be pretty easy.
+
+* Fiber support - Revactor already uses these with Ruby 1.9, also not
+  sure how TeeInput can be done with this.
+
+* Omnibus - haven't looked into it, probably like Revactor with 1.8?
+
+* Rubinius Actors - should be like Revactor and easily doable once
+  Rubinius gets more mature.

data/TUNING

@@ -0,0 +1,31 @@
+= Tuning \Rainbows!
+
+Most of the {tuning notes}[http://unicorn.bogomips.org/TUNING.html]
+apply to \Rainbows! as well.  \Rainbows! is not particularly optimized
+at the moment and is designed for applications that spend large amounts
+of the time waiting on network activity.  Thus memory usage and memory
+bandwidth for keeping connections open are often limiting factors as
+well.
+
+== \Rainbows! configuration
+
+* Don't set +worker_connections+ too high.  It is often better to start
+  denying requests and only serve the clients you can than to be
+  completely bogged down and be unusable for everybody.
+
+* Increase +worker_processes+ if you have resources (RAM/DB connections)
+  available.  Additional worker processes can better utilize SMP, are more
+  robust against crashes and are more likely to be fairly scheduled by
+  the kernel.
+
+== nginx configuration
+
+If you intend to use nginx as a reverse-proxy in front of \Rainbows!  to
+handle Comet applications, make sure you disable proxy response
+buffering in nginx:
+
+  proxy_buffering off;
+
+This can be disabled on a per-backend basis in nginx, so under no
+circumstances should you disable response buffering to Unicorn
+backends, only to \Rainbows! backends.

data/bin/rainbows

@@ -0,0 +1,166 @@
+#!/home/ew/bin/ruby
+# -*- encoding: binary -*-
+require 'unicorn/launcher'
+require 'rainbows'
+require 'optparse'
+
+env = "development"
+daemonize = false
+listeners = []
+options = { :listeners => listeners }
+host, port = Unicorn::Const::DEFAULT_HOST, Unicorn::Const::DEFAULT_PORT
+set_listener = false
+
+opts = OptionParser.new("", 24, '  ') do |opts|
+  opts.banner = "Usage: #{File.basename($0)} " \
+                "[ruby options] [unicorn options] [rackup config file]"
+
+  opts.separator "Ruby options:"
+
+  lineno = 1
+  opts.on("-e", "--eval LINE", "evaluate a LINE of code") do |line|
+    eval line, TOPLEVEL_BINDING, "-e", lineno
+    lineno += 1
+  end
+
+  opts.on("-d", "--debug", "set debugging flags (set $DEBUG to true)") do
+    $DEBUG = true
+  end
+
+  opts.on("-w", "--warn", "turn warnings on for your script") do
+    $-w = true
+  end
+
+  opts.on("-I", "--include PATH",
+          "specify $LOAD_PATH (may be used more than once)") do |path|
+    $LOAD_PATH.unshift(*path.split(/:/))
+  end
+
+  opts.on("-r", "--require LIBRARY",
+          "require the library, before executing your script") do |library|
+    require library
+  end
+
+  opts.separator "Unicorn options:"
+
+  # some of these switches exist for rackup command-line compatibility,
+
+  opts.on("-o", "--host HOST",
+          "listen on HOST (default: #{Unicorn::Const::DEFAULT_HOST})") do |h|
+    host = h
+    set_listener = true
+  end
+
+  opts.on("-p", "--port PORT",
+          "use PORT (default: #{Unicorn::Const::DEFAULT_PORT})") do |p|
+    port = p.to_i
+    set_listener = true
+  end
+
+  opts.on("-E", "--env ENVIRONMENT",
+          "use ENVIRONMENT for defaults (default: development)") do |e|
+    env = e
+  end
+
+  opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
+    daemonize = d ? true : false
+  end
+
+  opts.on("-P", "--pid FILE", "DEPRECATED") do |f|
+    warn %q{Use of --pid/-P is strongly discouraged}
+    warn %q{Use the 'pid' directive in the Unicorn config file instead}
+    options[:pid] = File.expand_path(f)
+  end
+
+  opts.on("-s", "--server SERVER",
+          "this flag only exists for compatibility") do |s|
+    warn "-s/--server only exists for compatibility with rackup"
+  end
+
+  # Unicorn-specific stuff
+  opts.on("-l", "--listen {HOST:PORT|PATH}",
+          "listen on HOST:PORT or PATH",
+          "this may be specified multiple times",
+          "(default: #{Unicorn::Const::DEFAULT_LISTEN})") do |address|
+    listeners << address
+  end
+
+  opts.on("-c", "--config-file FILE", "Unicorn-specific config file") do |f|
+    options[:config_file] = File.expand_path(f)
+  end
+
+  # I'm avoiding Unicorn-specific config options on the command-line.
+  # IMNSHO, config options on the command-line are redundant given
+  # config files and make things unnecessarily complicated with multiple
+  # places to look for a config option.
+
+  opts.separator "Common options:"
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
+    exit
+  end
+
+  opts.on_tail("-v", "--version", "Show version") do
+    puts "unicorn v#{Unicorn::Const::UNICORN_VERSION}"
+    exit
+  end
+
+  opts.parse! ARGV
+end
+
+config = ARGV[0] || "config.ru"
+abort "configuration file #{config} not found" unless File.exist?(config)
+
+if config =~ /\.ru$/
+  # parse embedded command-line options in config.ru comments
+  if File.open(config, "rb") { |fp| fp.sysread(fp.stat.size) } =~ /^#\\(.*)/
+    opts.parse! $1.split(/\s+/)
+  end
+end
+
+require 'pp' if $DEBUG
+
+app = lambda do ||
+  # require Rack as late as possible in case $LOAD_PATH is modified
+  # in config.ru or command-line
+  inner_app = case config
+  when /\.ru$/
+    raw = File.open(config, "rb") { |fp| fp.sysread(fp.stat.size) }
+    raw.sub!(/^__END__\n.*/, '')
+    eval("Rack::Builder.new {(#{raw}\n)}.to_app", nil, config)
+  else
+    require config
+    Object.const_get(File.basename(config, '.rb').capitalize)
+  end
+  pp({ :inner_app => inner_app }) if $DEBUG
+  case env
+  when "development"
+    Rack::Builder.new do
+      use Rack::CommonLogger, $stderr
+      use Rack::ShowExceptions
+      use Rack::Lint
+      run inner_app
+    end.to_app
+  when "deployment"
+    Rack::Builder.new do
+      use Rack::CommonLogger, $stderr
+      run inner_app
+    end.to_app
+  else
+    inner_app
+  end
+end
+
+listeners << "#{host}:#{port}" if set_listener
+
+if $DEBUG
+  pp({
+    :unicorn_options => options,
+    :app => app,
+    :daemonize => daemonize,
+  })
+end
+
+Unicorn::Launcher.daemonize! if daemonize
+Rainbows.run(app, options)

data/lib/rainbows.rb

@@ -0,0 +1,53 @@
+# -*- encoding: binary -*-
+require 'unicorn'
+
+module Rainbows
+
+  require 'rainbows/const'
+  require 'rainbows/http_server'
+  require 'rainbows/http_response'
+  require 'rainbows/base'
+
+  class << self
+
+    # runs the Rainbows! HttpServer with +app+ and +options+ and does
+    # not return until the server has exited.
+    def run(app, options = {})
+      HttpServer.new(app, options).start.join
+    end
+  end
+
+  # configures Rainbows! with a given concurrency model to +use+ and
+  # a +worker_connections+ upper-bound.  This method may be called
+  # inside a Unicorn/Rainbows configuration file:
+  #
+  #   Rainbows! do
+  #     use :Revactor # this may also be :ThreadSpawn or :ThreadPool
+  #     worker_connections 128
+  #   end
+  #
+  # See the documentation for the respective Revactor, ThreadSpawn,
+  # and ThreadPool classes for descriptions and recommendations for
+  # each of them.
+  def Rainbows!(&block)
+    block_given? or raise ArgumentError, "Rainbows! requires a block"
+    HttpServer.setup(block)
+  end
+
+  # maps models to default worker counts, default worker count numbers are
+  # pretty arbitrary and tuning them to your application and hardware is
+  # highly recommended
+  MODEL_WORKER_CONNECTIONS = {
+    :Base => 1, # this one can't change
+    :Revactor => 50,
+    :ThreadSpawn => 30,
+    :ThreadPool => 10,
+  }.each do |model, _|
+    u = model.to_s.gsub(/([a-z0-9])([A-Z0-9])/) { "#{$1}_#{$2.downcase!}" }
+    autoload model, "rainbows/#{u.downcase!}"
+  end
+
+end
+
+# inject the Rainbows! method into Unicorn::Configurator
+Unicorn::Configurator.class_eval { include Rainbows }

data/lib/rainbows/base.rb

@@ -0,0 +1,69 @@
+# -*- encoding: binary -*-
+
+module Rainbows
+
+  # base class for Rainbows concurrency models
+  module Base
+
+    include Unicorn
+    include Rainbows::Const
+
+    # write a response without caring if it went out or not for error
+    # messages.
+    # TODO: merge into Unicorn::HttpServer
+    def emergency_response(client, response_str)
+      client.write_nonblock(response_str) rescue nil
+      client.close rescue nil
+    end
+
+    # once a client is accepted, it is processed in its entirety here
+    # in 3 easy steps: read request, call app, write app response
+    def process_client(client)
+      buf = client.readpartial(CHUNK_SIZE)
+      hp = HttpParser.new
+      env = {}
+      remote_addr = TCPSocket === client ? client.peeraddr.last : LOCALHOST
+
+      begin
+        while ! hp.headers(env, buf)
+          buf << client.readpartial(CHUNK_SIZE)
+        end
+
+        env[RACK_INPUT] = 0 == hp.content_length ?
+                 HttpRequest::NULL_IO :
+                 Unicorn::TeeInput.new(client, env, hp, buf)
+        env[REMOTE_ADDR] = remote_addr
+        response = app.call(env.update(RACK_DEFAULTS))
+
+        if 100 == response.first.to_i
+          client.write(EXPECT_100_RESPONSE)
+          env.delete(HTTP_EXPECT)
+          response = app.call(env)
+        end
+
+        out = [ hp.keepalive? ? CONN_ALIVE : CONN_CLOSE ] if hp.headers?
+        HttpResponse.write(client, response, out)
+      end while hp.keepalive? and hp.reset.nil? and env.clear
+      client.close
+    # if we get any error, try to write something back to the client
+    # assuming we haven't closed the socket, but don't get hung up
+    # if the socket is already closed or broken.  We'll always ensure
+    # the socket is closed at the end of this function
+    rescue EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF
+      emergency_response(client, ERROR_500_RESPONSE)
+    rescue HttpParserError # try to tell the client they're bad
+      buf.empty? or emergency_response(client, ERROR_400_RESPONSE)
+    rescue Object => e
+      emergency_response(client, ERROR_500_RESPONSE)
+      logger.error "Read error: #{e.inspect}"
+      logger.error e.backtrace.join("\n")
+    end
+
+    def self.included(klass)
+      HttpServer.constants.each do |x|
+        klass.const_set(x, HttpServer.const_get(x))
+      end
+    end
+
+  end
+end

data/lib/rainbows/const.rb

@@ -0,0 +1,24 @@
+# -*- encoding: binary -*-
+
+module Rainbows
+
+  module Const
+    RAINBOWS_VERSION = '0.1.0'
+
+    include Unicorn::Const
+
+    RACK_DEFAULTS = ::Unicorn::HttpRequest::DEFAULTS.merge({
+
+      # we need to observe many of the rules for thread-safety even
+      # with Revactor or Rev, so we're considered multithread-ed even
+      # when we're not technically...
+      "rack.multithread" => true,
+      "SERVER_SOFTWARE" => "Rainbows! #{RAINBOWS_VERSION}",
+    })
+
+    CONN_CLOSE = "Connection: close\r\n"
+    CONN_ALIVE = "Connection: keep-alive\r\n"
+    LOCALHOST = "127.0.0.1"
+
+  end
+end