unicorn 0.4.2 → 0.5.0

data/CHANGELOG

@@ -1,3 +1,4 @@
+v0.5.0 - {after,before}_fork API change, small tweaks/fixes
 v0.4.2 - fix Rails ARStore, FD leak prevention, descriptive proctitles
 v0.4.1 - Rails support, per-listener backlog and {snd,rcv}buf
 v0.2.3 - Unlink Tempfiles after use (they were closed, just not unlinked)

data/DESIGN

@@ -37,10 +37,11 @@
 
 * The number of worker processes should be scaled to the number of
   CPUs, memory or even spindles you have.  If you have an existing
-  Mongrel cluster, using the same amount of processes should work.
-  Let a full-HTTP-request-buffering reverse proxy like nginx manage
-  concurrency to thousands of slow clients for you.  Unicorn scaling
-  should only be concerned about limits of your backend system(s).
+  Mongrel cluster on a single-threaded app, using the same amount of
+  processes should work.  Let a full-HTTP-request-buffering reverse
+  proxy like nginx manage concurrency to thousands of slow clients for
+  you.  Unicorn scaling should only be concerned about limits of your
+  backend system(s).
 
 * Load balancing between worker processes is done by the OS kernel.
   All workers share a common set of listener sockets and does
@@ -56,8 +57,8 @@
 
 * Blocking I/O is used for clients.  This allows a simpler code path
   to be followed within the Ruby interpreter and fewer syscalls.
-  Applications that use threads should continue to work if Unicorn
-  is serving LAN or localhost clients.
+  Applications that use threads continue to work if Unicorn
+  is only serving LAN or localhost clients.
 
 * Timeout implementation is done via fchmod(2) in each worker
   on a shared file descriptor to update st_ctime on the inode.
@@ -67,8 +68,9 @@
   pwrite(2)/pread(2) are supported by base MRI, nor are they as
   portable on UNIX systems as fchmod(2).
 
-* SIGKILL is used to terminate the timed-out workers as reliably
-  as possible on a UNIX system.
+* SIGKILL is used to terminate the timed-out workers from misbehaving apps
+  as reliably as possible on a UNIX system.  The default timeout is a
+  generous 60 seconds (same default as in Mongrel).
 
 * The poor performance of select() on large FD sets is avoided
   as few file descriptors are used in each worker.
@@ -81,5 +83,7 @@
   the master to its death.
 
 * There is never any explicit real-time dependency or communication
-  between the worker processes themselves nor to the master process.
-  Synchronization is handled entirely by the OS kernel.
+  between the worker processes nor to the master process.
+  Synchronization is handled entirely by the OS kernel and shared
+  resources are never accessed by the worker when it is servicing
+  a client.

data/GNUmakefile

@@ -66,8 +66,19 @@ $(slow_tests):
 	@$(MAKE) $(shell $(awk_slow) $@)
 
 TEST_OPTS = -v
-run_test = @echo '*** $(arg)$(extra) ***'; \
-  setsid $(ruby) $(arg) $(TEST_OPTS) >$(t) 2>&1 || \
+TEST_OPTS = -v
+ifndef V
+       quiet_pre = @echo '* $(arg)$(extra)';
+       quiet_post = >$(t) 2>&1
+else
+       # we can't rely on -o pipefail outside of bash 3+,
+       # so we use a stamp file to indicate success and
+       # have rm fail if the stamp didn't get created
+       stamp = $@$(log_suffix).ok
+       quiet_pre = @echo $(ruby) $(arg) $(TEST_OPTS); ! test -f $(stamp) && (
+       quiet_post = && > $(stamp) )>&2 | tee $(t); rm $(stamp) 2>/dev/null
+endif
+run_test = $(quiet_pre) setsid $(ruby) -w $(arg) $(TEST_OPTS) $(quiet_post) || \
   (sed "s,^,$(extra): ," >&2 < $(t); exit 1)
 
 %.n: arg = $(subst .n,,$(subst --, -n ,$@))
@@ -134,6 +145,5 @@ $(T_r).%.r: export UNICORN_RAILS_TEST_VERSION = $(rv)
 $(T_r).%.r: export RAILS_GIT_REPO = $(CURDIR)/$(rails_git)
 $(T_r).%.r: $(test_prefix)/.stamp $(rails_git)/info/cloned-stamp
 	$(run_test)
-	@sed 's,^,$(rv): ,' < $(t)
 
 .PHONY: doc $(T) $(slow_tests) Manifest

data/README

@@ -1,33 +1,35 @@
-= Unicorn: Unix + LAN/localhost-optimized fork of Mongrel
-
-Unicorn is designed to only serve fast clients.  See the PHILOSOPHY
-and DESIGN documents for more details regarding this.
+= Unicorn: Unix + LAN/localhost-only fork of Mongrel
 
 == Features
 
-* Built on the solid Mongrel code base and takes full advantage
-  of functionality exclusive to Unix-like operating systems.
+* Designed for Rack, Unix, fast clients, and ease-of-debugging.  We
+  cut out all things that are better-supported by nginx or Rack.
 
 * Mostly written in Ruby, only the HTTP parser (stolen and trimmed
   down from Mongrel) is written in C.  Unicorn is compatible with
-  both Ruby 1.8 and 1.9.
+  both Ruby 1.8 and 1.9.  A pure-Ruby (but still Unix-only) version
+  is planned.
 
 * Process management: Unicorn will reap and restart workers that
   die from broken apps.  There is no need to manage multiple processes
-  yourself.
+  or ports yourself.  Unicorn can spawn and manage any fixed number of
+  worker processes you choose to scale to your backend.
 
 * Load balancing is done entirely by the operating system kernel.
-  Requests never pile up behind a busy worker.
+  Requests never pile up behind a busy worker process.
 
 * Does not care if your application is thread-safe or not, workers
   all run within their own isolated address space and only serve one
-  client at a time...
+  client at a time.
 
 * Supports all Rack applications, along with pre-Rack versions of
   Ruby on Rails via a Rack wrapper.
 
-* Builtin log rotation of all log files in your application via USR1
-  signal.
+* Builtin reopening of all log files in your application via
+  USR1 signal.  This allows logrotate to rotate files atomically and
+  quickly via rename instead of the racy and slow copytruncate method.
+  Unicorn also takes steps to ensure multi-line log entries from one
+  request all stay within the same file.
 
 * nginx-style binary re-execution without losing connections.
   You can upgrade Unicorn, your entire application, libraries
@@ -35,10 +37,12 @@ and DESIGN documents for more details regarding this.
   installed in the same path.
 
 * before_fork and after_fork hooks in case your application
-  has special needs when dealing with forked processes.
+  has special needs when dealing with forked processes.  These
+  should not be needed when the "preload_app" directive is
+  false (the default).
 
 * Can be used with copy-on-write-friendly memory management
-  to save memory.
+  to save memory (by setting "preload_app" to true).
 
 * Able to listen on multiple interfaces including UNIX sockets,
   each worker process can also bind to a private port via the
@@ -50,7 +54,7 @@ Unicorn is copyright 2009 Eric Wong and contributors.
 It is based on Mongrel and carries the same license:
 
 Mongrel is copyright 2007 Zed A. Shaw and contributors. It is licensed
-under the Ruby license and the GPL2. See the include LICENSE file for
+under the Ruby license and the GPL2. See the included LICENSE file for
 details.
 
 Unicorn is 100% Free Software.
@@ -81,8 +85,8 @@ If you have web browser software for the World Wide Web
 (on the Information Superhighway), you may browse the code from
 your web browser and download the latest snapshot tarballs here:
 
-* http://git.bogomips.org/cgit/unicorn.git (this server runs Unicorn!)
-* http://repo.or.cz/w/unicorn.git (gitweb mirror)
+* http://git.bogomips.org/cgit/unicorn.git (cgit)
+* http://repo.or.cz/w/unicorn.git (gitweb)
 
 == Usage
 
@@ -118,9 +122,12 @@ options.
 
 == Disclaimer
 
-Like the creatures themselves, production deployments of Unicorn are rare.
-There is NO WARRANTY whatsoever if anything goes wrong, but let us know and
-we'll try our best to fix it.
+Like the creatures themselves, production deployments of Unicorn are
+rare or even non-existent.  There is NO WARRANTY whatsoever if anything
+goes wrong, but let us know and we'll try our best to fix it.
+
+Unicorn is designed to only serve fast clients.  See the PHILOSOPHY
+and DESIGN documents for more details regarding this.
 
 Rainbows are NOT included.
 
@@ -137,5 +144,5 @@ Rainbows are NOT included.
 
 == Contact
 
-Newsgroup and mailing list maybe coming...
 Email Eric Wong at normalperson@yhbt.net for now.
+Newsgroup and mailing list maybe coming...

data/SIGNALS

@@ -7,7 +7,7 @@ processes are documented here as well.
 === Master Process
 
  * HUP - reload config file and gracefully restart all workers
-   If preload_app is false (the default), the application code
+   If "preload_app" is false (the default), the application code
    will be reloaded when workers are restarted as well.
 
  * INT/TERM - quick shutdown, kills all workers immediately
@@ -31,12 +31,19 @@ Sending signals directly to the worker processes should not normally be
 needed.  If the master process is running, any exited worker will be
 automatically respawned.
 
- * INT/TERM - quick shutdown, immediately exit
+ * INT/TERM - Quick shutdown, immediately exit.
+   Unless WINCH has been sent to the master (or the master is killed),
+   the master process will respawn a worker to replace this one.
 
- * QUIT - gracefully exit after finishing the current request
+ * QUIT - Gracefully exit after finishing the current request.
+   Unless WINCH has been sent to the master (or the master is killed),
+   the master process will respawn a worker to replace this one.
 
- * USR1 - reopen all logs owned by the worker process
+ * USR1 - Reopen all logs owned by the worker process.
    See Unicorn::Util.reopen_logs for what is considered a log.
+   Log files are not reopened until it is done processing
+   the current request, so multiple log lines for one request
+   (as done by Rails) will not be split across multiple logs.
 
 === Procedure to replace a running unicorn executable
 

data/TODO

@@ -1,7 +1,5 @@
 == 1.0.0
 
-  * tests for preload_app boolean
-
   * reexec_worker_processes config option:
     This is the number of worker processes to startup initially
     when being reexecuted.

data/ext/unicorn/http11/http11.c

@@ -24,6 +24,7 @@ static VALUE sym_http_body;
 #define HTTP_PREFIX "HTTP_"
 #define HTTP_PREFIX_LEN (sizeof(HTTP_PREFIX) - 1)
 
+static VALUE global_rack_url_scheme;
 static VALUE global_request_method;
 static VALUE global_request_uri;
 static VALUE global_fragment;
@@ -37,8 +38,10 @@ static VALUE global_server_port;
 static VALUE global_server_protocol;
 static VALUE global_server_protocol_value;
 static VALUE global_http_host;
+static VALUE global_http_x_forwarded_proto;
 static VALUE global_port_80;
 static VALUE global_localhost;
+static VALUE global_http;
 
 /** Defines common length and error messages for input length validation. */
 #define DEF_MAX_LENGTH(N,length) const size_t MAX_##N##_LENGTH = length; const char *MAX_##N##_LENGTH_ERR = "HTTP element " # N  " is longer than the " # length " allowed length."
@@ -106,26 +109,12 @@ static struct common_field common_http_fields[] = {
 	f("USER_AGENT"),
 	f("VIA"),
 	f("X_FORWARDED_FOR"), /* common for proxies */
+	f("X_FORWARDED_PROTO"), /* common for proxies */
 	f("X_REAL_IP"), /* common for proxies */
 	f("WARNING")
 # undef f
 };
 
-/*
- * qsort(3) and bsearch(3) improve average performance slightly, but may
- * not be worth it for lack of portability to certain platforms...
- */
-#if defined(HAVE_QSORT_BSEARCH)
-/* sort by length, then by name if there's a tie */
-static int common_field_cmp(const void *a, const void *b)
-{
-  struct common_field *cfa = (struct common_field *)a;
-  struct common_field *cfb = (struct common_field *)b;
-  signed long diff = cfa->len - cfb->len;
-  return diff ? diff : memcmp(cfa->name, cfb->name, cfa->len);
-}
-#endif /* HAVE_QSORT_BSEARCH */
-
 /* this function is not performance-critical */
 static void init_common_fields(void)
 {
@@ -146,28 +135,10 @@ static void init_common_fields(void)
     cf->value = rb_obj_freeze(cf->value);
     rb_global_variable(&cf->value);
   }
-
-#if defined(HAVE_QSORT_BSEARCH)
-  qsort(common_http_fields,
-        ARRAY_SIZE(common_http_fields),
-        sizeof(struct common_field),
-        common_field_cmp);
-#endif /* HAVE_QSORT_BSEARCH */
 }
 
 static VALUE find_common_field_value(const char *field, size_t flen)
 {
-#if defined(HAVE_QSORT_BSEARCH)
-  struct common_field key;
-  struct common_field *found;
-  key.name = field;
-  key.len = (signed long)flen;
-  found = (struct common_field *)bsearch(&key, common_http_fields,
-                                         ARRAY_SIZE(common_http_fields),
-                                         sizeof(struct common_field),
-                                         common_field_cmp);
-  return found ? found->value : Qnil;
-#else /* !HAVE_QSORT_BSEARCH */
   int i;
   struct common_field *cf = common_http_fields;
   for(i = 0; i < ARRAY_SIZE(common_http_fields); i++, cf++) {
@@ -175,7 +146,6 @@ static VALUE find_common_field_value(const char *field, size_t flen)
       return cf->value;
   }
   return Qnil;
-#endif /* !HAVE_QSORT_BSEARCH */
 }
 
 static void http_field(void *data, const char *field,
@@ -282,6 +252,16 @@ static void header_done(void *data, const char *at, size_t length)
   VALUE temp = Qnil;
   char *colon = NULL;
 
+  /* set rack.url_scheme to "https" or "http", no others are allowed by Rack */
+  temp = rb_hash_aref(req, global_http_x_forwarded_proto);
+  switch (temp == Qnil ? 0 : RSTRING_LEN(temp)) {
+  case 5: if (!memcmp("https", RSTRING_PTR(temp), 5)) break;
+  case 4: if (!memcmp("http", RSTRING_PTR(temp), 4)) break;
+  default: temp = global_http;
+  }
+  rb_hash_aset(req, global_rack_url_scheme, temp);
+
+  /* set the SERVER_NAME and SERVER_PORT variables */
   if((temp = rb_hash_aref(req, global_http_host)) != Qnil) {
     colon = memchr(RSTRING_PTR(temp), ':', RSTRING_LEN(temp));
     if(colon != NULL) {
@@ -406,6 +386,7 @@ void Init_http11()
 
   mUnicorn = rb_define_module("Unicorn");
 
+  DEF_GLOBAL(rack_url_scheme, "rack.url_scheme");
   DEF_GLOBAL(request_method, "REQUEST_METHOD");
   DEF_GLOBAL(request_uri, "REQUEST_URI");
   DEF_GLOBAL(fragment, "FRAGMENT");
@@ -419,8 +400,10 @@ void Init_http11()
   DEF_GLOBAL(server_protocol, "SERVER_PROTOCOL");
   DEF_GLOBAL(server_protocol_value, "HTTP/1.1");
   DEF_GLOBAL(http_host, "HTTP_HOST");
+  DEF_GLOBAL(http_x_forwarded_proto, "HTTP_X_FORWARDED_PROTO");
   DEF_GLOBAL(port_80, "80");
   DEF_GLOBAL(localhost, "localhost");
+  DEF_GLOBAL(http, "http");
 
   eHttpParserError = rb_define_class_under(mUnicorn, "HttpParserError", rb_eIOError);
 

data/lib/unicorn.rb

@@ -26,14 +26,13 @@ module Unicorn
     attr_reader :logger
     include ::Unicorn::SocketHelper
 
+    SIG_QUEUE = []
     DEFAULT_START_CTX = {
       :argv => ARGV.map { |arg| arg.dup },
       # don't rely on Dir.pwd here since it's not symlink-aware, and
       # symlink dirs are the default with Capistrano...
       :cwd => `/bin/sh -c pwd`.chomp("\n"),
       :zero => $0.dup,
-      :environ => {}.merge!(ENV),
-      :umask => File.umask,
     }.freeze
 
     Worker = Struct.new(:nr, :tempfile) unless defined?(Worker)
@@ -53,12 +52,12 @@ module Unicorn
       @start_ctx = DEFAULT_START_CTX.dup
       @start_ctx.merge!(start_ctx) if start_ctx
       @app = app
-      @sig_queue = []
       @master_pid = $$
       @workers = Hash.new
       @io_purgatory = [] # prevents IO objects in here from being GC-ed
       @request = @rd_sig = @wr_sig = nil
       @reexec_pid = 0
+      @init_listeners = options[:listeners] ? options[:listeners].dup : []
       @config = Configurator.new(options.merge(:use_defaults => true))
       @listener_opts = {}
       @config.commit!(self, :skip => [:listeners, :pid])
@@ -73,9 +72,9 @@ module Unicorn
       # before they become UNIXServer or TCPServer
       inherited = ENV['UNICORN_FD'].to_s.split(/,/).map do |fd|
         io = Socket.for_fd(fd.to_i)
-        set_server_sockopt(io)
+        set_server_sockopt(io, @listener_opts[sock_name(io)])
         @io_purgatory << io
-        logger.info "inherited: #{io} fd=#{fd} addr=#{sock_name(io)}"
+        logger.info "inherited addr=#{sock_name(io)} fd=#{fd}"
         server_cast(io)
       end
 
@@ -104,15 +103,27 @@ module Unicorn
     # replaces current listener set with +listeners+.  This will
     # close the socket if it will not exist in the new listener set
     def listeners=(listeners)
-      cur_names = listener_names
+      cur_names, dead_names = [], []
+      listener_names.each do |name|
+        if "/" == name[0..0]
+          # mark unlinked sockets as dead so we can rebind them
+          (File.socket?(name) ? cur_names : dead_names) << name
+        else
+          cur_names << name
+        end
+      end
       set_names = listener_names(listeners)
-      dead_names = cur_names - set_names
+      dead_names += cur_names - set_names
+      dead_names.uniq!
 
       @listeners.delete_if do |io|
         if dead_names.include?(sock_name(io))
-          @io_purgatory.delete_if { |pio| pio.fileno == io.fileno }
-          true
+          @io_purgatory.delete_if do |pio|
+            pio.fileno == io.fileno && (pio.close rescue nil).nil? # true
+          end
+          (io.close rescue nil).nil? # true
         else
+          set_server_sockopt(io, @listener_opts[sock_name(io)])
           false
         end
       end
@@ -141,12 +152,11 @@ module Unicorn
       return if String === address && listener_names.include?(address)
 
       if io = bind_listen(address, opt)
-        if Socket == io.class
+        unless TCPServer === io || UNIXServer === io
           @io_purgatory << io
           io = server_cast(io)
         end
-        logger.info "#{io} listening on PID:#{$$} " \
-                    "fd=#{io.fileno} addr=#{sock_name(io)}"
+        logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}"
         @listeners << io
       else
         logger.error "adding listener failed addr=#{address} (in use)"
@@ -172,7 +182,7 @@ module Unicorn
       begin
         loop do
           reap_all_workers
-          case (mode = @sig_queue.shift)
+          case (mode = SIG_QUEUE.shift)
           when nil
             murder_lazy_workers
             spawn_missing_workers if respawn
@@ -219,7 +229,7 @@ module Unicorn
         retry
       end
       stop # gracefully shutdown all workers on our way out
-      logger.info "master PID:#{$$} join complete"
+      logger.info "master complete"
       unlink_pid_safe(@pid) if @pid
     end
 
@@ -247,11 +257,11 @@ module Unicorn
     # defer a signal for later processing in #join (master process)
     def trap_deferred(signal)
       trap(signal) do |sig_nr|
-        if @sig_queue.size < 5
-          @sig_queue << signal
+        if SIG_QUEUE.size < 5
+          SIG_QUEUE << signal
           awaken_master
         else
-          logger.error "ignoring SIG#{signal}, queue=#{@sig_queue.inspect}"
+          logger.error "ignoring SIG#{signal}, queue=#{SIG_QUEUE.inspect}"
         end
       end
     end
@@ -326,10 +336,8 @@ module Unicorn
       end
 
       @reexec_pid = fork do
-        ENV.replace(@start_ctx[:environ])
         listener_fds = @listeners.map { |sock| sock.fileno }
         ENV['UNICORN_FD'] = listener_fds.join(',')
-        File.umask(@start_ctx[:umask])
         Dir.chdir(@start_ctx[:cwd])
         cmd = [ @start_ctx[:zero] ] + @start_ctx[:argv]
 
@@ -375,14 +383,13 @@ module Unicorn
           Dir.chdir(@start_ctx[:cwd])
         rescue Errno::ENOENT => err
           logger.fatal "#{err.inspect} (#{@start_ctx[:cwd]})"
-          @sig_queue << :QUIT # forcibly emulate SIGQUIT
+          SIG_QUEUE << :QUIT # forcibly emulate SIGQUIT
           return
         end
         tempfile = Tempfile.new('') # as short as possible to save dir space
         tempfile.unlink # don't allow other processes to find or see it
-        tempfile.sync = true
         worker = Worker.new(worker_nr, tempfile)
-        @before_fork.call(self, worker.nr)
+        @before_fork.call(self, worker)
         pid = fork { worker_loop(worker) }
         @workers[pid] = worker
       end
@@ -391,11 +398,9 @@ module Unicorn
     # once a client is accepted, it is processed in its entirety here
     # in 3 easy steps: read request, call app, write app response
     def process_client(client)
-      client.nonblock = false
-      set_client_sockopt(client) if TCPSocket === client
-      env = @request.read(client)
-      app_response = @app.call(env)
-      HttpResponse.write(client, app_response)
+      # one syscall less than "client.nonblock = false":
+      client.fcntl(Fcntl::F_SETFL, File::RDWR)
+      HttpResponse.write(client, @app.call(@request.read(client)))
     # if we get any error, try to write something back to the client
     # assuming we haven't closed the socket, but don't get hung up
     # if the socket is already closed or broken.  We'll always ensure
@@ -423,23 +428,19 @@ module Unicorn
     # traps for USR1, USR2, and HUP may be set in the @after_fork Proc
     # by the user.
     def init_worker_process(worker)
-      build_app! unless @preload_app
-      @sig_queue.clear
-      QUEUE_SIGS.each { |sig| trap(sig, 'IGNORE') }
+      QUEUE_SIGS.each { |sig| trap(sig, 'DEFAULT') }
       trap(:CHLD, 'DEFAULT')
-
+      SIG_QUEUE.clear
       proc_name "worker[#{worker.nr}]"
       @rd_sig.close if @rd_sig
       @wr_sig.close if @wr_sig
       @workers.values.each { |other| other.tempfile.close rescue nil }
-      @workers.clear
-      @start_ctx.clear
       @start_ctx = @workers = @rd_sig = @wr_sig = nil
       @listeners.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
-      ENV.delete('UNICORN_FD')
-      @after_fork.call(self, worker.nr) if @after_fork
       worker.tempfile.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
+      @after_fork.call(self, worker) if @after_fork # can drop perms
       @request = HttpRequest.new(logger)
+      build_app! unless @preload_app
     end
 
     # runs inside each forked worker, this sits around and waits
@@ -452,9 +453,8 @@ module Unicorn
       alive = true
       ready = @listeners
       client = nil
-      [:TERM, :INT].each { |sig| trap(sig) { exit(0) } } # instant shutdown
       trap(:QUIT) do
-        alive = false
+        alive = false # graceful shutdown
         @listeners.each { |sock| sock.close rescue nil } # break IO.select
       end
       reopen_logs, (rd, wr) = false, IO.pipe
@@ -569,6 +569,7 @@ module Unicorn
     def load_config!
       begin
         logger.info "reloading config_file=#{@config.config_file}"
+        @config[:listeners].replace(@init_listeners)
         @config.reload
         @config.commit!(self)
         kill_each_worker(:QUIT)