swiftiply 0.5.0

data/README

@@ -0,0 +1,126 @@
+Swiftiply v. 0.5.0  (http://swiftiply.swiftcore.org)
+
+Swiftiply is a backend agnostic clustering proxy for web applications that is
+specifically designed to support HTTP traffic from web frameworks. Unlike Pen
+(http://siag.nu/pen/), Swiftiply is not intended as a general purpose load
+balancer for tcp protocols and unlike HAProxy (http://haproxy.1wt.eu/), it is
+not a highly configurable general purpose proxy overflowing with features.
+
+What it is, though, is a very fast, narrowly targetted clustering proxy.
+In back to back comparisons of Swiftiply to HAProxy, Swiftiply reliably
+outperforms HAProxy (tested using IOWA, Rails, Merb, and Ramaze backend
+processes running Mongrel).
+
+Swiftiply works differently from a traditional proxy.  In Swiftiply, the
+backend processes are clients of the Swiftiply server -- they make persistent
+socket connections to Swiftiply.  One of the major advantages to this
+architecture is that it allows one to start or stop backend processes at will,
+with no configuration of the proxy.  The obvious disadvantage is that this is
+not behavior that backends typically expect.
+
+Because Mongrel is the preferred deployment method for most Ruby frameworks,
+Swiftiply includes a version of Mongrel (found in swiftcore/swiftiplied_mongrel.rb)
+that has been modified to work as a swiftiply client.  This should be
+transparent to any existing Mongrel handlers, allowing them all to with
+Swiftiply.
+
+In addition, as an offshoot of the swiftiplied_mongrel, there is a second
+version that is available.  This other version is found in
+swiftcore/evented_mongrel.rb; it is a version of Mongrel that has its network
+traffic handled by EventMachine, creating a Mongrel that runs in an event
+based mode instead of a threaded mode.  For many applications, running in an
+event based mode will give better throughput than running in a threaded mode,
+especially when there are concurrent requests coming in.
+
+This is because the event based operation handles requests efficiently, on
+a first come, first served basis, without the overhead of threads.  For the
+typical Rails application, this means that request handling may be slightly
+faster than the threaded Mongrel for single, non-concurrent requests.  When
+there are concurrent requests, though, the differential increases quickly.
+
+
+FRAMEWORK SUPPORT
+
+
+Swiftcore IOWA
+
+IOWA has built in support for running in evented and clustered modes.
+
+
+Rails
+
+Swiftiply provides a _REPLACEMENT_ to mongrel_rails that, throught the use
+of an environment variable, can be told to run in either the evented mode or
+the swiftiplied mode.
+
+To run a Rails app in evented mode, set the EVENT environment variable.  On
+a unixlike system:
+
+env EVENT=1 mongrel_rails
+
+will do it.
+
+To run in swiftiplied mode:
+
+env SWIFTIPLY=1 mongrel_rails
+
+Because Swiftiply backends connect to the Swiftiply server, they all connect
+on the same port.  This is important.  Each of the backends runs against the
+same port.  To make it easier to start multiple Rails backends, a helper
+script, swiftiply_mongrel_rails, is provided.  It is just a light wrapper
+around mongrel_rails that will let one start N backends, with proper pid
+files, and stop them.
+
+
+Merb
+
+The merb source (trunk only, at this point), has Swiftiply support that works
+just like the Rails support, built in.
+
+
+Ramaze
+
+A couple adapters for Ramaze are included, to allow Ramaze to run with either
+the evented or the swiftiplied mongrels.  They are installed into
+
+ramaze/adapter/evented_mongrel.rb
+ramaze/adapter/swiftiplied_mongrel.rb
+
+
+Other Frameworks
+
+Swiftiply has been tested with Camping and Nitro, as well.  Direct support for
+them is not yet bundled, but will be in an upcoming release.
+
+
+CONFIGURATION
+
+Swiftiply takes a single configuration file which defines for it where it
+should listen for incoming connections, whether it should daemonize itself,
+and then provides a map of incoming domain names and the address/port to
+proxy that traffic to.  That outgoing address/port is where the backends for
+that site will connect to.
+
+Here's an example:
+
+cluster_address: swiftcore.org
+cluster_port: 80
+daemonize: true
+map:
+  - incoming:
+    - swiftcore.org
+    - www.swiftcore.org
+    outgoing: 127.0.0.1:30000
+    default: true
+  - incoming: iowa.swiftcore.org
+    outgoing: 127.0.0.1:30010
+  - incoming: analogger.swiftcore.org
+    outgoing: 127.0.0.1:30020
+  - incoming:
+    - swiftiply.com
+    - www.swiftiply.com
+    - swiftiply.swiftcore.org
+    outgoing: 127.0.0.1:30030
+
+
+

data/bin/mongrel_rails

@@ -0,0 +1,254 @@
+#!/usr/bin/env ruby
+# Copyright (c) 2005 Zed A. Shaw 
+# You can redistribute it and/or modify it under the same terms as Ruby.
+#
+# Additional work donated by contributors.  See http://mongrel.rubyforge.org/attributions.html 
+# for more information.
+
+require 'rubygems'
+require 'yaml'
+if ENV['EVENT']
+  require 'swiftcore/evented_mongrel' 
+  puts "Using Evented Mongrel"
+elsif ENV['SWIFT']
+  require 'swiftcore/swiftiplied_mongrel' 
+  puts "Using Evented Mongrel"  
+else
+ require 'mongrel'
+end
+require 'mongrel/rails'
+require 'etc'
+require 'cgi_multipart_eof_fix' rescue nil
+
+module Mongrel
+  class Start < GemPlugin::Plugin "/commands"
+    include Mongrel::Command::Base
+
+    def configure
+      options [
+        ["-e", "--environment ENV", "Rails environment to run as", :@environment, ENV['RAILS_ENV'] || "development"],
+        ["-d", "--daemonize", "Run daemonized in the background", :@daemon, false],
+        ['-p', '--port PORT', "Which port to bind to", :@port, 3000],
+        ['-a', '--address ADDR', "Address to bind to", :@address, "0.0.0.0"],
+        ['-l', '--log FILE', "Where to write log messages", :@log_file, "log/mongrel.log"],
+        ['-P', '--pid FILE', "Where to write the PID", :@pid_file, "log/mongrel.pid"],
+        ['-n', '--num-procs INT', "Number of processors active before clients denied", :@num_procs, 1024],
+        ['-t', '--timeout TIME', "Timeout all requests after 100th seconds time", :@timeout, 0],
+        ['-m', '--mime PATH', "A YAML file that lists additional MIME types", :@mime_map, nil],
+        ['-c', '--chdir PATH', "Change to dir before starting (will be expanded)", :@cwd, Dir.pwd],
+        ['-r', '--root PATH', "Set the document root (default 'public')", :@docroot, "public"],
+        ['-B', '--debug', "Enable debugging mode", :@debug, false],
+        ['-C', '--config PATH', "Use a config file", :@config_file, nil],
+        ['-S', '--script PATH', "Load the given file as an extra config script", :@config_script, nil],
+        ['-G', '--generate PATH', "Generate a config file for use with -C", :@generate, nil],
+        ['', '--user USER', "User to run as", :@user, nil],
+        ['', '--group GROUP', "Group to run as", :@group, nil],
+        ['', '--prefix PATH', "URL prefix for Rails app", :@prefix, nil]
+      ]
+    end
+
+    def validate
+      @cwd = File.expand_path(@cwd)
+      valid_dir? @cwd, "Invalid path to change to during daemon mode: #@cwd"
+
+      # Change there to start, then we'll have to come back after daemonize
+      Dir.chdir(@cwd)
+
+      valid?(@prefix[0].chr == "/" && @prefix[-1].chr != "/", "Prefix must begin with / and not end in /") if @prefix
+      valid_dir? File.dirname(@log_file), "Path to log file not valid: #@log_file"
+      valid_dir? File.dirname(@pid_file), "Path to pid file not valid: #@pid_file"
+      valid_dir? @docroot, "Path to docroot not valid: #@docroot"
+      valid_exists? @mime_map, "MIME mapping file does not exist: #@mime_map" if @mime_map
+      valid_exists? @config_file, "Config file not there: #@config_file" if @config_file
+      valid_dir? File.dirname(@generate), "Problem accessing directory to #@generate" if @generate
+      valid_user? @user if @user
+      valid_group? @group if @group
+
+      return @valid
+    end
+
+    def run
+      # Config file settings will override command line settings
+      settings = { :host => @address,  :port => @port, :cwd => @cwd,
+        :log_file => @log_file, :pid_file => @pid_file, :environment => @environment,
+        :docroot => @docroot, :mime_map => @mime_map, :daemon => @daemon,
+        :debug => @debug, :includes => ["mongrel"], :config_script => @config_script,
+        :num_processors => @num_procs, :timeout => @timeout,
+        :user => @user, :group => @group, :prefix => @prefix, :config_file => @config_file
+      }
+
+      if @generate
+        STDERR.puts "** Writing config to \"#@generate\"."
+        open(@generate, "w") {|f| f.write(settings.to_yaml) }
+        STDERR.puts "** Finished.  Run \"mongrel_rails -C #@generate\" to use the config file."
+        exit 0
+      end
+
+      if @config_file
+        settings.merge! YAML.load_file(@config_file)
+        STDERR.puts "** Loading settings from #{@config_file} (they override command line)." unless settings[:daemon]
+      end
+
+      config = Mongrel::Rails::RailsConfigurator.new(settings) do
+        if defaults[:daemon]
+          if File.exist? defaults[:pid_file]
+            log "!!! PID file #{defaults[:pid_file]} already exists.  Mongrel could be running already.  Check your #{defaults[:log_file]} for errors."
+            log "!!! Exiting with error.  You must stop mongrel and clear the .pid before I'll attempt a start."
+            exit 1
+          end
+
+          daemonize
+          log "Daemonized, any open files are closed.  Look at #{defaults[:pid_file]} and #{defaults[:log_file]} for info."
+          log "Settings loaded from #{@config_file} (they override command line)." if @config_file
+        end
+
+        log "Starting Mongrel listening at #{defaults[:host]}:#{defaults[:port]}"
+
+        listener do
+          mime = {}
+          if defaults[:mime_map]
+            log "Loading additional MIME types from #{defaults[:mime_map]}"
+            mime = load_mime_map(defaults[:mime_map], mime)
+          end
+
+          if defaults[:debug]
+            log "Installing debugging prefixed filters. Look in log/mongrel_debug for the files."
+            debug "/"
+          end
+
+          log "Starting Rails with #{defaults[:environment]} environment..."
+          log "Mounting Rails at #{defaults[:prefix]}..." if defaults[:prefix]
+          uri defaults[:prefix] || "/", :handler => rails(:mime => mime, :prefix => defaults[:prefix])
+          log "Rails loaded."
+
+          log "Loading any Rails specific GemPlugins"
+          load_plugins
+
+          if defaults[:config_script]
+            log "Loading #{defaults[:config_script]} external config script"
+            run_config(defaults[:config_script])
+          end
+
+          setup_rails_signals
+        end
+      end
+
+      config.run
+      config.log "Mongrel available at #{settings[:host]}:#{settings[:port]}"
+
+      if config.defaults[:daemon]
+        config.write_pid_file
+      else
+        config.log "Use CTRL-C to stop." 
+      end
+
+      config.join
+
+      if config.needs_restart
+        if RUBY_PLATFORM !~ /mswin/
+          cmd = "ruby #{__FILE__} start #{original_args.join(' ')}"
+          config.log "Restarting with arguments:  #{cmd}"
+          config.stop
+          config.remove_pid_file
+
+          if config.defaults[:daemon]
+            system cmd
+          else
+            STDERR.puts "Can't restart unless in daemon mode."
+            exit 1
+          end
+        else
+          config.log "Win32 does not support restarts. Exiting."
+        end
+      end
+    end
+  end
+
+  def Mongrel::send_signal(signal, pid_file)
+    pid = open(pid_file).read.to_i
+    print "Sending #{signal} to Mongrel at PID #{pid}..."
+    begin
+      Process.kill(signal, pid)
+    rescue Errno::ESRCH
+      puts "Process does not exist.  Not running."
+    end
+
+    puts "Done."
+  end
+
+
+  class Stop < GemPlugin::Plugin "/commands"
+    include Mongrel::Command::Base
+
+    def configure 
+      options [ 
+        ['-c', '--chdir PATH', "Change to dir before starting (will be expanded).", :@cwd, "."],
+        ['-f', '--force', "Force the shutdown (kill -9).", :@force, false],
+        ['-w', '--wait SECONDS', "Wait SECONDS before forcing shutdown", :@wait, "0"], 
+        ['-P', '--pid FILE', "Where the PID file is located.", :@pid_file, "log/mongrel.pid"]
+      ]
+    end
+
+    def validate
+      @cwd = File.expand_path(@cwd)
+      valid_dir? @cwd, "Invalid path to change to during daemon mode: #@cwd"
+
+      Dir.chdir @cwd
+
+      valid_exists? @pid_file, "PID file #@pid_file does not exist.  Not running?"
+      return @valid
+    end
+
+    def run
+      if @force
+        @wait.to_i.times do |waiting|
+          exit(0) if not File.exist? @pid_file
+          sleep 1
+        end
+
+        Mongrel::send_signal("KILL", @pid_file) if File.exist? @pid_file
+      else
+        Mongrel::send_signal("TERM", @pid_file)
+      end
+    end
+  end
+
+
+  class Restart < GemPlugin::Plugin "/commands"
+    include Mongrel::Command::Base
+
+    def configure 
+      options [ 
+        ['-c', '--chdir PATH', "Change to dir before starting (will be expanded)", :@cwd, '.'],
+        ['-s', '--soft', "Do a soft restart rather than a process exit restart", :@soft, false],
+        ['-P', '--pid FILE', "Where the PID file is located", :@pid_file, "log/mongrel.pid"]
+      ]
+    end
+
+    def validate
+      @cwd = File.expand_path(@cwd)
+      valid_dir? @cwd, "Invalid path to change to during daemon mode: #@cwd"
+
+      Dir.chdir @cwd
+
+      valid_exists? @pid_file, "PID file #@pid_file does not exist.  Not running?"
+      return @valid
+    end
+
+    def run
+      if @soft
+        Mongrel::send_signal("HUP", @pid_file)
+      else
+        Mongrel::send_signal("USR2", @pid_file)
+      end
+    end
+  end
+end
+
+
+GemPlugin::Manager.instance.load "mongrel" => GemPlugin::INCLUDE, "rails" => GemPlugin::EXCLUDE
+
+
+if not Mongrel::Command::Registry.instance.run ARGV
+  exit 1
+end

data/bin/swiftiply

@@ -0,0 +1,136 @@
+#!ruby
+
+require 'optparse'
+require 'yaml'
+
+begin
+	load_attempted ||= false
+	require 'swiftcore/Swiftiply'
+rescue LoadError => e
+	unless load_attempted
+		load_attempted = true
+		require 'rubygems'
+		retry
+	end
+	raise e
+end
+
+module Swiftcore
+	class SwiftiplyExec
+		Ccluster_address = 'cluster_address'.freeze
+		Ccluster_port = 'cluster_port'.freeze
+		Cbackend_address = 'backend_address'.freeze
+		Cbackend_port = 'backend_port'.freeze
+		Cmap = 'map'.freeze
+		Cincoming = 'incoming'.freeze
+		Coutgoing = 'outgoing'.freeze
+		Ckeepalive = 'keepalive'.freeze
+		Cdaemonize = 'daemonize'.freeze
+		Curl = 'url'.freeze
+		Chost = 'host'.freeze
+		Cport = 'port'.freeze
+		Ctimeout = 'timeout'.freeze
+
+#####
+#
+# --cluster-address
+# --cluster-port
+# --config-file -c (default swiftiply.cnf)
+# --daemonize -d
+# 
+# Config file format (YAML):
+# 
+# cluster_address:
+# cluster_port:
+# daemonize:
+# map:
+#   - incoming:
+#     - 127.0.0.1:8080
+#     - foo.bar.com:8090
+#     url:
+#			keepalive:
+#			outgoing:
+# 
+#####
+		def self.parse_options(config = {})
+			OptionParser.new do |opts|
+				opts.banner = 'Usage: swiftiply.rb [options]'
+				opts.separator ''
+				opts.on('-c','--config CONFFILE',"The configuration file to read.") do |conf|
+					config = YAML.load(File.open(conf))
+					postprocess_config_load(config)
+				end
+				opts.on('--cluster-address [ADDRESS]',String,'The hostname/IP address that swiftiply will listen for connections on.') do |address|
+					config[Ccluster_address] = address
+				end
+				opts.on('--cluster-port [PORT]',Integer,'The port that swiftiply will listen for connections on.') do |port|
+					config[Ccluster_port] = port
+				end
+				opts.on('--backend-address [ADDRESS]',String,'The hostname/IP address that swiftiply will listen for backend connections on.') do |address|
+					config[Cbackend_address] = address
+				end
+				opts.on('--backend-port [PORT]',Integer,'The port that swiftiply will listen for backend connections on.') do |port|
+					config[Cbackend_port] = port
+				end
+				opts.on('-d','--daemonize [YN]',[:y,:yes,:n,:no],'Whether swiftiply should put itself into the background.') do |yn|
+					config[Cdaemonize] = yn.to_s =~ /^y/i
+				end
+				opts.on('-t','--timeout [SECONDS]',Integer,'The server unavailable timeout.  Defaults to 3 seconds.') do |timeout|
+					config[Ctimeout] = timeout
+				end
+				opts.on('-p','--print-config','Print the full configuration.') do
+					verify_config(config)
+					require 'pp'
+					pp config
+					exit
+				end
+
+			end.parse!
+			puts("Configuration failed validation; exiting.") && exit unless verify_config(config)
+			config
+		end
+
+		def self.daemonize
+			if (child_pid = fork)
+				puts "PID #{child_pid}"
+				exit!
+			end
+
+			Process.setsid
+
+		rescue Exception
+			puts "Platform (#{RUBY_PLATFORM}) does not appear to support fork/setsid; skipping"
+		end
+
+		def self.postprocess_config_load(config)
+			config[Cmap] = [] unless Array === config[Cmap]
+			config[Ctimeout] ||= 3
+			config[Cmap].each do |m|
+				m[Ckeepalive] = true unless m.has_key?(Ckeepalive)
+				m[Ckeepalive] = !!m[Ckeepalive]
+				m[Coutgoing] = [m[Coutgoing]] unless Array === m[Coutgoing]
+				m[Cincoming] = [m[Cincoming]] unless Array === m[Cincoming]
+			end
+		end
+
+		def self.verify_config(config)
+			if config[Cbackend_address] and config[Cbackend_port]
+				config[Cmap] << {Cincoming => nil, Curl => nil, Coutgoing => "#{config[Cbackend_address]}:#{config[Cbackend_port]}", Ckeepalive => true}
+			end
+			true
+		end
+
+		def self.run
+			config = parse_options
+			daemonize if config[Cdaemonize]
+			Swiftcore::Swiftiply.run(config)
+		end
+
+		def self.on_windows?
+			return @on_windows unless @on_windows.nil?
+      @on_windows = !![/mswin/i, /cygwin/i, /mingw/i, /bccwin/i, /wince/i].find {|p| RUBY_PLATFORM =~ p}
+		end
+	end
+end
+
+Swiftcore::SwiftiplyExec.run