pipemaster 0.3.1

data/Gemfile

@@ -0,0 +1,5 @@
+source "http://gemcutter.org"
+gem "gemcutter"
+gem "rake"
+gem "unicorn", "0.96.1"
+gem "yard"

data/LICENSE

@@ -0,0 +1,55 @@
+Pipemaster is copyrighted free software by all contributors, see logs in
+revision control for names and email addresses of all of them.  You can
+redistribute it and/or modify it under either the terms of the
+{GPL2}[http://www.gnu.org/licenses/gpl-2.0.txt] (see link:COPYING) or
+the conditions below:
+
+  1. You may make and give away verbatim copies of the source form of the
+     software without restriction, provided that you duplicate all of the
+     original copyright notices and associated disclaimers.
+
+  2. You may modify your copy of the software in any way, provided that
+     you do at least ONE of the following:
+
+       a) place your modifications in the Public Domain or otherwise make them
+       Freely Available, such as by posting said modifications to Usenet or an
+       equivalent medium, or by allowing the author to include your
+       modifications in the software.
+
+       b) use the modified software only within your corporation or
+          organization.
+
+       c) rename any non-standard executables so the names do not conflict with
+       standard executables, which must also be provided.
+
+       d) make other distribution arrangements with the author.
+
+  3. You may distribute the software in object code or executable
+     form, provided that you do at least ONE of the following:
+
+       a) distribute the executables and library files of the software,
+       together with instructions (in the manual page or equivalent) on where
+       to get the original distribution.
+
+       b) accompany the distribution with the machine-readable source of the
+       software.
+
+       c) give non-standard executables non-standard names, with
+          instructions on where to get the original software distribution.
+
+       d) make other distribution arrangements with the author.
+
+  4. You may modify and include the part of the software into any other
+     software (possibly commercial).  But some files in the distribution
+     are not written by the author, so that they are not under this terms.
+
+  5. The scripts and library files supplied as input to or produced as
+     output from the software do not automatically fall under the
+     copyright of the software, but belong to whomever generated them,
+     and may be sold commercially, and may be aggregated with this
+     software.
+
+  6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
+     IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+     WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+     PURPOSE.

data/README.rdoc

@@ -0,0 +1,61 @@
+You pipe from here, to a process that runs over there.  Pipemaster forks,
+redirects and runs your command.
+
+== Why
+
+I've got a short task I want to perform.  To validate an incoming email, parse
+the message and store the results in the database(*).  I setup Postfix to pipe
+incoming emails into a Ruby script.
+
+Ruby processes are fairly cheap, until you get into loading the mail library,
+the database library, the ORM, the application logic, the ... you get the
+picture.
+
+I use Pipemaster to fire up the main process once, and the Pipemaster client to
+run commands on that server.  Still have to make sure the code is light and
+fast, but I did eliminate the significant initialization overhead.
+
+As you can guess, Pipemaster supports piping input and output streams, and uses
+forking (sorry Windows; for JRuby see Nailgun).
+
+* Processing emails as they come allows the application to reject unauthorized
+  senders immediately by replying with an SMTP code.  The alternative, accepting
+  the email and later on sending a bounce, leads to backscatter (see
+  http://en.wikipedia.org/wiki/Backscatter_(e-mail)).
+
+
+== Using Pipemaster
+
+Step 1:  Create a Pipemaster file.  For example:
+
+  #!highlight/ruby
+  command :echo do |*args|
+    first, rest = $stdin.read.split
+    $stdout << [first, args, rest].flatten.join(" ")
+  end
+
+Step 2:  Fire up the Pipemaster server:
+
+  $ pipemaster --server
+  I, [2010-02-18T12:57:57.739230 #5460]  INFO -- : master process ready
+  I, [2010-02-18T12:57:57.739606 #5460]  INFO -- : listening on addr=127.0.0.1:7887 fd=3
+  
+Step 3:  For a new shell, execute a command:
+
+  $ echo "Stand down!" | ruby -Ilib bin/pipemaster echo upside
+  Stand upside down!
+
+
+== License
+
+Pipemaster is copyright of Assaf Arkin.  It is based on the awesome Unicorn Web
+server and therefore uses the same license.
+
+Unicorn is copyright 2009 by all contributors (see logs in git).
+It is based on Mongrel and carries the same license.
+
+Mongrel is copyright 2007 Zed A. Shaw and contributors. It is licensed
+under the Ruby license and the GPL2. See the included LICENSE file for
+details.
+
+Pipemaster is 100% Free Software.

data/Rakefile

@@ -0,0 +1,51 @@
+require "rake/testtask"
+
+spec = Gem::Specification.load(File.expand_path("pipemaster.gemspec", File.dirname(__FILE__)))
+
+desc "Push new release to gemcutter and git tag"
+task :push do
+  sh "git push"
+  puts "Tagging version #{spec.version} .."
+  sh "git tag v#{spec.version}"
+  sh "git push --tag"
+  puts "Building and pushing gem .."
+  sh "gem build #{spec.name}.gemspec"
+  sh "gem push #{spec.name}-#{spec.version}.gem"
+end
+
+desc "Install #{spec.name} locally"
+task :install do
+  sh "gem build #{spec.name}.gemspec"
+  sudo = "sudo" unless File.writable?( Gem::ConfigMap[:bindir])
+  sh "#{sudo} gem install #{spec.name}-#{spec.version}.gem"
+end
+
+
+task :default=>:test
+desc "Run all tests using Redis mock (also default task)"
+Rake::TestTask.new do |task|
+  task.test_files = FileList['test/unit/test_*.rb']
+  if Rake.application.options.trace
+    #task.warning = true
+    task.verbose = true
+  elsif Rake.application.options.silent
+    task.ruby_opts << "-W0"
+  else
+    task.verbose = true
+  end
+end
+
+
+begin
+  require "yard"
+  YARD::Rake::YardocTask.new(:yardoc) do |task|
+    task.files  = FileList["lib/**/*.rb"]
+    task.options = "--output", "html", "--title", "Pipemaster #{spec.version}", "--main", "README.rdoc", "--files", "CHANGELOG"
+  end
+rescue LoadError
+end
+
+desc "Create documentation in html directory"
+task :docs=>[:yardoc]
+desc "Remove temporary files and directories"
+task(:clobber) { rm_rf "html" }

data/bin/pipemaster

@@ -0,0 +1,100 @@
+#!/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
+# -*- encoding: binary -*-
+require "pipemaster"
+require "optparse"
+
+ENV["PIPE_ENV"] ||= "development"
+server = false
+daemonize = false
+listeners = []
+options = { :listeners => listeners }
+
+opts = OptionParser.new("", 24, '  ') do |opts|
+  opts.banner = "Usage: #{File.basename($0)}\n" \
+                "[ruby options] [pipemaster options] command [args]\n" \
+                "[ruby options] [pipemaster options] --server [Pipefile]"
+
+  opts.separator "Ruby options:"
+
+  lineno = 1
+  opts.on("-e", "--eval LINE", "evaluate a LINE of code") do |line|
+    eval line, TOPLEVEL_BINDING, "-e", lineno
+    lineno += 1
+  end
+
+  opts.on("-d", "--debug", "set debugging flags (set $DEBUG to true)") do
+    $DEBUG = true
+  end
+
+  opts.on("-w", "--warn", "turn warnings on for your script") do
+    $-w = true
+  end
+
+  opts.on("-I", "--include PATH",
+          "specify $LOAD_PATH (may be used more than once)") do |path|
+    $LOAD_PATH.unshift(*path.split(/:/))
+  end
+
+  opts.on("-r", "--require LIBRARY",
+          "require the library, before executing your script") do |library|
+    require library
+  end
+
+  opts.separator "Pipemaster options:"
+
+  opts.on("-s", "--socket {HOST:PORT|PATH}",
+          "communicate on HOST:PORT or PATH",
+          "for server, this may be specified multiple times",
+          "(default: #{Pipemaster::DEFAULT_LISTEN})") do |address|
+    listeners << address
+  end
+
+  opts.on("-E", "--env ENVIRONMENT",
+          "use ENVIRONMENT for defaults (default: development)") do |e|
+    ENV["PIPE_ENV"] = e
+  end
+
+  opts.on("-S", "--server", "run as server (default: client)") do |s|
+    server = s ? true : false
+  end
+
+  opts.on("-D", "--daemonize", "run daemonized in the background") do |d|
+    daemonize = d ? true : false
+  end
+
+  opts.separator "Common options:"
+
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts.to_s.gsub(/^.*DEPRECATED.*$/s, '')
+    exit
+  end
+
+  opts.on_tail("-v", "--version", "Show version") do
+    puts "Pipemaster v#{Pipemaster::VERSION}"
+    exit
+  end
+
+  opts.parse! ARGV
+end
+
+if server
+
+  gem "unicorn"
+  require "pipemaster/server"
+  require "unicorn/launcher"
+  config = ARGV[0] || "Pipefile"
+  abort "configuration file #{config} not found" unless File.exist?(config)
+  options[:config_file] = config
+  options[:working_directory] = File.dirname(config)
+  Unicorn::Launcher.daemonize!(options) if daemonize
+  Pipemaster.run(options)
+
+else
+
+  require "pipemaster/client"
+  client = Pipemaster::Client.new(listeners.first)
+  client.input = $stdin unless $stdin.isatty
+  client.output = $stdout
+  exit client.request(*ARGV)
+
+end

data/lib/pipemaster.rb

@@ -0,0 +1,8 @@
+module Pipemaster
+  # Version number.
+  VERSION = Gem::Specification.load(File.expand_path("../pipemaster.gemspec", File.dirname(__FILE__))).version.to_s
+
+  DEFAULT_HOST = "127.0.0.1"
+  DEFAULT_PORT = 7887
+  DEFAULT_LISTEN = "#{DEFAULT_HOST}:#{DEFAULT_PORT}"
+end

data/lib/pipemaster/client.rb

@@ -0,0 +1,121 @@
+require "socket"
+
+module Pipemaster
+  # Pipemaster client.  Use this to send commands to the Pipemaster server.
+  #
+  # For example:
+  #   c = Pipemaster::Client.new
+  #   c.request "motd"
+  #   puts c.output.string
+  #   => "Toilet out of order please use floor below."
+  #
+  #   c = Pipemaster::Client.new(9988)
+  #   c.input = File.open("image.png")
+  #   c.output = File.open("thumbnail.png", "w")
+  #   c.request "transform", "thumbnail"
+  class Client
+    BUFFER_SIZE = 16 * 1024
+
+    # Address can be "x.x.x.x:port", ":port" or UNIX socket file name.
+    def initialize(address)
+      @address = expand_addr(address || DEFAULT_LISTEN)
+    end
+
+    # Set this to supply an input stream (for commands that read from $stdin).
+    attr_accessor :input
+
+    # Set this to supply an output stream, or read the output (defaults to
+    # StringIO).
+    attr_accessor :output
+
+    # Make a request.  First argument is the command name.  All other arguments
+    # are optional.  Returns the exit code (usually 0).  Will raise IOError if
+    # it can't talk to the server, or the server closed the connection
+    # prematurely.
+    def request(command, *args)
+      # Connect and send arguments.
+      socket = connect(@address)
+      socket.sync = true
+      header = ([command] + args).join("\0")
+      socket << [header.size].pack("N") << header
+      socket.flush
+
+      @output ||= StringIO.new
+
+      # If there's input, stream it over to the socket, while streaming the
+      # response back.
+      inputbuf, stdoutbuf = "", ""
+      if @input
+        while selected = select([@input, socket])
+          begin
+            if selected.first.include?(@input)
+              @input.readpartial(BUFFER_SIZE, inputbuf)
+              socket.write inputbuf
+            elsif selected.first.include?(socket)
+              raise IOError, "Server closed socket" if socket.eof?
+              @output.write stdoutbuf if stdoutbuf
+              stdoutbuf = socket.readpartial(BUFFER_SIZE)
+            end
+          rescue EOFError
+            break
+          end
+        end
+      end
+      socket.close_write # tell other side there's no more input
+
+      # Read remaining response and stream to output.  Remember that very last
+      # byte is return code.
+      while selected = select([socket])
+        break if socket.eof?
+        @output.write stdoutbuf if stdoutbuf
+        stdoutbuf = socket.readpartial(BUFFER_SIZE)
+      end
+      if stdoutbuf && stdoutbuf.size > 0
+        status = stdoutbuf[-1]
+        @output.write stdoutbuf[0..-2]
+        return status.ord
+      else
+        raise IOError, "Server closed socket" if socket.eof?
+      end
+    ensure
+      socket.close rescue nil
+    end
+
+  private
+
+    def connect(address)
+      if address[0] == ?/
+        UNIXSocket.open(address)
+      elsif address =~ /^(\d+\.\d+\.\d+\.\d+):(\d+)$/
+        TCPSocket.open($1, $2.to_i)
+      else
+        raise ArgumentError, "Don't know how to bind: #{address}"
+      end
+    end
+  
+    # expands "unix:path/to/foo" to a socket relative to the current path
+    # expands pathnames of sockets if relative to "~" or "~username"
+    # expands "*:port and ":port" to "127.0.0.1:port"
+    def expand_addr(address) #:nodoc
+      return "0.0.0.0:#{address}" if Integer === address
+      return address unless String === address
+
+      case address
+      when %r{\Aunix:(.*)\z}
+        File.expand_path($1)
+      when %r{\A~}
+        File.expand_path(address)
+      when %r{\A(?:\*:)?(\d+)\z}
+        "127.0.0.1:#$1"
+      when %r{\A(.*):(\d+)\z}
+        # canonicalize the name
+        packed = Socket.pack_sockaddr_in($2.to_i, $1)
+        Socket.unpack_sockaddr_in(packed).reverse!.join(':')
+      else
+        address
+      end
+    end
+
+  end
+
+end

data/lib/pipemaster/configurator.rb

@@ -0,0 +1,228 @@
+require 'socket'
+require 'logger'
+
+module Pipemaster
+
+  # Implements a simple DSL for configuring a Pipemaster server.
+  class Configurator < Unicorn::Configurator
+
+    # Default settings for Pipemaster
+    DEFAULTS = {
+      :logger => Logger.new($stderr),
+      :after_fork => lambda { |server, worker|
+          server.logger.info("spawned pid=#{$$}")
+        },
+      :before_fork => lambda { |server, worker|
+          server.logger.info("spawning...")
+        },
+      :before_exec => lambda { |server|
+          server.logger.info("forked child re-executing...")
+        },
+      :pid => nil,
+      :commands => {},
+      :timeout => 60
+    }
+
+    def initialize(defaults = {}) #:nodoc:
+      self.set = Hash.new(:unset)
+      use_defaults = defaults.delete(:use_defaults)
+      self.config_file = defaults.delete(:config_file)
+      set.merge!(DEFAULTS) if use_defaults
+      defaults.each { |key, value| self.send(key, value) }
+      Hash === set[:listener_opts] or
+          set[:listener_opts] = Hash.new { |hash,key| hash[key] = {} }
+      Array === set[:listeners] or set[:listeners] = []
+      reload
+    end
+
+    # Sets object to the +new+ Logger-like object.  The new logger-like
+    # object must respond to the following methods:
+    #  +debug+, +info+, +warn+, +error+, +fatal+, +close+
+    def logger(new)
+      super
+    end
+
+    def commands(hash)
+      set[:commands] = hash
+    end
+
+    # Defines a command.
+    def command(name, &block)
+      set[:commands][name.to_sym] = block
+    end
+
+    autoload :Etc, 'etc'
+    # Change user/group ownership of the master process.
+    def user(user, group = nil)
+      # we do not protect the caller, checking Process.euid == 0 is
+      # insufficient because modern systems have fine-grained
+      # capabilities.  Let the caller handle any and all errors.
+      uid = Etc.getpwnam(user).uid
+      gid = Etc.getgrnam(group).gid if group
+      Unicorn::Util.chown_logs(uid, gid)
+      if gid && Process.egid != gid
+        Process.initgroups(user, gid)
+        Process::GID.change_privilege(gid)
+      end
+      Process.euid != uid and Process::UID.change_privilege(uid)
+    end
+
+    # Sets the working directory for Pipemaster.  Defaults to the location
+    # of the Pipefile.  This may be a symlink.
+    def working_directory(path)
+      # just let chdir raise errors
+      path = File.expand_path(path)
+      if config_file &&
+         config_file[0] != ?/ &&
+         ! test(?r, "#{path}/#{config_file}")
+        raise ArgumentError,
+              "pipefile=#{config_file} would not be accessible in" \
+              " working_directory=#{path}"
+      end
+      Dir.chdir(path)
+      Server::START_CTX[:cwd] = ENV["PWD"] = path
+    end
+
+    # Sets after_fork hook to a given block.  This block will be called by
+    # the worker after forking.
+    def after_fork(*args, &block)
+      set_hook(:after_fork, block_given? ? block : args[0])
+    end
+
+    # Sets before_fork hook to a given block.  This block will be called by
+    # the worker before forking.
+    def before_fork(*args, &block)
+      set_hook(:before_fork, block_given? ? block : args[0])
+    end
+
+    # Sets the before_exec hook to a given Proc object.  This
+    # Proc object will be called by the master process right
+    # before exec()-ing the new binary.  This is useful
+    # for freeing certain OS resources that you do NOT wish to
+    # share with the reexeced child process.
+    # There is no corresponding after_exec hook (for obvious reasons).
+    def before_exec(*args, &block)
+      set_hook(:before_exec, block_given? ? block : args[0], 1)
+    end
+
+    # Sets listeners to the given +addresses+, replacing or augmenting the
+    # current set.  This is for internal API use only, do not use it in your
+    # Pipemaster config file.  Use listen instead.
+    def listeners(addresses) # :nodoc:
+      Array === addresses or addresses = Array(addresses)
+      addresses.map! { |addr| expand_addr(addr) }
+      set[:listeners] = addresses
+    end
+
+    # Does nothing interesting for now.
+    def timeout(seconds)
+    end
+
+    # Does nothing interesting for now.
+    def worker_processes(nr)
+    end
+
+    # Adds an +address+ to the existing listener set.
+    #
+    # The following options may be specified (but are generally not needed):
+    #
+    # +:backlog+: this is the backlog of the listen() syscall.
+    #
+    # Some operating systems allow negative values here to specify the
+    # maximum allowable value.  In most cases, this number is only
+    # recommendation and there are other OS-specific tunables and
+    # variables that can affect this number.  See the listen(2)
+    # syscall documentation of your OS for the exact semantics of
+    # this.
+    #
+    # If you are running pipemaster on multiple machines, lowering this number
+    # can help your load balancer detect when a machine is overloaded
+    # and give requests to a different machine.
+    #
+    # Default: 1024
+    #
+    # +:rcvbuf+, +:sndbuf+: maximum receive and send buffer sizes of sockets
+    #
+    # These correspond to the SO_RCVBUF and SO_SNDBUF settings which
+    # can be set via the setsockopt(2) syscall.  Some kernels
+    # (e.g. Linux 2.4+) have intelligent auto-tuning mechanisms and
+    # there is no need (and it is sometimes detrimental) to specify them.
+    #
+    # See the socket API documentation of your operating system
+    # to determine the exact semantics of these settings and
+    # other operating system-specific knobs where they can be
+    # specified.
+    #
+    # Defaults: operating system defaults
+    #
+    # +:tcp_nodelay+: disables Nagle's algorithm on TCP sockets
+    #
+    # This has no effect on UNIX sockets.
+    #
+    # Default: operating system defaults (usually Nagle's algorithm enabled)
+    #
+    # +:tcp_nopush+: enables TCP_CORK in Linux or TCP_NOPUSH in FreeBSD
+    #
+    # This will prevent partial TCP frames from being sent out.
+    # Enabling +tcp_nopush+ is generally not needed or recommended as
+    # controlling +tcp_nodelay+ already provides sufficient latency
+    # reduction whereas Pipemaster does not know when the best times are
+    # for flushing corked sockets.
+    #
+    # This has no effect on UNIX sockets.
+    #
+    # +:tries+: times to retry binding a socket if it is already in use
+    #
+    # A negative number indicates we will retry indefinitely, this is
+    # useful for migrations and upgrades when individual workers
+    # are binding to different ports.
+    #
+    # Default: 5
+    #
+    # +:delay+: seconds to wait between successive +tries+
+    #
+    # Default: 0.5 seconds
+    #
+    # +:umask+: sets the file mode creation mask for UNIX sockets
+    #
+    # Typically UNIX domain sockets are created with more liberal
+    # file permissions than the rest of the application.  By default,
+    # we create UNIX domain sockets to be readable and writable by
+    # all local users to give them the same accessibility as
+    # locally-bound TCP listeners.
+    #
+    # This has no effect on TCP listeners.
+    #
+    # Default: 0 (world read/writable)
+    def listen(address, opt = {})
+      super
+    end
+
+    # Sets the +path+ for the PID file of the Pipemaster master process
+    def pid(path)
+      set_path(:pid, path)
+    end
+
+    # Allow redirecting $stderr to a given path.  Unlike doing this from
+    # the shell, this allows the Pipemaster process to know the path its
+    # writing to and rotate the file if it is used for logging.  The
+    # file will be opened with the File::APPEND flag and writes
+    # synchronized to the kernel (but not necessarily to _disk_) so
+    # multiple processes can safely append to it.
+    def stderr_path(path)
+      set_path(:stderr_path, path)
+    end
+
+    # Same as stderr_path, except for $stdout
+    def stdout_path(path)
+      set_path(:stdout_path, path)
+    end
+
+  private
+
+    def preload_app(bool)
+    end
+
+  end
+end
+