packet 0.1.0

data/LICENSE

@@ -0,0 +1,4 @@
+Packet is copyrighted free software by Hemant Kumar <mail@gnufied.org>
+You can redistribute it and/or modify it under either the terms of the GPL
+version 2 (see the file GPL), or Ruby License.
+

data/README

@@ -0,0 +1,251 @@
+Packet is a pure ruby library for writing network applications in Ruby.
+It follows Evented Model of network programming and implements almost all the
+features provided by EventMachine.
+
+It also provides real easy to user UNIX workers for concurrent programming.
+
+Its best to have some examples going:
+
+== Examples
+=== A Simple Echo Server:
+require "rubygems"
+require "packet"
+
+class Foo
+  def receive_data p_data
+    send_data(p_data)
+  end
+
+  def post_init
+    puts "Client connected"
+  end
+
+  def connection_completed
+    puts "Whoa man"
+  end
+  
+  def unbind
+    puts "Client Disconnected"
+  end
+end
+
+Packet::Reactor.run do |t_reactor|
+  t_reactor.start_server("localhost",11006,Foo)
+end
+
+Those new to network programming with events and callbacks, will note that,
+each time a new client connects an instance of class Foo is instantiated.
+When client writes some data to the socket, receive_data method is invoked.
+
+Although Packet implements an API similar to EventMachine, but it differs
+slightly because of the fact that, for a packet app, there can be more than one
+reactor loop running and hence, we don't use Packet.start_server(...).
+
+=== A Simple Http Client
+class WikiHandler
+  def receive_data p_data
+    p p_data
+  end
+
+  def post_init
+  end
+
+  def unbind
+  end
+
+  def connection_completed
+    send_data("GET / \r\n")
+  end
+end
+
+Packet::Reactor.run do |t_reactor|
+  t_reactor.connect("en.wikipedia.org",80,WikiHandler)
+end
+
+=== Using Callbacks and Deferables
+ Documentation to come.
+
+=== Using Workers
+  Packet enables you to write simple workers, which will run in 
+  different process and gives you nice 
+  evented handle for concurrent execution of various tasks. 
+
+  When, you are writing a scalable networking application 
+  using Event Model of network programming,
+  sometimes when processing of certain events take time, 
+  your event loop is stuck there. With green
+  threads, you don't really have a way of paralleling 
+  your request processing. Packet library, allows
+  you to write simple workers, for executing long 
+  running tasks. You can pass data and callbacks as an
+  argument.
+
+  When you are going to use workers in
+  your application, you need to define
+  constant WORKER_ROOT,
+  which is the directory location, where
+  your workers are located. All the workers defined in that directory
+  will be automatically, picked and forked in a
+  new process when your packet app starts. So, a typical
+  packet_app, that wants to use workers, will look like this:
+
+  packet_app_root
+    |
+    |__  lib
+    |
+    |___ worker
+    |
+    |___ config
+    |
+    |___ log
+
+  You would define WORKER_ROOT = PACKET_APP_ROOT/worker
+
+  All the workers must inherit class Packet::Worker, and hence a 
+  general skeleton of worker will look like:
+
+    class FooWorker < Packet::Worker
+      set_worker_name :foo_worker #=> This is necessary.
+      def receive_data p_data
+      end
+      
+      def connection_completed
+      end
+      
+      def unbind
+      end
+      
+      def post_init
+      end
+    end
+
+  All the forked workers are connected to master via
+  UNIX sockets, and hence messages passed to workers from master
+  will be available in receive_data method. Also,
+  when you are passing messages to workers, or worker is passing 
+  message to master ( in a nutshell, all the internal
+  communication between workers and master ) directly takes
+  place using ruby objects. All the passed ruby objects are
+  dumped and marshalled across unix sockets in a non blocking 
+  manner. BinParser class parses dumped binary objects and
+  makes sure, packets received at other end are complete.
+  Usually, you wouldn't need to worry about this little detail.
+
+  Packet provides various ways of interacting with
+  workers. Usually, when a worker is instantiated, a proxy for
+  that worker will also be instantiated at master
+  process. Packet automatically provides a worker proxy(See meta_pimp.rb)
+  for you, but  if you need to multiplex/demultiplex
+  requests based on certain criteria, you may as well define your 
+  own worker proxies. Code, would like something like this:
+
+    class FooWorker < Packet::Worker
+      set_worker_proxy :foo_handler
+    end
+
+  When you define, :foo_handler as a proxy for
+  this worker, packet is gonna search for FooHandler class and 
+  instantiate it when the worker gets started. All 
+  the worker proxies must inherit from Packet::Pimp. 
+  Have a look at, Packet::MetaPimp,
+  which acts as a meta pimp for all the workers,
+  which don't have a explicit worker proxy defined.
+
+=== A complete Case :
+
+    Just for kicks, lets write a sample server,
+    which evals whatever clients send to it. But, assuming this 'eval' of 
+    client data can be potentially time/cpu
+    consuming ( not to mention dangerous too ), we are gonna ask our eval_worker, to
+    perform eval and return the result to master process, which in 
+    turn returns the result to happy client.
+
+    # APP_ROOT/bin/eval_server.rb
+    EVAL_APP_ROOT = File.expand_path(File.join(File.dirname(__FILE__) + "/.."))
+    ["bin","worker","lib"].each { |x| $LOAD_PATH.unshift(EVAL_APP_ROOT + "/#{x}")}
+    WORKER_ROOT = EVAL_APP_ROOT + "/worker"
+    
+    require "packet"
+    class EvalServer
+      def receive_data p_data
+        ask_worker(:eval_worker,:data => p_data, :type => :request)
+      end
+  
+      # will be called, when any worker sends data back to master process
+      # it should be noted that, you may have several instances of eval_server in
+      # your master, for each connected client, but worker_receive will be always
+      # be invoked for the instance, which originally made the request.
+      # If you need fine control, over this behaviour, you can implement a worker proxy 
+      # on the lines of meta_pimp class. This API will change in future perhaps, as i
+      # expect, better ideas to come.  
+      def worker_receive p_data
+        send_data "#{p_data[:data]}\n"
+      end
+    
+      def show_result p_data
+        send_data("#{p_data[:response]}\n")
+      end
+    
+      def connection_completed
+      end
+    
+      def post_init
+      end
+    
+      def wow
+        puts "Wow"
+      end
+    end
+    
+    Packet::Reactor.run do |t_reactor|
+      t_reactor.start_server("localhost", 11006,EvalServer) do |instance|
+        instance.wow
+      end
+    end
+
+   # APP_ROOT/worker/eval_worker.rb
+     class EvalWorker < Packet::Worker
+     set_worker_name :eval_worker
+     def worker_init
+       p "Starting no proxy worker"
+     end
+   
+     def receive_data data_obj
+       eval_data = eval(data_obj[:data])
+       data_obj[:data] = eval_data
+       data_obj[:type] = :response
+       send_data(data_obj)
+     end
+   end
+
+=== Passing file descriptors to workers :
+  Packet, allows this style of load
+  distribution as well, you can use method send_fd to
+  pass file descriptors to workers. 
+  WARNING: This feature hasn't been tested well.
+
+=== Disable auto loading of certain workers:
+  Sometimes, you would need to start a
+  worker at runtime and don't want this pre-forking mechanism.
+  Packet, allows this. You just need to define
+  "set_no_auto_load true" in your worker class and worker
+  will not be automatically forked. Although name is a bit misleading perhaps.
+
+  Now, at runtime, you can call start_worker(:foo_worker, options)
+  to start a worker as usual. It should
+  be noted that, forking a worker, which is already
+  forked can be disastrous, since worker names are being
+  used as unique keys that represent a worker.
+
+== Performance:
+  Although written in pure ruby, packet performs
+  reasonably well. Mongrel, running on top of Packet is a tad
+  slower than Mongrel running on top of EventMachine. More benchmarks coming soon.
+
+== Credits
+  Francis for awesome EventMachine lib, which has constantly acted as an inspiration.
+  Ezra, for being a early user and porting mongrel to run on top of packet      
+
+
+
+

data/Rakefile

@@ -0,0 +1,87 @@
+require 'rubygems'
+require 'rake'
+require 'rake/clean'
+require 'rake/gempackagetask'
+require 'rake/rdoctask'
+require 'rake/testtask'
+require 'spec/rake/spectask'
+require 'fileutils'
+def __DIR__
+  File.dirname(__FILE__)
+end
+include FileUtils
+
+NAME = "packet"
+$LOAD_PATH.unshift __DIR__+'/lib'
+require 'packet'
+
+CLEAN.include ['**/.*.sw?', '*.gem', '.config']
+
+
+@windows = (PLATFORM =~ /win32/)
+
+SUDO = @windows ? "" : (ENV["SUDO_COMMAND"] || "sudo")
+
+
+
+desc "Packages up Packet."
+task :default => [:package]
+
+task :doc => [:rdoc]
+
+
+Rake::RDocTask.new do |rdoc|
+      files = ['README', 'LICENSE', 'CHANGELOG',
+               'lib/**/*.rb']
+      rdoc.rdoc_files.add(files)
+      rdoc.main = 'README'
+      rdoc.title = 'Packet Docs'
+      rdoc.rdoc_dir = 'doc/rdoc'
+      rdoc.options << '--line-numbers' << '--inline-source'
+end
+
+spec = Gem::Specification.new do |s|
+  s.name = NAME
+  s.version = Packet::VERSION
+  s.platform = Gem::Platform::RUBY
+  s.has_rdoc = true
+  s.extra_rdoc_files = ["README", "LICENSE", 'TODO']
+  #s.rdoc_options += RDOC_OPTS +
+  #  ['--exclude', '^(app|uploads)']
+  s.summary = "Packet, Events... we got em."
+  s.description = s.summary
+  s.author = "Hemant"
+  s.email = 'foo@bar.com'
+  s.homepage = 'http://code.google.com/p/packet/'
+  s.required_ruby_version = '>= 1.8.4'
+
+  s.files = %w(LICENSE README Rakefile TODO) + Dir.glob("{bin,spec,lib,examples,script}/**/*")
+
+  s.require_path = "lib"
+  s.bindir = "bin"
+end
+
+Rake::GemPackageTask.new(spec) do |p|
+  #p.need_tar = true
+  p.gem_spec = spec
+end
+
+task :install do
+  sh %{rake package}
+  sh %{#{SUDO} gem install pkg/#{NAME}-#{Packet::VERSION} --no-rdoc --no-ri}
+end
+
+task :uninstall => [:clean] do
+  sh %{#{SUDO} gem uninstall #{NAME}}
+end
+
+##############################################################################
+# SVN
+##############################################################################
+
+desc "Add new files to subversion"
+task :svn_add do
+   system "svn status | grep '^\?' | sed -e 's/? *//' | sed -e 's/ /\ /g' | xargs svn add"
+end
+
+

data/bin/packet_mongrel.rb

@@ -0,0 +1,215 @@
+# This module rewrites pieces of the very good Mongrel web server in
+# order to change it from a threaded application to an event based
+# application running inside an Packet event loop.  It should
+# be compatible with the existing Mongrel handlers for Rails,
+# Camping, Nitro, etc....
+PACKET_APP = File.expand_path(File.join(File.dirname(__FILE__) + "/.."))
+["bin","config","parser","worker","framework","lib","pimp"].each { |x| $LOAD_PATH.unshift(PACKET_APP + "/#{x}")}
+require "packet"
+require 'packet'
+
+module Mongrel
+  class MongrelProtocol < Packet::Connection
+    def post_init
+      @parser = HttpParser.new
+      @params = HttpParams.new
+      @nparsed = 0
+      @request = nil
+      @request_len = nil
+      @linebuffer = ''
+    end
+
+    def receive_data data
+      @linebuffer << data
+      @nparsed = @parser.execute(@params, @linebuffer, @nparsed) unless @parser.finished?
+      if @parser.finished?
+        if @request_len.nil?
+          @request_len = @params[::Mongrel::Const::CONTENT_LENGTH].to_i
+          script_name, path_info, handlers = ::Mongrel::HttpServer::Instance.classifier.resolve(@params[::Mongrel::Const::REQUEST_PATH])
+          if handlers
+            @params[::Mongrel::Const::PATH_INFO] = path_info
+            @params[::Mongrel::Const::SCRIPT_NAME] = script_name
+            @params[::Mongrel::Const::REMOTE_ADDR] = @params[::Mongrel::Const::HTTP_X_FORWARDED_FOR] #|| ::Socket.unpack_sockaddr_in(get_peername)[1]
+            @notifiers = handlers.select { |h| h.request_notify }
+          end
+          if @request_len > ::Mongrel::Const::MAX_BODY
+            new_buffer = Tempfile.new(::Mongrel::Const::MONGREL_TMP_BASE)
+            new_buffer.binmode
+            new_buffer << @linebuffer[@nparsed..-1]
+            @linebuffer = new_buffer
+          else
+            @linebuffer = StringIO.new(@linebuffer[@nparsed..-1])
+            @linebuffer.pos = @linebuffer.length
+          end
+        end
+        if @linebuffer.length >= @request_len
+          @linebuffer.rewind
+          ::Mongrel::HttpServer::Instance.process_http_request(@params,@linebuffer,self)
+          @linebuffer.delete if Tempfile === @linebuffer
+        end
+      elsif @linebuffer.length > ::Mongrel::Const::MAX_HEADER
+        close_connection
+        raise ::Mongrel::HttpParserError.new("HEADER is longer than allowed, aborting client early.")
+      end
+    rescue ::Mongrel::HttpParserError
+      if $mongrel_debug_client
+        STDERR.puts "#{Time.now}: BAD CLIENT (#{params[Const::HTTP_X_FORWARDED_FOR] || client.peeraddr.last}): #$!"
+        STDERR.puts "#{Time.now}: REQUEST DATA: #{data.inspect}\n---\nPARAMS: #{params.inspect}\n---\n"
+      end
+      close_connection
+    rescue Exception => e
+      close_connection
+      raise e
+    end
+
+    def write data
+      send_data data
+    end
+
+    def closed?
+      false
+    end
+
+  end
+
+  class HttpServer
+    def initialize(host, port, num_processors=950, x=0, y=nil) # Deal with Mongrel 1.0.1 or earlier, as well as later.
+      @socket = nil
+      @classifier = URIClassifier.new
+      @host = host
+      @port = port
+      @workers = ThreadGroup.new
+      if y
+        @throttle = x
+        @timeout = y || 60
+      else
+        @timeout = x
+      end
+      @num_processors = num_processors #num_processors is pointless for evented....
+      @death_time = 60
+      self.class.const_set(:Instance,self)
+    end
+
+    def run
+      trap('INT') { raise StopServer }
+      trap('TERM') { raise StopServer }
+      @acceptor = Thread.new do
+        Packet::Reactor.run do |t_reactor|
+          begin
+            t_reactor.start_server(@host,@port,MongrelProtocol)
+          rescue StopServer
+            t_reactor.start_server
+          end
+        end
+      end
+    end
+
+    def process_http_request(params,linebuffer,client)
+      if not params[Const::REQUEST_PATH]
+        uri = URI.parse(params[Const::REQUEST_URI])
+        params[Const::REQUEST_PATH] = uri.request_uri
+      end
+
+      raise "No REQUEST PATH" if not params[Const::REQUEST_PATH]
+
+      script_name, path_info, handlers = @classifier.resolve(params[Const::REQUEST_PATH])
+
+      if handlers
+        notifiers = handlers.select { |h| h.request_notify }
+        request = HttpRequest.new(params, linebuffer, notifiers)
+
+        # request is good so far, continue processing the response
+        response = HttpResponse.new(client)
+
+        # Process each handler in registered order until we run out or one finalizes the response.
+        dispatch_to_handlers(handlers,request,response)
+
+        # And finally, if nobody closed the response off, we finalize it.
+        unless response.done
+          response.finished
+        else
+          response.close_connection
+        end
+      else
+        # Didn't find it, return a stock 404 response.
+        client.send_data(Const::ERROR_404_RESPONSE)
+        client.close_connection
+      end
+    end
+
+    def dispatch_to_handlers(handlers,request,response)
+      handlers.each do |handler|
+        handler.process(request, response)
+        break if response.done
+      end
+    end
+  end
+
+  class HttpRequest
+    def initialize(params, linebuffer, dispatchers)
+      @params = params
+      @dispatchers = dispatchers
+      @body = linebuffer
+    end
+  end
+
+  class HttpResponse
+    def send_file(path, small_file = false)
+      File.open(path, "rb") do |f|
+        while chunk = f.read(Const::CHUNK_SIZE) and chunk.length > 0
+          begin
+            write(chunk)
+          rescue Object => exc
+            break
+          end
+        end
+      end
+      @body_sent = true
+    end
+
+    def write(data)
+      @socket.send_data data
+    end
+
+    def close_connection_after_writing
+      @socket.close_connection
+    end
+
+    def socket_error(details)
+      @socket.close_connection
+      done = true
+      raise details
+    end
+
+    def finished
+      send_status
+      send_header
+      send_body
+      @socket.close_connection
+    end
+  end
+
+  class Configurator
+    # This version fixes a bug in the regular Mongrel version by adding
+    # initialization of groups.
+    def change_privilege(user, group)
+      if user and group
+        log "Initializing groups for {#user}:{#group}."
+        Process.initgroups(user,Etc.getgrnam(group).gid)
+      end
+
+      if group
+        log "Changing group to #{group}."
+        Process::GID.change_privilege(Etc.getgrnam(group).gid)
+      end
+
+      if user
+        log "Changing user to #{user}."
+        Process::UID.change_privilege(Etc.getpwnam(user).uid)
+      end
+    rescue Errno::EPERM
+      log "FAILED to change user:group #{user}:#{group}: #$!"
+      exit 1
+    end
+  end
+end