starling 0.9.3 → 0.9.8

data/CHANGELOG

@@ -0,0 +1,49 @@
+== 0.9.8
+* add fix to enable relative paths <david@motorator.com>
+* fix tests so they don't run 10 times due to a stupid bug with how the server is forked <seth@mojodna.net>
+* fix some other tests <romeda@gmail.com>
+* fix some error messages <romeda@gmail.com>
+* probably some other things <romeda@gmail.com>
+
+== 0.9.7.9
+* properly complain if the spool directory isn't writable <seth@mojodna.net>
+* assume group and user privileges in a working order <seth@mojodna.net>
+* support string user / group names in addition to uid/gids <seth@mojodna.net>
+
+== 0.9.7.7
+* added init.d scripts for redhat and ubuntu by Mike Perham <mperham@gmail.com>
+* fixed dependencies for SyslogLogger, eventmachine and memcache-client by Mike Perham <mperham@gmail.com>
+* added starling_top script to monitor starling server by Mike Perham <mperham@gmail.com>
+* fixed starling_top to use 22122 as port by Abdul-Rahman Advany <abdulrahman@advany.com>
+
+== 0.9.7.6 2008-06-24
+* removed client code (will be added to different project) by Abdul-Rahman Advany <abdulrahman@advany.com>
+
+== 0.9.7.5 2008-05-04
+* added worker class, using starling client you can now run them in background by Abdul-Rahman Advany <abdulrahman@advany.com>
+ - handles creation of threadpool
+ - handles fetching of messages and passing these to the threads
+ - handles pushing of processed messages to starling again if needed
+
+== 0.9.7 2008-05-03
+* merged branch of AnotherBritt and Glenn Rempe by Abdul-Rahman Advany <abdulrahman@advany.com>
+* rspeced tests by Abdul-Rahman Advany <abdulrahman@advany.com>
+
+==  0.9.6 2008-04-30
+* logging of message lifecycle as :age by AnotherBritt <?>
+* added some extra logging options by AnotherBritt <?>
+* added some test for epoll by AnotherBritt <?>
+
+== 0.9.5.4 2008-04-28
+
+* Bumped version number by Glenn Rempe <glenn.rempe@gmail.com>
+* Purged all old RubyForge config.  Now GitHub friendly by Glenn Rempe <glenn.rempe@gmail.com>
+* New gemspec for GitHub gem auto-build-serve by Glenn Rempe <glenn.rempe@gmail.com>
+
+== 0.9.4 2008-01-31 ==
+* Evented code added using EventMachine by Chris Wanstrath <chris@ozmm.org>
+
+== 2007-11-02
+
+* Initial release
+

data/{License.txt → LICENSE}

@@ -1,4 +1,4 @@
-Copyright (c) 2007 Blaine Cook, Twitter, Inc.
+Copyright (c) 2007 FIXME full name
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.rdoc

@@ -0,0 +1,106 @@
+= Name
+
+Starling - a light weight server for reliable distributed message passing.
+
+= Description
+
+Starling is a powerful but simple messaging server that enables reliable
+distributed queuing with an absolutely minimal overhead. It speaks the
+MemCache protocol for maximum cross-platform compatibility. Any language
+that speaks MemCache can take advantage of Starling's queue facilities.
+
+= Installation
+
+  This fork of the Starling source is hosted at GitHub and can be found at:
+
+  http://github.com/starling/starling/tree/master
+
+  The original source was to be found at RubyForge but no longer exists there.
+
+  GitHub serves gems prefixed by a username to differentiate different forks.
+  This project can be installed with:
+
+  # THIS COMMAND ONE TIME ONLY
+  gem sources -a http://gems.github.com/
+
+  # As often as you like
+  sudo gem install starling-starling
+
+  See http://gems.github.com/ if you want more info about GitHub and gems.
+
+= Quick Start Usage
+
+  # View the Starling help and usage message
+  starling --help
+
+  # In a console window start the Starling server.  By default
+  # it runs verbosely in the foreground, listening on 127.0.0.1:22122
+  # and stores its files under /tmp/starling:
+  starling
+
+  # In a new console test the put and get of messages on a queue:
+
+  irb
+  >> require 'starling'
+  => true
+  >> starling = Starling.new('127.0.0.1:22122')
+  => MemCache: 1 servers, 1 buckets, ns: nil, ro: false
+  >> starling.set('my_queue', 12345)
+  => nil
+  >> starling.get('my_queue')
+  => 12345
+
+  # You can do a simple loop over a queue with something like:
+  >> loop { puts starling.get('my_queue'); sleep 1 }
+  12345
+  nil
+  nil
+  ...
+
+  For more information run the following in a new console:
+  
+  'gem server'
+
+  This will start a gem server on http://localhost:8808/ which you can view in your
+  browser to see the RDocs for the gem. Or generate rdocs by running the following
+  in a new console:
+	
+  'rdoc'
+
+= Using fiveruns memcache-client
+
+	memcache-client from fiveruns has a couple of fixed added like supporting failover and retry on failure.
+	
+	This fork of the memcache-client source is hosted at GitHub and can be found at:
+
+	http://github.com/fiveruns/memcache-client/tree/master
+
+	It can be installed using:
+
+	# THIS COMMAND ONE TIME ONLY
+	gem sources -a http://gems.github.com/
+
+	# As often as you like
+	sudo gem install fiveruns-memcache-client
+
+= Known Issues
+
+* Starling is "slow" as far as messaging systems are concerned. In practice,
+  it's fast enough.
+
+== TODO
+
+* Implement memcached instead of memcache as a client interface (to make it faster)
+
+= Authors
+
+* Blaine Cook <romeda@gmail.com>
+* Chris Wanstrath <chris@ozmm.org>
+* AnotherBritt <?>
+* Glenn Rempe <?>
+* Abdul-Rahman Advany <abdulrahman@advany.com>
+
+= Copyright
+
+Starling - a light-weight server for reliable distributed message passing.
+Copyright 2007-2008 Blaine Cook <blaine@twitter.com>, Twitter Inc.

data/Rakefile

@@ -1,4 +1,22 @@
-require 'config/requirements'
-require 'config/hoe' # setup Hoe + all gem configuration
-
-Dir['tasks/**/*.rake'].each { |rake| load rake }
+require 'rubygems'
+require 'rake/rdoctask'
+require 'spec/rake/spectask'
+
+task :default => :spec
+
+task :install do
+  sh %{gem build starling.gemspec}
+  sh %{sudo gem install starling-*.gem}
+end
+
+Spec::Rake::SpecTask.new do |t|
+  t.ruby_opts = ['-rtest/unit']
+  t.spec_files = FileList['spec/*_spec.rb']
+  t.fail_on_error = true
+end
+  
+Rake::RDocTask.new do |rd|
+  rd.main = "README.rdoc"
+  rd.rdoc_files.include("README.rdoc", "lib/**/*.rb")
+  rd.rdoc_dir = 'doc'
+end

data/bin/starling

@@ -1,4 +1,6 @@
 #!/usr/bin/env ruby
 
-require 'starling/runner'
+require 'starling/server_runner'
+
 StarlingServer::Runner.run
+

data/bin/starling_top

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'ostruct'
+require 'socket'
+
+@options = OpenStruct.new
+@options.hostname = 'localhost'
+@options.port = 22122
+
+op = OptionParser.new do |opts|
+	opts.banner = "STARLING TOP\nUsage: startop [options]"
+	opts.separator "General Options:"
+	opts.on("-h HOSTNAME", "--hostname=HOSTNAME", "Hostname [default: localhost]") do |h|
+		@options.hostname = h
+	end
+	opts.on("-p PORT", "--port=PORT", Integer, "Port [default: 22122]") do |p|
+		@options.port = p
+	end
+	opts.on_tail("--help", "Show this message") do
+		puts opts
+		exit
+	end
+end
+op.parse!
+
+
+def stats_data
+  data = ''
+  sock = TCPSocket.new(@options.hostname, @options.port)
+  sock.print("stats\r\n")
+  sock.flush
+  # memcached does not close the socket once it is done writing
+  # the stats data.  We need to read line by line until we detect
+  # the END line and then stop/close on our side.
+  stats = sock.gets
+  while true
+    data += stats
+    break if stats.strip == 'END'
+    stats = sock.gets
+  end
+  sock.close
+  data
+end
+
+def parse(stats_data)
+  stats = []
+  stats_data.each_line do |line|
+    stats << "#{$1}: #{$2}" if line =~ /STAT (\w+) (\S+)/
+  end
+  stats.sort
+end
+
+stats = parse(stats_data)
+stats.each do |stat|
+	puts stat
+end

data/etc/sample-config.yml

@@ -0,0 +1,9 @@
+starling:
+  port: 22122
+  pid_file: /tmp/starling/starling.pid
+  queue_path: /tmp/starling/spool
+  timeout: 0
+  syslog_channel: starling-tampopo
+  log_level: DEBUG
+  daemonize: true
+

data/etc/starling.redhat

@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# starling    This shell script takes care of starting and stopping
+#                    the starling server
+# chkconfig: 345 98 98
+# description: The starling queue server
+
+#determine where the 'pidof' executable is located
+if [ -e /bin/pidof ]; then
+  PIDOF="/bin/pidof"
+elif [ -e /sbin/pidof ]; then
+  PIDOF="/sbin/pidof"
+elif [ -e /usr/local/bin/pidof ]; then
+  PIDOF="/usr/local/bin/pidof"
+elif [ -e /bin/pgrep ]; then
+  PIDOF="/bin/pgrep"
+elif [ -e /usr/bin/pgrep ]; then
+  PIDOF="/usr/bin/pgrep"
+elif [ -e /usr/local/bin/pgrep ]; then
+  PIDOF="/usr/local/bin/pgrep"
+else
+  echo "Could not find pidof or pgrep"
+fi
+
+PROGDIR="/usr/bin"
+PROGNAME="starling"
+OPTIONS="-u nobody -g nobody -L /var/log/starling.log -q /var/spool/starling"
+
+start() {
+  pid=`$PIDOF $PROGNAME`
+  if [ "$pid" != "" ]; then
+    echo "$PROGDIR$PROGNAME already running: $pid"
+  else
+    echo "Starting $PROGDIR$PROGNAME"
+    cd $PROGDIR
+    nohup $PROGDIR$PROGNAME $OPTIONS &
+  fi
+}
+
+stop() {
+  pid=`$PIDOF $PROGNAME`
+  if [ "$pid" != "" ]; then
+    echo "Stopping $PROGDIR$PROGNAME: $pid"
+    kill $pid
+  else
+    echo "$PROGDIR$PROGNAME not running"
+  fi
+}
+
+case "$1" in
+start)
+  start
+;;
+stop)
+  stop
+;;
+restart)
+  stop
+  sleep 3
+  start
+;;
+
+esac

data/etc/starling.ubuntu

@@ -0,0 +1,71 @@
+#! /bin/sh
+### BEGIN INIT INFO
+# Provides: starling
+# Required-Start: $local_fs $remote_fs
+# Required-Stop: $local_fs $remote_fs
+# Default-Start: 2 3 4 5
+# Default-Stop: S 0 1 6
+# Short-Description: Starling queue server
+# Description: The Starling distributed, transactional queue server
+### END INIT INFO
+# Author: Twitter
+# Version: 0.9.7.7
+
+set -e
+
+DUSER=starling
+DGROUP=starling
+LOGFILE=/var/log/starling/starling.log
+SPOOLDIR=/var/spool/starling
+PORT=22122
+LISTEN=0.0.0.0
+PIDFILE=/var/run/starling/starling.pid
+
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+NAME=starling
+DESC="Starling"
+INSTALL_DIR=/usr/local/bin
+DAEMON=$INSTALL_DIR/$NAME
+SCRIPTNAME=/etc/init.d/$NAME
+OPTS="-h $LISTEN -p $PORT -d -q $SPOOLDIR -P $PIDFILE -L $LOGFILE"
+
+. /lib/lsb/init-functions
+
+
+# Gracefully exit if the package has been removed.
+test -x $DAEMON || exit 0
+
+d_start() {
+	log_begin_msg "Starting Starling Server..."
+		start-stop-daemon -c $DUSER:$DGROUP --start --quiet --pidfile $PIDFILE --exec $DAEMON \
+		-- $OPTS || log_end_msg 1
+		log_end_msg 0
+}
+
+d_stop() {
+	log_begin_msg "Stopping Starling Server..."
+		start-stop-daemon -c $DUSER:$DGROUP --stop --quiet --pidfile $PIDFILE \
+		|| log_end_msg 1
+		log_end_msg 0
+}
+
+case "$1" in
+start)
+d_start
+;;
+stop)
+d_stop
+;;
+restart|force-reload|reload)
+d_stop
+sleep 2
+d_start
+;;
+*)
+echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
+exit 3
+;;
+esac
+
+exit 0
+

data/lib/starling.rb

@@ -6,7 +6,7 @@ class Starling < MemCache
 
   ##
   # fetch an item from a queue.
-  
+
   def get(*args)
     loop do
       response = super(*args)
@@ -16,13 +16,13 @@ class Starling < MemCache
   end
 
   ##
-  # insert +value+ into +queue+. 
+  # insert +value+ into +queue+.
   #
   # +expiry+ is expressed as a UNIX timestamp
-  # 
+  #
   # If +raw+ is true, +value+ will not be Marshalled. If +raw+ = :yaml, +value+
   # will be serialized with YAML, instead.
-  
+
   def set(queue, value, expiry = 0, raw = false)
     retries = 0
     begin

data/lib/starling/handler.rb

@@ -4,8 +4,8 @@ module StarlingServer
   # This is an internal class that's used by Starling::Server to handle the
   # MemCache protocol and act as an interface between the Server and the
   # QueueCollection.
-  
-  class Handler
+
+  class Handler < EventMachine::Connection
 
     DATA_PACK_FMT = "Ia*".freeze
 
@@ -13,18 +13,18 @@ module StarlingServer
     ERR_UNKNOWN_COMMAND = "CLIENT_ERROR bad command line format\r\n".freeze
 
     # GET Responses
-    GET_COMMAND = /^get (.{1,250})\r\n$/
+    GET_COMMAND = /\Aget (.{1,250})\s*\r\n/m
     GET_RESPONSE       = "VALUE %s %s %s\r\n%s\r\nEND\r\n".freeze
     GET_RESPONSE_EMPTY = "END\r\n".freeze
 
     # SET Responses
-    SET_COMMAND = /^set (.{1,250}) ([0-9]+) ([0-9]+) ([0-9]+)\r\n$/
+    SET_COMMAND = /\Aset (.{1,250}) ([0-9]+) ([0-9]+) ([0-9]+)\r\n/m
     SET_RESPONSE_SUCCESS  = "STORED\r\n".freeze
     SET_RESPONSE_FAILURE  = "NOT STORED\r\n".freeze
     SET_CLIENT_DATA_ERROR = "CLIENT_ERROR bad data chunk\r\nERROR\r\n".freeze
 
     # STAT Response
-    STATS_COMMAND = /stats\r\n$/
+    STATS_COMMAND = /\Astats\r\n/m
     STATS_RESPONSE = "STAT pid %d
 STAT uptime %d
 STAT time %d
@@ -47,77 +47,118 @@ STAT limit_maxbytes %d
     QUEUE_STATS_RESPONSE = "STAT queue_%s_items %d
 STAT queue_%s_total_items %d
 STAT queue_%s_logsize %d
-STAT queue_%s_expired_items %d\n".freeze
+STAT queue_%s_expired_items %d
+STAT queue_%s_age %d\r\n".freeze
+
+    SHUTDOWN_COMMAND = /\Ashutdown\r\n/m
+
+
+    @@next_session_id = 1
 
     ##
     # Creates a new handler for the MemCache protocol that communicates with a
     # given client.
 
-    def initialize(client, server, queue_collection)
-      @client = client
-      @server = server
-      @queue_collection = queue_collection
-      @expiry_stats = Hash.new(0)
+    def initialize(options = {})
+      @opts = options
     end
 
     ##
     # Process incoming commands from the attached client.
 
-    def run
-      while running? do
-        process_command(@client.readline)
-        Thread.current[:last_activity] = Time.now
+    def post_init
+      @stash = []
+      @data = ""
+      @data_buf = ""
+      @server = @opts[:server]
+      @logger = StarlingServer::Base.logger
+      @expiry_stats = Hash.new(0)
+      @expected_length = nil
+      @server.stats[:total_connections] += 1
+      set_comm_inactivity_timeout @opts[:timeout]
+      @queue_collection = @opts[:queue]
+
+      @session_id = @@next_session_id
+      @@next_session_id += 1
+
+      peer = Socket.unpack_sockaddr_in(get_peername)
+      #@logger.debug "(#{@session_id}) New session from #{peer[1]}:#{peer[0]}"
+    end
+
+    def receive_data(incoming)
+      @server.stats[:bytes_read] += incoming.size
+      @data << incoming
+
+      while data = @data.slice!(/.*?\r\n/m)
+        response = process(data)
       end
+
+      send_data response if response
     end
 
-    private
+    def process(data)
+      data = @data_buf + data if @data_buf.size > 0
+      # our only non-normal state is consuming an object's data
+      # when @expected_length is present
+      if @expected_length && data.size == @expected_length
+        response = set_data(data)
+        @data_buf = ""
+        return response
+      elsif @expected_length
+        @data_buf = data
+        return
+      end
+
+      case data
+      when SET_COMMAND
+        @server.stats[:set_requests] += 1
+        set($1, $2, $3, $4.to_i)
+      when GET_COMMAND
+        @server.stats[:get_requests] += 1
+        get($1)
+      when STATS_COMMAND
+        stats
+      when SHUTDOWN_COMMAND
+        # no point in responding, they'll never get it.
+        Runner::shutdown
+      else
+        logger.warn "Unknown command: #{data}."
+        respond ERR_UNKNOWN_COMMAND
+      end
+    rescue => e
+      logger.error "Error handling request: #{e}."
+      logger.debug e.backtrace.join("\n")
+      respond GET_RESPONSE_EMPTY
+    end
 
-    def running?
-      !Thread.current[:shutdown]
+    def unbind
+      #@logger.debug "(#{@session_id}) connection ends"
     end
 
+  private
     def respond(str, *args)
       response = sprintf(str, *args)
       @server.stats[:bytes_written] += response.length
-      @client.write response
+      response
     end
 
-    def process_command(command)
-      begin
-        @server.stats[:bytes_read] += command.length
-        case command
-        when SET_COMMAND
-          @server.stats[:set_requests] += 1
-          set($1, $2, $3, $4.to_i)
-        when GET_COMMAND
-          @server.stats[:get_requests] += 1
-          get($1)
-        when STATS_COMMAND
-          stats
-        else
-          logger.warn "Unknown command: #{command[0,4]}.\nFull command was #{command}."
-          respond ERR_UNKNOWN_COMMAND
-        end
-      rescue => e
-        logger.error "Error handling request: #{e}."
-        logger.debug e.backtrace.join("\n")
-        respond GET_RESPONSE_EMPTY
-      end
+    def set(key, flags, expiry, len)
+      @expected_length = len + 2
+      @stash = [ key, flags, expiry ]
+      nil
     end
 
-    def set(key, flags, expiry, len)
-      data = @client.read(len)
-      data_end = @client.read(2)
-      @server.stats[:bytes_read] += len + 2
-      if data_end == "\r\n" && data.size == len
-        internal_data = [expiry.to_i, data].pack(DATA_PACK_FMT)
-        if @queue_collection.put(key, internal_data)
-          respond SET_RESPONSE_SUCCESS
-        else
-          respond SET_RESPONSE_FAILURE
-        end
+    def set_data(incoming)
+      key, flags, expiry = @stash
+      data = incoming.slice(0...@expected_length-2)
+      @stash = []
+      @expected_length = nil
+
+      internal_data = [expiry.to_i, data].pack(DATA_PACK_FMT)
+      if @queue_collection.put(key, internal_data)
+        respond SET_RESPONSE_SUCCESS
       else
-        respond SET_CLIENT_DATA_ERROR
+        respond SET_RESPONSE_FAILURE
       end
     end
 
@@ -141,7 +182,7 @@ STAT queue_%s_expired_items %d\n".freeze
     end
 
     def stats
-      respond STATS_RESPONSE, 
+      respond STATS_RESPONSE,
         Process.pid, # pid
         Time.now - @server.stats(:start_time), # uptime
         Time.now.to_i, # time
@@ -169,13 +210,13 @@ STAT queue_%s_expired_items %d\n".freeze
                       k, v.length,
                       k, v.total_items,
                       k, v.logsize,
-                      k, @expiry_stats[k])
+                      k, @expiry_stats[k],
+                      k, v.current_age)
       end
     end
 
     def logger
-      @server.logger
+      @logger
     end
-
   end
 end