flamingo 0.1 → 0.2.0

data/README.md

@@ -2,9 +2,13 @@ Flamingo
 ========
 Flamingo is a resque-based system for handling the Twitter Streaming API.
 
-This is *early alpha* code. There will be a lot of change and things like tests 
-coming in the future. That said, it does work so give it a try if you have the 
-need.
+This is *early alpha* code. Parts of it are graceful, like the curve of a
+flamingo's neck: it capably processes the multiple high-volume sample and filter
+streams that power tweetreach.com. Many parts of it are ungainly, like a
+flamingo's knees: this is early code, and it will change rapidly. And parts of
+it are mired in muck, like a flamingo's feet: it has too few tests, and surely
+some configuration we forgot to tell you about. That said, it does work: give it
+a try if you have the need.
 
 Dependencies
 ------------
@@ -13,6 +17,8 @@ Dependencies
 * sinatra
 * twitter-stream
 * yajl-ruby
+* active_support
+* redis-namespace
 
 By default, the `resque` gem installs the latest 2.x `redis` gem, so if
 you are using Redis 1.x, you may want to swap it out.
@@ -30,44 +36,133 @@ Getting Started
 1. Install the gem
         sudo gem install flamingo
 
-2. Create a config file (see `examples/flamingo.yml`) with at least a username and password
+2. Create a config file (see `examples/flamingo.yml`) with at least a username
+and password. You can store this in ~/flamingo.yml or specify it on the
+commandline (see below)
 
-        username: USERNAME
+        username: SCREEN_NAME
         password: PASSWORD
-        stream: filter
+        
+        # should be "filter" or "sample", probably.
+        # Set the track terms for "filter" from the flamingo console (see README)
+        stream:   filter
+        
         logging:
-          dest: /YOUR/LOG/PATH.LOG
-          level: LOGLEVEL
+          dest:   /tmp/flamingo.log
+          level:  DEBUG
+
+        redis:
+          host:   0.0.0.0:6379
+        web:
+          host:   0.0.0.0:4711
 
     `LOGLEVEL` is one of the following:
     `DEBUG` < `INFO` < `WARN` < `ERROR` < `FATAL` < `UNKNOWN`
 
-3. Start the Redis server
+3. Start the Redis server, and (optionally) open the resque web dashboard:
 
         $ redis-server
+        $ resque-web
+
+4. To set your tracking terms and start the queue subscription, jump into the `flamingo` client (installed during `gem install`):
 
-4. Configure tracking using `flamingo` client (installed during `gem install`)
+        $ flamingo path/to/flamingo.yml
+
+5. This is a regular-old irb console, so anything ruby goes. First, register the terms you'd like to search on.  This doesn't have a direct effect: it just pokes the values into the database so that the wader knows what to listen for.
 
-        $ flamingo
         >> s = Stream.get(:filter)
-        >> s.params[:track] = %w(FOO BAR BAZ)
-        >> Subscription.new('YOUR_QUEUE').save
+        >> s.params[:track] = ["@cheaptweet", "austin", "#etsy"]
+
+    For now, use those three actual terms -- they'll give you a nice, testable receipt rate that is neither too slow ('...is this thing on?') nor torrential (you can watch the sream from your terminal window).  Also note that you don't have to escape the tracking terms: twitter-stream will handle all that.
+
+6. Your second task from the flamingo console is to route the incoming tweets onto a queue -- in this case the EXAMPLE queue. This is used by the flamingod we'll start next but has no direct effect now.
+
+        >> Subscription.new('EXAMPLE').save
 
-5. Start the Flamingo Daemon (`flamingod` installed during `gem install`)
+7. Start the Flamingo Daemon (`flamingod` installed during `gem install`), and also start watching its log file:
 
-        $ flamingod -c your/config/file.yml
+        $ flamingod -c path/to/flamingo.yml
+        $ tail -f /tmp/flamingo.log
 
+    If things go well, you'll see something like
+
+        [2010-07-20 05:58:07, INFO] - Loaded config file from flamingo.yml
+        [2010-07-20 05:58:07, INFO] - Flamingod starting children
+        [2010-07-20 05:58:07, INFO] - Flamingod starting new wader
+        [2010-07-20 05:58:07, INFO] - Flamingod starting new dispatcher
+        [2010-07-20 05:58:07, INFO] - Flamingod starting new web server
+        [2010-07-20 05:58:07, INFO] - Starting wader on pid=91008 under pid=91003
+        [2010-07-20 05:58:07, INFO] - Starting dispatcher on pid=91009 under pid=91003
+        [2010-07-20 05:58:12, INFO] - Listening on stream: /1/statuses/filter.json?track=%23etsy,austin,cheaptweet
+        ... short initial delay ....
+        [2010-07-20 05:58:42, DEBUG] - Wader dispatched event
+        [2010-07-20 05:58:42, DEBUG] - Put job on subscription queue EXAMPLE for {"text":If you ever visit Austin make sure to go to Torchy's Tacos",...
+
+    On the resque-web dashboard, you should see a queue come up called EXAMPLE, with jobs accruing. There will only be 0 of 0 workers working: let's fix that
         
-6. Consume events with a resque worker
+8. You'll consume those events with a resque worker, something like the following but more audacious:
 
         class HandleFlamingoEvent
-          
           # type: One of "tweet" or "delete"
           # event: a hash of the json data from twitter
           def self.perform(type,event)
-            # Do stuff with the data
+            # Do stuff with the data, probably something more interesting than this:
+            puts [type, event].inspect
           end
-          
         end
+
+9. Start the worker task (see `examples/Rakefile`):
         
-        $ QUEUE=YOUR_QUEUE rake resque:work
+        $ QUEUE=EXAMPLE rake -t examples/Rakefile resque:work
+
+   Two things should now happen:
+   * The pent-up jobs from the EXAMPLE queue should spray across your console
+   * The resque dashboard should show the queue being emptied as a result 
+
+
+   
+Overview
+--------
+
+Flamingo uses EventMachine, sinatra and the twitter-stream API library to
+efficiently route and process stream and dispatch events. Here are the
+components of the flamingo flock:
+
+*flamingo daemon (flamingod)*
+
+Coordinates the wader process (initiates stream request, pushes each response
+into the queue), the Sinatra webserver (handles subscriptions and changing 
+stream parameters), and a set of dispatchers (routes responses).
+
+You can control flamingod with the following signals:
+
+* TERM and INT will kill the flamingod parent process, and signal each child with TERM
+* USR1 will restart the wader gracefully. This is used to change stream parameters
+
+*wader*
+
+The wader process starts the stream and dispatches stream responses as they arrive into a Resque queue.
+
+*web server*
+
+The flamingo webserver code creates and manages stream requests using a
+lightweight Sinatra responder.
+
+*workers*
+
+This is the part you write. These are standard resque workers, living on one or
+many machines, doing anything that your heart can imagine and your fingers can
+code.
+
+
+TODO
+-----
+* OAuth instructions
+    
+
+Flamingo
+--------
+
+Here is a photo of a flamingo:
+
+![Flamingo!](http://farm4.static.flickr.com/3438/3302580937_0ec540b73e_z_d.jpg "Flamingo Photo by William Warby, CC-BY License: http://www.flickr.com/photos/wwarby/3302580937 :: photo taken 21 Feb 2009 in Dagnall, England.")

data/examples/Rakefile

@@ -1,15 +1,22 @@
-# Simple example that reads from a subscription queue and writes the events 
-# to STDOUT
+#
+# Simple example: reads from a subscription queue, writes the events to STDOUT
+#
 # Usage (from this directory):
-#   $ QUEUE=YOUR_QUEUE rake resque:work
+#   $ QUEUE=EXAMPLES rake resque:work
 
 require 'rubygems'
 require 'resque/tasks'
 
 class HandleFlamingoEvent
-  
-  def self.perform(type,event)
-    puts type, event
+
+  #
+  # type: One of "tweet" or "delete"
+  # event: a hash of the json data from twitter
+  #
+  #def self.perform(type, event_info, event)
+  def self.perform(type, event)
+    # Do stuff with the data, probably something more interesting than this:
+    puts [type, event].inspect
   end
-  
-end
+
+end

data/examples/flamingo.yml

@@ -1,10 +1,22 @@
-username: SCREEN_NAME
-password: PASSWORD
-stream: filter
+username:       SCREEN_NAME
+password:       PASSWORD
+
+# either "filter" or "sample"
+# For filter, set the terms to track in the flamingo console (see README.md)
+stream:         filter
+
+# Point the logs where you like.
+# Should change the log level from DEBUG to INFO before you deploy: allowed levels are
+#    DEBUG < INFO < WARN < ERROR < FATAL < UNKNOWN
 logging:
-  dest: /your/log/file/here.log
-  level: INFO
+  dest:         /tmp/flamingo.log
+  level:        DEBUG
+
+# Where is the redis server the flamingod processes should connect to?
 redis:
-  host: 0.0.0.0:6379
+  host:         0.0.0.0:6379
+
+# What port and interface should the flamingod web_server listen on?
+# use 0.0.0.0 for all interfaces, 127.0.0.1 to listen on only localhost
 web:
-  host: 0.0.0.0:4711
+  host:         0.0.0.0:4711

data/lib/flamingo.rb

@@ -17,6 +17,7 @@ require 'flamingo/stream_params'
 require 'flamingo/stream'
 require 'flamingo/subscription'
 require 'flamingo/wader'
+require 'flamingo/daemon/trap_keeper'
 require 'flamingo/daemon/pid_file'
 require 'flamingo/daemon/child_process'
 require 'flamingo/daemon/dispatcher_process'
@@ -37,6 +38,10 @@ module Flamingo
       logger.info "Loaded config file from #{config_file}"
     end
     
+    def config=(config)
+      @config = config 
+    end
+    
     def config
       @config
     end
@@ -98,6 +103,10 @@ module Flamingo
       @logger ||= new_logger
     end
     
+    def logger=(logger)
+      @logger = logger
+    end
+    
     private
       def root_dir
         File.expand_path(File.dirname(__FILE__)+'/..')

data/lib/flamingo/daemon/child_process.rb

@@ -1,6 +1,10 @@
 module Flamingo
   module Daemon
     class ChildProcess
+      
+      # For process-scoping of traps
+      include TrapKeeper
+      
       attr_accessor :pid
   
       def kill(sig)
@@ -8,6 +12,17 @@ module Flamingo
       end
       alias_method :signal, :kill
       
+      def running?
+        # Borrowed from daemons gem
+        Process.kill(0, pid)
+        return true
+      rescue Errno::ESRCH
+        return false
+      rescue ::Exception
+        # for example on EPERM (process exists but does not belong to us)
+        return true
+      end
+      
       def start
         self.pid = fork { run }
       end      

data/lib/flamingo/daemon/flamingod.rb

@@ -1,7 +1,25 @@
 module Flamingo
   module Daemon
+    #
+    # Flamingod is the main overseer of the Flamingo flock.
+    #
+    # Starts three sets of children:
+    #
+    # * A wader process: initiates stream request, pushes each response into the queue
+    # * A Sinatra server: lightweight responder to create and manage subscriptions
+    # * A set of dispatchers: worker processes that handle each stream response.
+    #
+    # You can control the flamingod with the following signals:
+    #
+    # * TERM and INT will kill the flamingod parent process, and signal each
+    #   child with TERM
+    # * USR1 will restart the wader gracefully.
+    #
     class Flamingod
-
+      
+      # For process-scoping of traps
+      include TrapKeeper
+      
       def exit_signaled?
         @exit_signaled
       end
@@ -40,14 +58,27 @@ module Flamingo
       end
 
       def restart_wader
-        Flamingo.logger.info "Flamingod restarting wader pid=#{@wader.pid} with SIGINT"
-        @wader.kill("INT")
+        if @wader
+          Flamingo.logger.info "Flamingod restarting wader pid=#{@wader.pid} with SIGINT"
+          @wader.kill("INT")
+        else
+          Flamingo.logger.info "Wader is not started. Attempting to start new wader."
+          @wader = start_new_wader
+        end
       end
 
       def signal_children(sig)
         pids = (children.map {|c| c.pid}).join(",")
         Flamingo.logger.info "Flamingod sending SIG#{sig} to pids=#{pids}"
-        children.each {|child| child.signal(sig) }
+        children.each do |child|
+          if child.running?
+            begin
+              child.signal(sig)
+            rescue => e
+              Flamingo.logger.info "Failure sending SIG#{sig} to child #{child.pid}: #{e}"
+            end
+          end
+        end
       end
 
       def terminate!
@@ -57,7 +88,7 @@ module Flamingo
       end
 
       def children
-        [@wader,@web_server] + @dispatchers
+        ([@wader,@web_server] + @dispatchers).compact
       end
 
       def start_children
@@ -67,12 +98,17 @@ module Flamingo
         @web_server = start_new_web_server
       end
 
+      #
+      # Unless signaled externally, waits in an endless loop. If any child
+      # process terminates, it restarts that process.
+      # TODO Needs intelligent behavior so we don't get endless loops
       def wait_on_children()
         until exit_signaled?
           child_pid = Process.wait(-1)
+          child_status = $?
           unless exit_signaled?
-            if @wader.pid == child_pid
-              @wader = start_new_wader
+            if @wader && @wader.pid == child_pid
+              handle_wader_exit(child_status)
             elsif @web_server.pid == child_pid
               @web_server = start_new_web_server
             elsif (to_delete = @dispatchers.find{|d| d.pid == child_pid})
@@ -84,6 +120,17 @@ module Flamingo
           end
         end
       end
+      
+      def handle_wader_exit(status)
+        if WaderProcess.fatal_exit?(status)
+          Flamingo.logger.error "Wader exited with status "+
+            "#{status.exitstatus} and cannot be automatically restarted"
+          $stderr.write("Wader exited with fatal error. Check the the log.")
+          terminate!
+        else
+          @wader = start_new_wader
+        end
+      end
 
       def run_as_daemon
         pid_file = PidFile.new

data/lib/flamingo/daemon/trap_keeper.rb

@@ -0,0 +1,22 @@
+module Flamingo
+  module Daemon
+    module TrapKeeper
+      
+      # Use instead of Kernel.trap to ensure that only the process that 
+      # originally registered the trap has its block executed. This is necessary 
+      # for cases where we fork after setting up traps since the child process 
+      # gets the traps from the parent.
+      def trap(signal,&block)
+        owner_pid = Process.pid
+        Kernel.trap(signal) do
+          if Process.pid == owner_pid
+            block.call
+          end
+        end
+      end
+      
+      module_function :trap
+      
+    end
+  end
+end

data/lib/flamingo/daemon/wader_process.rb

@@ -1,6 +1,29 @@
 module Flamingo
   module Daemon
     class WaderProcess < ChildProcess
+
+      # Exit codes
+      EXIT_CLEAN = 0
+
+      # Non-fatal exit code - For transient network errors where a retry is 
+      # likely to resolve the problem
+      EXIT_UNKNOWN_ERROR      = 001
+      EXIT_MAX_RECONNECTS     = 002
+      EXIT_SERVER_UNAVAILABLE = 003
+
+      # 1XX is a fatal exit code - Human intervention or a configuration change 
+      # is necessary to get the wader started
+      EXIT_FATAL_RANGE    = 100..199
+      EXIT_AUTHENTICATION = 100
+      EXIT_UNKNOWN_STREAM = 101
+      EXIT_INVALID_PARAMS = 102
+      
+      class << self
+        def fatal_exit?(status)
+          status && EXIT_FATAL_RANGE.include?(status.exitstatus)
+        end
+      end
+      
       def register_signal_handlers
         trap("INT")  { stop }
       end
@@ -16,13 +39,39 @@ module Flamingo
 
         @wader = Flamingo::Wader.new(screen_name,password,stream)
         Flamingo.logger.info "Starting wader on pid=#{Process.pid} under pid=#{Process.ppid}"
-        @wader.run
-        Flamingo.logger.info "Wader pid=#{Process.pid} stopped"
+        
+        exit_code = EXIT_CLEAN
+        begin
+          @wader.run
+        rescue => e
+          exit_code = error_exit_code(e)
+        end
+        
+        Flamingo.logger.info "Wader pid=#{Process.pid} exited with code #{exit_code}"
+        exit(exit_code)
       end
       
       def stop
         @wader.stop
       end
+
+      private
+        def error_exit_code(ex)
+          case ex
+            when Flamingo::Wader::AuthenticationError 
+              then EXIT_AUTHENTICATION
+            when Flamingo::Wader::UnknownStreamError
+              then EXIT_UNKNOWN_STREAM
+            when Flamingo::Wader::InvalidParametersError
+              then EXIT_INVALID_PARAMS
+            when Flamingo::Wader::MaxReconnectsExceededError
+              then EXIT_MAX_RECONNECTS
+            when Flamingo::Wader::ServerUnavailableError
+              then EXIT_SERVER_UNAVAILABLE
+            else
+              EXIT_UNKNOWN_ERROR
+          end
+        end
     end
   end
 end

data/lib/flamingo/dispatch_event.rb

@@ -1,40 +1,41 @@
 module Flamingo
   class DispatchEvent
-    
+
     @queue = :flamingo
     @parser = Yajl::Parser.new(:symbolize_keys => true)
-    
+
     class << self
-    
+
+      #
+      # TODO Track stats including: tweets per second and last tweet time
+      # TODO Provide some first-level check for repeated status ids
+      # TODO Consider subscribers for receiving particular terms - do the heavy
+      #      lifting of parsing tweets and delivering them to particular subscribers
+      # TODO Consider window of tweets (approx 3 seconds) and sort before
+      #      dispatching to improve in-order delivery (helps with "k-sorted")
+      #
       def perform(event_json)
-        #TODO Track stats including: tweets per second and last tweet time
-        #TODO Provide some first-level check for repeated status ids
-        #TODO Consider subscribers for receiving particular terms - do the heavy 
-        #     lifting of parsing tweets and delivering them to particular subscribers
-        #TODO Consider window of tweets (approx 3 seconds) and sort before 
-        #     dispatching to improve in-order delivery (helps with "k-sorted") 
         type, event = typed_event(parse(event_json))
-#        Flamingo.logger.info Flamingo.router.destinations(type,event).inspect
         Subscription.all.each do |sub|
           Resque::Job.create(sub.name, "HandleFlamingoEvent", type, event)
           Flamingo.logger.debug "Put job on subscription queue #{sub.name} for #{event_json}"
         end
       end
-      
+
       def parse(json)
         @parser.parse(json)
       end
-      
+
       def typed_event(event)
         if event[:delete]
-          [:delete,event[:delete]]
+          [:delete, event[:delete]]
         elsif event[:link]
-          [:link,event[:link]]
+          [:link,   event[:link]]
         else
-          [:tweet,event]
+          [:tweet,  event]
         end
       end
-      
+
     end
   end
 end

data/lib/flamingo/stream_params.rb

@@ -1,41 +1,42 @@
 module Flamingo
-  
+  #
+  # Facade for redis:
+  # database object that behaves like a hash
+  #
   class StreamParams
-    
     include Enumerable
-    
     attr_accessor :stream_name
-    
+
     def initialize(stream_name)
       self.stream_name = stream_name
     end
-    
+
     def set(key,*values)
       delete(key)
       add(key,*values)
     end
-    
+
     def []=(key,values)
       values = [values] unless values.is_a?(Array)
       set(key,*values)
     end
-    
+
     def add(key,*values)
       values.each do |value|
         Flamingo.redis.sadd redis_key(key), value
       end
     end
-    
+
     def remove(key,*values)
       values.each do |value|
         Flamingo.redis.srem redis_key(key), value
       end
     end
-    
+
     def delete(key)
       Flamingo.redis.del redis_key(key)
     end
-    
+
     def get(key)
       Flamingo.redis.smembers redis_key(key)
     end
@@ -53,22 +54,20 @@ module Flamingo
         h
       end
     end
-    
+
     def each
       keys.each do |key|
         yield(key,get(key))
       end
     end
-    
+
     private
       def redis_key_pattern
         "streams/#{stream_name}?*"
       end
-      
+
       def redis_key(key)
         "streams/#{stream_name}?#{key}"
       end
-    
   end
-  
-end
+end

data/lib/flamingo/subscription.rb

@@ -1,9 +1,9 @@
 module Flamingo
-  
+  #
+  # Track stream subscriptions in the Redis db.
+  #
   class Subscription
-    
     class << self
-      
       def all
         Flamingo.redis.smembers("subscriptions").map do |name|
           new(name)
@@ -15,25 +15,22 @@ module Flamingo
           Subscription.new(name)
         end
       end
-      
     end
-    
+
     attr_accessor :name
-        
     def initialize(name)
-      self.name = name  
+      self.name = name
     end
-    
+
     def save
       Flamingo.logger.info("Adding #{name} to subscriptions")
       Flamingo.redis.sadd("subscriptions",name)
     end
-    
+
     def delete
       Flamingo.logger.info("Removing #{name} from subscriptions")
       Flamingo.redis.srem("subscriptions",name)
     end
-    
+
   end
-  
 end

data/lib/flamingo/version.rb

@@ -1,3 +1,3 @@
 module Flamingo
-  Version = VERSION = '0.1'
+  Version = VERSION = '0.2.0'
 end

data/lib/flamingo/wader.rb

@@ -1,57 +1,149 @@
 module Flamingo
   class Wader
     
-    attr_accessor :screen_name, :password, :stream, :connection
+    class WaderError < StandardError 
+    end
+
+    class HttpStatusError < WaderError
+      
+      attr_accessor :code
+      
+      def initialize(message,code)
+        super(message)  
+        self.code = code
+      end
+    end
+
+    # Errors from certain HTTP Statuses
+    class AuthenticationError < HttpStatusError; end
+    class UnknownStreamError < HttpStatusError; end
+    class InvalidParametersError < HttpStatusError; end
     
+    # Fatal error from too many reconnection attempts
+    class MaxReconnectsExceededError < WaderError; end
+      
+    # Raised if the server is just not available, e.g. Twitter is down
+    class ServerUnavailableError < WaderError; end
+    
+    attr_accessor :screen_name, :password, :stream, :connection,
+      :server_unavailable_max_retries, 
+      :server_unavailable_wait, 
+      :server_unavailable_retries
+
     def initialize(screen_name,password,stream)
       self.screen_name = screen_name
       self.password = password
       self.stream = stream
+      self.server_unavailable_max_retries = 5
+      self.server_unavailable_wait = 60
     end
-    
+
+    #
+    # The main EventMachine run loop
+    #
+    # Start the stream listener (using twitter-stream, http://github.com/voloko/twitter-stream)
+    # Listen for responses and errors;
+    #   dispatch each for later handling
+    #
     def run
-      EventMachine::run do
-        self.connection = stream.connect(:auth=>"#{screen_name}:#{password}")
-        Flamingo.logger.info("Listening on stream: #{stream.path}")
-  
-        connection.each_item do |event_json|
-          dispatch_event(event_json)
+      self.server_unavailable_retries = 0
+      begin
+        connect_and_run
+      rescue => e
+        # This is largely to get around a bug in Twitter-Stream that should 
+        # be fixed in the next release. If the server is just not there on 
+        # the first try, it blows up. Hopefully this code can be removed after 
+        # that release.
+        Flamingo.logger.warn "Failure initiating connection. Most likely "+
+          "because server is unavailable.\n#{e}\n#{e.backtrace.join("\n\t")}"
+        if server_unavailable_retries < server_unavailable_max_retries
+          sleep(server_unavailable_wait)
+          self.server_unavailable_retries += 1
+          retry
+        else
+          raise ServerUnavailableError.new
         end
-  
-        connection.on_error do |message|
-          dispatch_error(:generic,message)
-        end
-  
-        connection.on_reconnect do |timeout, retries|
-          dispatch_error(:reconnection,
-            "Will reconnect after #{timeout}. Retry \##{retries}",
-            {:timeout=>timeout,:retries=>retries}
-          )
-        end
-  
-        connection.on_max_reconnects do |timeout, retries|
-          dispatch_error(:fatal,
-            "Failed to reconnect after #{retries} retries",
-            {:timeout=>timeout,:retries=>retries}
-          )
-        end
-      end  
+      end
+      raise @error if @error
     end
-   
+    
+    def retries
+      if connection
+        # This is weird but necessary because twitter-stream increments the 
+        # reconnect_retries a bit oddly. They are incremented prior to the 
+        # actual reconnect which means that the last reconnect_retries value 
+        # is 1 more than the real value.
+        rs = connection.reconnect_retries
+        rs == 0 ? 0 : rs - 1
+      else
+        0
+      end
+    end
+
     def stop
-      connection.stop
+      if connection
+        connection.stop
+      end
       EM.stop
     end
     
     private
-      def dispatch_event(event_json)
-        Flamingo.logger.debug "Wader dispatched event"
-        Resque.enqueue(Flamingo::DispatchEvent,event_json)
+      def connect_and_run
+        EventMachine::run do
+          self.connection = stream.connect(:auth=>"#{screen_name}:#{password}")
+          Flamingo.logger.info("Listening on stream: #{stream.path}")
+  
+          connection.each_item do |event_json|
+            dispatch_event(event_json)
+          end
+  
+          connection.on_error do |message|
+            handle_connection_error(message)
+          end
+  
+          connection.on_reconnect do |timeout, retries|
+            Flamingo.logger.warn "Failed to connect. Will reconnect after "+
+              "#{timeout}. Retry \##{retries}"
+          end
+  
+          connection.on_max_reconnects do |timeout, retries|
+            stop_and_raise!(MaxReconnectsExceededError.new(
+              "Failed to reconnect after #{retries-1} retries"
+            ))
+          end
+        end
+      end
+      
+      # Decides what to do with specific connection errors. For explanations 
+      # of various HTTP status codes from the Streaming API, see:
+      # http://dev.twitter.com/pages/streaming_api_response_codes
+      def handle_connection_error(message)
+        code = connection.code # HTTP status code
+        if [401,403].include?(code)
+          stop_and_raise!(AuthenticationError.new(message,code))
+        elsif code == 404
+          stop_and_raise!(UnknownStreamError.new(message,code))
+        elsif [406,413,416].include?(code)
+          stop_and_raise!(InvalidParametersError.new(message,code))
+        elsif code && code > 0
+          Flamingo.logger.warn "Received non-fatal HTTP status #{code} with "+
+            "message \"#{message}\". Will retry."
+        else
+          Flamingo.logger.warn "Unknown connection error: #{message}. "+
+            "Will retry." 
+        end        
       end
       
-      def dispatch_error(type,message,data={})
-        Flamingo.logger.error "Received error: #{message}"
-        Resque.enqueue(Flamingo::DispatchError,type,message,data)
+      def stop_and_raise!(error)
+        Flamingo.logger.error "Stopping wader due to error: #{error}"
+        stop
+        @error = error
+      end
+      
+      def dispatch_event(event_json)
+        Flamingo.logger.debug "Wader dispatched event"
+        Resque.enqueue(Flamingo::DispatchEvent, event_json)
       end
+
   end
 end

metadata

@@ -1,12 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: flamingo
 version: !ruby/object:Gem::Version 
-  hash: 9
+  hash: 23
   prerelease: false
   segments: 
   - 0
-  - 1
-  version: "0.1"
+  - 2
+  - 0
+  version: 0.2.0
 platform: ruby
 authors: 
 - Hayes Davis
@@ -15,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-07-19 00:00:00 -05:00
+date: 2010-08-01 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -130,6 +131,22 @@ dependencies:
         version: 2.1.0
   type: :runtime
   version_requirements: *id007
+- !ruby/object:Gem::Dependency 
+  name: mockingbird
+  prerelease: false
+  requirement: &id008 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 27
+        segments: 
+        - 0
+        - 1
+        - 0
+        version: 0.1.0
+  type: :development
+  version_requirements: *id008
 description: "    Flamingo makes it easy to wade through the Twitter Streaming API by \n    handling all connectivity and resource management for you. You just tell \n    it what to track and consume the information in a resque queue. \n\n    Flamingo isn't a traditional ruby gem. You don't require it into your code.\n    Instead, it's designed to run as a daemon like redis or mysql. It provides \n    a REST interface to change the parameters sent to the Twitter Streaming \n    resource. All events from the streaming API are placed on a resque job \n    queue where your application can process them.\n\n"
 email: hayes@appozite.com
 executables: 
@@ -148,6 +165,7 @@ files:
 - lib/flamingo/daemon/dispatcher_process.rb
 - lib/flamingo/daemon/flamingod.rb
 - lib/flamingo/daemon/pid_file.rb
+- lib/flamingo/daemon/trap_keeper.rb
 - lib/flamingo/daemon/wader_process.rb
 - lib/flamingo/daemon/web_server_process.rb
 - lib/flamingo/dispatch_error.rb