flamingo 0.3.1 → 0.4.0

data/README.md

@@ -1,14 +1,19 @@
 Flamingo
 ========
-Flamingo is a resque-based system for handling the Twitter Streaming API.
-
-This is *early alpha* code. Parts of it are graceful, like the curve of a
-flamingo's neck: it capably processes the multiple high-volume sample and filter
-streams that power tweetreach.com. Many parts of it are ungainly, like a
-flamingo's knees: this is early code, and it will change rapidly. And parts of
-it are mired in muck, like a flamingo's feet: it has too few tests, and surely
-some configuration we forgot to tell you about. That said, it does work: give it
-a try if you have the need.
+Flamingo is a service for connecting to and processing events from the Twitter 
+Streaming API. Here are the highlights:
+
+* It runs as a daemon that you communicate with via a REST API interface.
+* Handles all the work of intelligently managing connections to the 
+  Streaming API (handling things like backoffs and reconnects).
+* Stream events (tweets) can be stored directly to a file on disk via the 
+  built in event log functionality. This is useful for collecting data for 
+  further batch processing of incoming data via hadoop, for example.
+* Stream events can be placed on a Resque queue for downstream processing. This 
+  is an easy way to connect your application logic for processing tweets.
+* It provides helpful metrics like stream rates, event counts, limit information 
+  available via the REST endpoint /meta.json.
+* It supports a minimal configuration REPL via the flamingo command.
 
 Dependencies
 ------------
@@ -17,6 +22,11 @@ few dependencies and they are very specific. We plan to have fewer dependencies
 and be more liberal with versions soon. Right now these gems and versions are 
 what is working well in production for us.
 
+Caveat Emptor
+-------------
+This is *alpha* code. However, it processes multiple high-volume streams 
+in production as part of TweetReach.com.
+
 Getting Started
 ---------------
 1. Install the gem
@@ -63,7 +73,7 @@ commandline (see below)
 
 6. Your second task from the flamingo console is to route the incoming tweets onto a queue -- in this case the EXAMPLE queue. This is used by the flamingod we'll start next but has no direct effect now.
 
-        >> Subscription.new('EXAMPLE').save
+        >> Subscription.new('example').save
 
 7. Start the Flamingo Daemon (`flamingod` installed during `gem install`), and also start watching its log file:
 
@@ -80,11 +90,8 @@ commandline (see below)
         [2010-07-20 05:58:07, INFO] - Starting wader on pid=91008 under pid=91003
         [2010-07-20 05:58:07, INFO] - Starting dispatcher on pid=91009 under pid=91003
         [2010-07-20 05:58:12, INFO] - Listening on stream: /1/statuses/filter.json?track=%23etsy,austin,cheaptweet
-        ... short initial delay ....
-        [2010-07-20 05:58:42, DEBUG] - Wader dispatched event
-        [2010-07-20 05:58:42, DEBUG] - Put job on subscription queue EXAMPLE for {"text":If you ever visit Austin make sure to go to Torchy's Tacos",...
 
-    On the resque-web dashboard, you should see a queue come up called EXAMPLE, with jobs accruing. There will only be 0 of 0 workers working: let's fix that
+    On the resque-web dashboard, you should see a queue come up called example, with jobs accruing. There will only be 0 of 0 workers working: let's fix that
         
 8. You'll consume those events with a resque worker, something like the following but more audacious:
 
@@ -99,10 +106,10 @@ commandline (see below)
 
 9. Start the worker task (see `examples/Rakefile`):
         
-        $ QUEUE=EXAMPLE rake -t examples/Rakefile resque:work
+        $ QUEUE=example rake -t examples/Rakefile resque:work
 
    Two things should now happen:
-   * The pent-up jobs from the EXAMPLE queue should spray across your console
+   * The pent-up jobs from the example queue should spray across your console
    * The resque dashboard should show the queue being emptied as a result 
    
 10. Interact with your running flamingod instance via the REST API (by default it is on port 4711)
@@ -122,7 +129,7 @@ components of the flamingo flock:
 
 Coordinates the wader process (initiates stream request, pushes each response
 into the queue), the Sinatra webserver (handles subscriptions and changing 
-stream parameters), and a set of dispatchers (routes responses).
+stream parameters), and a dispatcher (routes events to subscribers).
 
 You can control flamingod with the following signals:
 
@@ -131,7 +138,12 @@ You can control flamingod with the following signals:
 
 *wader*
 
-The wader process starts the stream and dispatches stream responses as they arrive into a Resque queue.
+The wader process starts the stream and queues events as they arrive into a redis list.
+
+*dispatcher*
+
+The dispatcher process retrieves events from the dispatch queue, writes them to 
+the event log (if configured) and to any subscriptions (if configured).
 
 *web server*
 

data/examples/flamingo.yml

@@ -11,6 +11,17 @@ stream:         filter
 logging:
   dest:         /tmp/flamingo.log
   level:        DEBUG
+  
+# Event logging (optional) 
+# Allows you to log the raw JSON of stream events to a set of rotating files 
+# stored in the directory you specify. Size is the maximum number of events 
+# that will be written to a log file before it is rotated. If size is omitted 
+# no rotation will be done. If you expect a high volume stream, set this number 
+# to something relatively large or you will end up with lots of small log files.
+# 10000-100000 is probably a good place to start.
+  event:
+    dir:        /tmp/flamingo_events
+    size:       100000
 
 # Where is the redis server the flamingod processes should connect to?
 # By default, all keys are namespaced wih "flamingo". May be changed 

data/lib/flamingo.rb

@@ -11,8 +11,15 @@ require 'sinatra/base'
 
 require 'flamingo/version'
 require 'flamingo/config'
+require 'flamingo/logging/formatter'
+require 'flamingo/logging/utils'
+require 'flamingo/logging/event_log'
 require 'flamingo/meta'
-require 'flamingo/dispatch_event'
+require 'flamingo/stats/rate_counter'
+require 'flamingo/stats/events'
+require 'flamingo/stats/connection'
+require 'flamingo/dispatch_queue'
+require 'flamingo/dispatcher'
 require 'flamingo/stream_params'
 require 'flamingo/stream'
 require 'flamingo/subscription'
@@ -24,7 +31,6 @@ require 'flamingo/daemon/dispatcher_process'
 require 'flamingo/daemon/web_server_process'
 require 'flamingo/daemon/wader_process'
 require 'flamingo/daemon/flamingod'
-require 'flamingo/logging/formatter'
 require 'flamingo/web/server'
 
 module Flamingo
@@ -64,7 +70,7 @@ module Flamingo
     def redis=(server)
       host, port, db = server.split(':')
       redis = Redis.new(:host => host, :port => port,
-        :thread_safe => true, :db => db)
+        :thread_safe=>true, :db => db)
       @redis = Redis::Namespace.new(namespace, :redis => redis)
       
       # Ensure resque is configured to use this redis as well
@@ -92,21 +98,37 @@ module Flamingo
     end
     
     def dispatch_queue
-      @dispatch_queue ||= "#{namespace}:dispatch"
+      @dispatch_queue ||= DispatchQueue.new(redis)
     end
     
     def meta
-      @meta ||= Flamingo::Meta.new(redis)
+      @meta ||= Meta.new(redis)
+    end
+    
+    def event_stats
+      @event_stats ||= Stats::Events.new
+    end
+    
+    def connection_stats
+      @connection_stats ||= Stats::Connection.new
+    end    
+    
+    def new_event_log
+      if event_config = config.logging.event(nil) 
+        Logging::EventLog.new(event_config.dir,event_config.size(0))
+      else
+        nil
+      end
     end
     
     # Intended to be called after a fork so that we don't have 
     # issues with shared file descriptors, sockets, etc
     def reconnect!
-      reconnect_redis_client(@redis)      
-      reconnect_redis_client(Resque.redis)
       # Reload logger
       logger.close
       self.logger = new_logger
+      reconnect_redis_client(@redis)      
+      reconnect_redis_client(Resque.redis)
     end
     
     private

data/lib/flamingo/daemon/dispatcher_process.rb

@@ -1,15 +1,24 @@
 module Flamingo
   module Daemon
     class DispatcherProcess < ChildProcess
+      
       def run
-        worker = Resque::Worker.new(Flamingo.dispatch_queue)
-        def worker.procline(value)
-          # Hack to get around resque insisting on setting the proces name
-          $0 = "flamingod-dispatcher"
-        end
+        register_signal_handlers
+        $0 = "flamingod-dispatcher"
+        @dispatcher = Flamingo::Dispatcher.new
         Flamingo.logger.info "Starting dispatcher on pid=#{Process.pid} under pid=#{Process.ppid}"
-        worker.work(1) # Wait 1s between jobs
+        @dispatcher.run
       end
+      
+      def stop
+        @dispatcher.stop
+      end
+      
+      def register_signal_handlers
+        trap("INT")  { stop }
+        trap("TERM") { stop }
+      end    
+      
     end
   end
 end

data/lib/flamingo/daemon/flamingod.rb

@@ -168,6 +168,7 @@ module Flamingo
 
       def run
         $0 = 'flamingod'
+        Flamingo.logger.info "Flamingod version: #{Flamingo::VERSION}"
         set_process_meta_data
         trap_signals
         start_children

data/lib/flamingo/dispatch_queue.rb

@@ -0,0 +1,30 @@
+module Flamingo
+  class DispatchQueue
+    
+    attr_accessor :redis
+    
+    def initialize(redis)
+      self.redis = redis
+      @queue_name = "queue:dispatch"
+    end
+    
+    def enqueue(event)
+      redis.rpush(@queue_name,event)
+    end
+    
+    def dequeue
+      redis.lpop(@queue_name)
+    end
+    
+    def page(page_num,page_size=20)
+      start_index = page_num*page_size
+      end_index = start_index+page_size-1
+      redis.lrange(@queue_name,start_index,end_index)
+    end
+    
+    def size
+      redis.llen(@queue_name)
+    end
+    
+  end
+end

data/lib/flamingo/dispatcher.rb

@@ -0,0 +1,98 @@
+module Flamingo
+  class Dispatcher
+    
+    def initialize
+      @shutdown = false
+    end
+    
+    def stop
+      @shutdown = true  
+    end
+    
+    def run(wait_time=0.5)
+      init_event_log
+      while(!@shutdown) do
+        if event = next_event
+          dispatch(event)
+        else
+          if wait_time == 0
+            stop
+          else
+            wait(wait_time)
+          end
+        end
+      end     
+    end
+    
+    private
+      def next_event
+        Flamingo.dispatch_queue.dequeue
+      end
+    
+      def meta
+        Flamingo.meta
+      end
+      
+      def logger
+        Flamingo.logger
+      end
+      
+      def init_event_log
+        @event_log = Flamingo.new_event_log
+      end
+      
+      def event_log
+        @event_log
+      end
+      
+      def wait(time=0.5)
+        sleep(time) unless @shutdown
+      end
+
+      def dispatch(event_json)
+        type, event = typed_event(parse(event_json))
+        update_stats(type,event)
+        if event_log
+          event_log << event_json
+        end
+        if type == :limit
+          handle_limit(event)
+        end
+        Subscription.all.each do |sub|
+          Resque::Job.create(sub.name, "HandleFlamingoEvent", type, event)
+        end
+      rescue => e
+        handle_error(event_json,e)
+      end
+      
+      def update_stats(type, event)
+        Flamingo.event_stats.event!(type)
+      end
+      
+      def handle_error(event_json,error)
+        Logging::Utils.log_error(logger,
+          "Failure dispatching event: #{event_json}",error)
+      end
+      
+      def handle_limit(event)
+        skipped = event.values.first
+        Flamingo.connection_stats.limited!(skipped)
+        logger.warn "Rate limited: #{skipped} skipped"
+      end
+
+      def parse(json)
+        Yajl::Parser.parse(json,:symbolize_keys=>true)
+      end
+
+      def typed_event(event)
+        # Events with one {key: value} pair are used as control events from 
+        # Twitter. These include limit, delete, scrub_geo and others.
+        if event.size == 1
+          event.shift
+        else
+          [:tweet, event]
+        end
+      end    
+    
+  end
+end

data/lib/flamingo/logging/event_log.rb

@@ -0,0 +1,78 @@
+module Flamingo
+  module Logging
+    class EventLog
+      
+      attr_accessor :dir, :max_size
+      
+      def initialize(dir,size=10000)
+        self.dir = dir
+        self.max_size = size
+        @rotations = 0
+        rotate!
+        unless open?
+          raise "Failure opening log file"
+        end
+      end
+      
+      def append(event)
+        if should_rotate?
+          rotate!
+        end
+        @log << "#{event}\n"
+        @event_count += 1
+      end
+      alias_method :<<, :append
+      
+      def open?
+        !@log.nil?
+      end
+      
+      private
+        def should_rotate?
+          max_size > 0 && @event_count >= max_size
+        end
+        
+        def rotate!
+          close_current_log
+          open_new_log
+          if open?
+            symlink_current_log
+            update_counters
+          end
+        end
+        
+        def update_counters
+          @event_count = 0
+          @rotations += 1
+        end
+        
+        def close_current_log
+          @log.close if @log
+        rescue => e
+          Logging::Utils.log_error(Flamingo.logger,
+            "Failure closing event log #{@log_filename}",e)
+        end
+
+        def open_new_log
+          @log_filename = File.expand_path(File.join(dir,log_filename))
+          @log = File.open(@log_filename,'a')
+          @log.sync = true #Immediately flush all output
+        rescue => e
+          Logging::Utils.log_error(Flamingo.logger,
+            "Failure opening event log #{@log_filename}",e)
+          @log = nil
+        end
+      
+        def log_filename
+          ts = Time.now.strftime("%Y%m%d-%H%M%S")
+          "event-#{ts}-#{@rotations}.log"
+        end
+      
+        def symlink_current_log
+          current_log = File.expand_path(File.join(dir,"event.log"))
+          `ln -fs #{@log_filename} #{current_log}`
+        end
+      
+    end
+  end
+end

data/lib/flamingo/logging/utils.rb

@@ -0,0 +1,22 @@
+module Flamingo
+  module Logging
+    module Utils
+      
+      def log_error(logger, msg, e)
+        logger.error msg
+        logger.error error_trace(e,2)
+      end
+      
+      def error_trace(e,indent=0)
+        space = " "*indent
+        err = "#{space}#{e.class.name}: #{e.message}\n"
+        space = " "*(indent+2)
+        err << "#{space}#{e.backtrace.join("\n#{space}")}\n"
+        err
+      end
+      
+      extend self
+      
+    end
+  end
+end

data/lib/flamingo/stats/connection.rb

@@ -0,0 +1,59 @@
+module Flamingo
+  module Stats
+    class Connection
+      
+      START_TIME          = "conn:start:time"
+      START_EVENT_COUNT   = "conn:start:event_count" 
+      START_TWEET_COUNT   = "conn:start:tweet_count"
+      LIMIT_COUNT         = "conn:limit:count"
+      LIMIT_TIME          = "conn:limit:time"
+      COVERAGE            = "conn:coverage"      
+      
+      def connected!
+        meta.set(START_TIME,Time.now.to_i)
+        meta.set(START_EVENT_COUNT,event_stats.all_count)
+        meta.set(START_TWEET_COUNT,event_stats.tweet_count)
+        meta.set(COVERAGE,100)
+        meta.delete(LIMIT_COUNT)
+        meta.delete(LIMIT_TIME)
+      end
+      
+      def limited!(count)
+        meta.set(LIMIT_COUNT,count)
+        meta.set(LIMIT_TIME,Time.now.to_i)
+        meta.set(COVERAGE,coverage_rate)
+      end
+      
+      def received_tweets
+        event_stats.tweet_count - (meta.get(START_TWEET_COUNT) || 0)
+      end
+      
+      def skipped_tweets
+        meta.get(LIMIT_COUNT) || 0
+      end
+
+      def received_events
+        event_stats.all_count - (meta.get(START_EVENT_COUNT) || 0)
+      end      
+      
+      def coverage_rate
+        received = received_tweets
+        possible_tweets = received + skipped_tweets
+        if possible_tweets == 0
+          0
+        else
+          (received / possible_tweets.to_f)*100
+        end
+      end
+
+      def meta
+        Flamingo.meta
+      end
+
+      def event_stats
+        Flamingo.event_stats
+      end
+      
+    end
+  end
+end

data/lib/flamingo/stats/events.rb

@@ -0,0 +1,52 @@
+module Flamingo
+  module Stats
+    class Events
+      
+      ALL_COUNT   = "events:all_count"
+      RATE        = "events:rate"
+      LAST_TIME   = "events:last_time"
+      TYPE_COUNT  = "events:%s_count"
+      TWEET_COUNT = TYPE_COUNT % [:tweet]
+      
+      def initialize
+        @rate_counter = Flamingo::Stats::RateCounter.new(10) do |eps|
+          meta.set(RATE,eps)
+          logger.debug "%.3f eps" % [eps]
+        end
+      end
+      
+      def event!(type)
+        @rate_counter.event!
+        meta.incr(ALL_COUNT)
+        meta.set(LAST_TIME,Time.now.to_i)
+        meta.incr(TYPE_COUNT % [type])    
+      end
+      
+      def all_count
+        meta.get(ALL_COUNT) || 0
+      end
+      
+      def last_time
+        meta.get(LAST_TIME)
+      end
+      
+      def type_count(type)
+        meta.get(TYPE_COUNT % [type]) || 0
+      end
+      
+      def tweet_count
+        type_count(:tweet)
+      end
+      
+      private
+        def logger
+          Flamingo.logger
+        end
+        
+        def meta
+          Flamingo.meta
+        end
+      
+    end
+  end
+end

data/lib/flamingo/stats/rate_counter.rb

@@ -0,0 +1,38 @@
+module Flamingo
+  module Stats
+    
+    # Simple counter for measuring stream rates in events per second
+    class RateCounter
+  
+      attr_accessor :rate, :callback
+      
+      def initialize(sample_duration=60, &block)
+        @sample_duration = sample_duration
+        self.callback = block
+        start_sample
+      end
+          
+      def event!
+        @count += 1
+        if (diff = (now - @sample_start_time)) >= @sample_duration
+          self.rate = (@count / diff.to_f)
+          if callback
+            callback.call(rate)
+          end
+          start_sample
+        end
+      end
+      
+      private
+        def now
+          Time.now.to_i
+        end
+        
+        def start_sample
+          @sample_start_time = now
+          @count = 0
+        end
+
+    end  
+  end
+end

data/lib/flamingo/stream.rb

@@ -53,12 +53,16 @@ module Flamingo
         :name=>name,:resource=>resource,:params=>params.all
       )
     end
+
+    def query
+      params.map{|key,value| "#{key}=#{param_value(value)}" }.join("&")
+    end
     
-    private
-      def query
-        params.map{|key,value| "#{key}=#{param_value(value)}" }.join("&")
-      end
+    def to_s
+      "#{path}?#{query}"
+    end
     
+    private
       def param_value(val)
         case val
           when String then CGI.escape(val)

data/lib/flamingo/version.rb

@@ -1,3 +1,3 @@
 module Flamingo
-  Version = VERSION = '0.3.1'
-end
+  Version = VERSION = '0.4.0'
+end

data/lib/flamingo/wader.rb

@@ -90,8 +90,10 @@ module Flamingo
     private
       def connect_and_run
         EventMachine::run do
+          Flamingo.logger.info("Connecting to stream: #{stream}")
           self.connection = stream.connect(:auth=>"#{screen_name}:#{password}")
-          Flamingo.logger.info("Listening on stream: #{stream.path}")
+          Flamingo.logger.info("Connected to stream")
+          Flamingo.connection_stats.connected!
   
           connection.each_item do |event_json|
             dispatch_event(event_json)
@@ -141,7 +143,7 @@ module Flamingo
       end
       
       def dispatch_event(event_json)
-        Resque.enqueue(Flamingo::DispatchEvent, event_json)
+        Flamingo.dispatch_queue.enqueue(event_json)
       end
 
   end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: flamingo
 version: !ruby/object:Gem::Version 
-  hash: 17
+  hash: 15
   prerelease: false
   segments: 
   - 0
-  - 3
-  - 1
-  version: 0.3.1
+  - 4
+  - 0
+  version: 0.4.0
 platform: ruby
 authors: 
 - Hayes Davis
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-11-26 00:00:00 -08:00
+date: 2011-02-10 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -262,9 +262,15 @@ files:
 - lib/flamingo/daemon/trap_keeper.rb
 - lib/flamingo/daemon/wader_process.rb
 - lib/flamingo/daemon/web_server_process.rb
-- lib/flamingo/dispatch_event.rb
+- lib/flamingo/dispatch_queue.rb
+- lib/flamingo/dispatcher.rb
+- lib/flamingo/logging/event_log.rb
 - lib/flamingo/logging/formatter.rb
+- lib/flamingo/logging/utils.rb
 - lib/flamingo/meta.rb
+- lib/flamingo/stats/connection.rb
+- lib/flamingo/stats/events.rb
+- lib/flamingo/stats/rate_counter.rb
 - lib/flamingo/stream.rb
 - lib/flamingo/stream_params.rb
 - lib/flamingo/subscription.rb

data/lib/flamingo/dispatch_event.rb

@@ -1,51 +0,0 @@
-module Flamingo
-  class DispatchEvent
-
-    @parser = Yajl::Parser.new(:symbolize_keys => true)
-
-    class << self
-      
-      def queue
-        Flamingo.dispatch_queue
-      end
-      
-      def meta
-        Flamingo.meta
-      end
-
-      #
-      # TODO Track stats including: tweets per second and last tweet time
-      # TODO Provide some first-level check for repeated status ids
-      # TODO Consider subscribers for receiving particular terms - do the heavy
-      #      lifting of parsing tweets and delivering them to particular subscribers
-      # TODO Consider window of tweets (approx 3 seconds) and sort before
-      #      dispatching to improve in-order delivery (helps with "k-sorted")
-      #
-      def perform(event_json)
-        meta.incr("events:all_count")
-        meta.set("events:last_time",Time.now.utc.to_i)
-        type, event = typed_event(parse(event_json))
-        meta.incr("events:#{type}_count")
-        Subscription.all.each do |sub|
-          Resque::Job.create(sub.name, "HandleFlamingoEvent", type, event)
-          Flamingo.logger.debug "Put job on subscription queue #{sub.name}\n#{event_json}"
-        end
-      end
-
-      def parse(json)
-        @parser.parse(json)
-      end
-
-      def typed_event(event)
-        if event[:delete]
-          [:delete, event[:delete]]
-        elsif event[:link]
-          [:link,   event[:link]]
-        else
-          [:tweet,  event]
-        end
-      end
-
-    end
-  end
-end