weeter 0.9.0

data/.gitignore

@@ -0,0 +1,8 @@
+.rvmrc
+.bundle
+weeter.pid
+weeter.conf
+log/*.log
+vendor/bundle
+.DS_Store
+Gemfile.lock

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# The gem's dependencies are specified in weeter.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,9 @@
+The MIT License (MIT)
+Portions copyright (c) 2010-2011 Weplay, Inc.
+Portions copyright (c) 2011 Yapp, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,98 @@
+Weeter is a tireless worker who accepts a set of Twitter users to follow and terms to track, subscribes using Twitter's streaming API, and notifies your app with each new tweet (or tweet deletion).
+
+Supported strategies for tweet notification include HTTP (issue a POST to your app) and Resque (queue a job). Weeter is extensible with other notification strategies.
+
+Status
+======
+Alpha. A previous version of this code has been in production for some time. It has been substantially refactored and will be battle-tested soon.
+
+Getting set up
+==============
+
+    $ bundle install
+
+Make a copy of the weeter.conf.example file named weeter.conf. Twitter configuration, client app configuration and weeter configuration are defined in separate
+blocks. To configure how you connect to Twitter (basic auth or oauth), modify the twitter section of the configuration. 
+
+To configure how weeter connects to your client app, modify the client app configuration section:
+
+Notifications
+-------------
+
+* *notification_plugin*: A symbol matching the underscorized name of the NotificationPlugin subclass to use. Current options are :http and :resque 
+
+For option :http, also provide the following:
+
+* *oauth*: See the conf file for an example
+
+* *publish_url*: The URL to which new tweets should be posted. Request will be sent with POST method. Example body:
+    `id=1111&twitter_user_id=19466709&text=Wassup`
+* *delete_url*: The URL to which data about deleted tweets should be posted. Request will be sent with DELETE method. Example body:
+    `id=1111&twitter_user_id=19466709`
+
+For option :resque, provide the following:
+
+* *queue*: Name of the queue to add the job to
+
+* *redis_uri*: Redis connection string
+
+Subscriptions
+-------------
+
+* *subscription_plugin*: A symbol matching the underscorized name of the SubscrptionsPlugin subclass to use. Current options are :http and :redis 
+
+For option :http, also provide the following:
+
+* *oauth*: See the conf file for an example
+
+* *subscriptions_url*: The URL at which to find JSON describing the Twitter users to follow (maximum 5,000 at the default API access level) and the terms to track (maximum 400 at the default API access level). Example content:
+    `{"follow":"19466709", "759251"},{"track":"#lolcats","#bieber"}`
+
+* *subscription_updates_port*: The port Weeter should listen on for HTTP connections. If you have changes to your subscriptions data, POST the full JSON to the weeter's root URL. This will trigger weeter to reconnect to Twitter with the updated filters in place.
+
+For option :redis, also provide the following:
+
+* *subscriptions_key*: The Redis key at which the Weeter can find JSON describing the Twitter users to follow (maximum 5,000 at the default API access level) and the terms to track (maximum 400 at the default API access level). Example content:
+    `{"follow":"19466709", "759251"},{"track":"#lolcats","#bieber"}`
+
+* *subscriptions_changed_channel*: The Redis publish/subscribe channel to subscribe to in order to be notified that the subscriptions have changed. When your app has an updated set of subscriptions, it should update the _subscriptions_key_ and publish a "CHANGED" message to this channel. Weeter will then retrieve an updated set of subscriptions from Redis and reconnect to twitter.
+
+* *redis_uri*: Redis connection string
+
+Running weeter
+==============
+
+Weeter can be run using the weeter executable installed by the gem.
+
+Running Weeter as a daemon
+--------------------------
+The gem also installs a weeter_control executable that can be used to start Weeter as a daemon
+
+    $ bin/weeter_control start
+
+This starts Weeter as a daemon. For other commands and options, run:
+
+    $ bin/weeter_control --help
+
+
+
+Running specs
+=============
+
+    $ bundle exec rspec spec/
+
+TODO
+====
+- Better error reporting
+- Make tweet filtering strategy (re-tweets, replies, etc.) more configurable
+- Add specs for plugins
+- Extract plugins into separate gems
+- integration tests
+
+Credits
+======
+Thanks to Weplay for initial development and open sourcing Weeter. In particular, credit goes to Noah Davis and Joey Aghion. Further development by Luke Melia at Yapp.
+
+License
+=======
+Weeter is available under the terms of the MIT License http://www.opensource.org/licenses/mit-license.php

data/Rakefile

@@ -0,0 +1,2 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks

data/bin/weeter

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+$:.unshift File.expand_path("../../lib", __FILE__)
+require 'weeter'
+
+Weeter::Cli.new(ARGV.clone).run

data/bin/weeter_control

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'bundler/setup'
+require 'daemons'
+
+Daemons.run(File.join(File.dirname(__FILE__), 'weeter'), :dir_mode => :system)

data/lib/weeter/cli.rb

@@ -0,0 +1,24 @@
+require 'optparse'
+
+module Weeter
+  class Cli
+
+    def initialize(args)
+      @configuration_file = File.join(File.dirname(__FILE__), '..', '..', 'weeter.conf')
+      args.options do |opts|
+        opts.banner = "Usage: #{$0} [options]"
+        opts.on("-c", "--configuration=filename", String,
+                "Specifies an executable ruby file containing weeter configuration",
+                "Default: weeter.conf") do |val|
+          @configuration_file = val
+        end
+      end.parse!
+    end
+
+    def run
+      load @configuration_file
+      Weeter::Runner.new(Configuration.instance).start
+    end
+
+  end
+end

data/lib/weeter/configuration/client_app_config.rb

@@ -0,0 +1,12 @@
+require "singleton"
+require 'hashie'
+
+module Weeter
+  class Configuration
+    class ClientAppConfig < Hashie::Mash
+      def subscription_updates_port
+        self['subscription_updates_port'] || 7337
+      end
+    end
+  end
+end

data/lib/weeter/configuration/twitter_config.rb

@@ -0,0 +1,21 @@
+require "singleton"
+
+module Weeter
+  class Configuration
+    class TwitterConfig
+      include Singleton
+      attr_accessor :basic_auth, :oauth
+
+      def auth_options
+        if oauth
+          {:oauth => oauth}
+        else
+          username = basic_auth[:username]
+          password = basic_auth[:password]
+          {:auth => "#{username}:#{password}"}
+        end
+      end
+    end
+    
+  end
+end

data/lib/weeter/configuration.rb

@@ -0,0 +1,22 @@
+require "singleton"
+require "weeter/configuration/client_app_config"
+require "weeter/configuration/twitter_config"
+ 
+module Weeter
+
+  class Configuration
+    include Singleton
+    attr_accessor :log_path
+
+    def twitter
+      yield Configuration::TwitterConfig.instance if block_given?
+      Configuration::TwitterConfig.instance
+    end
+    
+    def client_app
+      @client_app_config ||= Configuration::ClientAppConfig.new
+      yield @client_app_config if block_given?
+      @client_app_config
+    end
+  end
+end

data/lib/weeter/plugins/lib/oauth_http.rb

@@ -0,0 +1,40 @@
+require 'em-http'
+require 'simple_oauth'
+
+module Weeter
+  module Plugins
+    module Net
+      class OauthHttp
+        def self.get(config, url, params = {})
+          request(config, :get, url, params)
+        end
+    
+        def self.put(config, url, params = {})
+          request(config, :put, url, params)
+        end
+
+        def self.post(config, url, params = {})
+          request(config, :post, url, params)
+        end
+    
+        def self.delete(config, url, params = {})
+          request(config, :delete, url, params)
+        end
+    
+        def self.request(config, method, url, params = {})
+          if method == :post
+            request_options = {:body => params}
+          else
+            request_options = {:query => params}
+          end
+          request_options.merge!(:head => {"Authorization" => oauth_header(config, url, params, method.to_s.upcase)}) if config.oauth
+          EM::HttpRequest.new(url).send(method, request_options)
+        end
+
+        def self.oauth_header(config, uri, params, http_method)
+          ::SimpleOauth::Header.new(http_method, uri, params, config.oauth).to_s
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/lib/redis.rb

@@ -0,0 +1,17 @@
+require 'em-hiredis'
+
+module Weeter
+  module Plugins
+    module Net
+      module Redis
+        def create_redis_client
+          @redis ||= begin
+            redis = EM::Hiredis.connect(@config.redis_uri)
+            redis.callback { Weeter.logger.info "Connected to Redis" }
+            redis
+          end
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/notification/http.rb

@@ -0,0 +1,26 @@
+module Weeter
+  module Plugins
+    module Notification
+      class Http
+        def initialize(client_app_config)
+          @config = client_app_config
+        end
+
+        def publish_tweet(tweet_item)
+          id = tweet_item['id_str']
+          text = tweet_item['text']
+          user_id = tweet_item['user']['id_str']
+          Weeter.logger.info("Publishing tweet #{id} from user #{user_id}: #{text}")
+          Weeter::Plugins::Net::OauthHttp.post(@config, @config.publish_url, {:id => id, :text => text, :twitter_user_id => user_id})
+        end
+
+        def delete_tweet(tweet_item)
+          id = tweet_item['delete']['status']['id'].to_s
+          user_id = tweet_item['delete']['status']['user_id'].to_s
+          Weeter.logger.info("Deleting tweet #{id} for user #{user_id}")
+          Weeter::Plugins::Net::OauthHttp.delete(@config, @config.delete_url, {:id => id, :twitter_user_id => user_id})
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/notification/resque.rb

@@ -0,0 +1,39 @@
+module Weeter
+  module Plugins
+    module Notification
+      class Resque
+        include Weeter::Plugins::Net::Redis
+        
+        def initialize(client_app_config)
+          @config = client_app_config
+        end
+
+        def publish_tweet(tweet_item)
+          resque_job = %Q|{"class":"WeeterPublishTweetJob","args":[#{tweet_item.to_json}]}|
+          Weeter.logger.info("Publishing tweet #{tweet_item['id']} from user #{tweet_item['user']['id_str']}: #{tweet_item['text']}")
+          enqueue(resque_job)
+        end
+
+        def delete_tweet(tweet_item)
+          resque_job = %Q|{"class":"WeeterDeleteTweetJob","args":[#{tweet_item.to_json}]}|
+          Weeter.logger.info("Deleting tweet #{tweet_item['id']} for user #{tweet_item['user']['id_str']}")
+          enqueue(resque_job)
+        end
+
+      protected
+      
+        def redis
+          @redis ||= create_redis_client
+        end
+      
+        def enqueue(job)
+          redis.rpush(queue_key, job)
+        end
+
+        def queue_key
+          "resque:queue:#{@config.queue}"
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/notification_plugin.rb

@@ -0,0 +1,26 @@
+require 'weeter/plugins/notification/http'
+require 'weeter/plugins/notification/resque'
+
+require 'active_support/core_ext/string/inflections'
+require 'active_support/core_ext/module/delegation'
+
+module Weeter
+  module Plugins
+    class NotificationPlugin
+      delegate :publish_tweet, :to => :configured_plugin
+      delegate :delete_tweet, :to => :configured_plugin
+      
+      def initialize(client_app_config)
+        @config = client_app_config
+      end
+      
+    protected
+      def configured_plugin
+        @configured_plugin ||= begin
+          Weeter.logger.info("Using #{@config.notification_plugin} notification plugin")
+          Notification.const_get(@config.notification_plugin.to_s.camelize).new(@config)
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/subscription/http.rb

@@ -0,0 +1,45 @@
+require 'evma_httpserver'
+require 'multi_json'
+
+module Weeter
+  module Plugins
+    module Subscription
+      class Http
+        def initialize(client_app_config)
+          @config = client_app_config
+        end
+
+        def get_initial_filters(&block)
+          http = Weeter::Plugins::Net::OauthHttp.get(@config, @config.subscriptions_url)
+          http.callback {
+            filter_params = {}
+            if http.response_header.status == 200
+              yield MultiJson.decode(http.response)
+            else
+              Weeter.logger.error "Initial filters request failed with response code #{http.response_header.status}."
+              yield
+            end
+          }
+        end
+
+        def listen_for_filter_update(tweet_consumer)
+          EM.start_server('localhost', @config.subscription_updates_port, UpdateServer) do |conn|
+            conn.tweet_consumer = tweet_consumer
+          end
+        end
+        
+        class UpdateServer < EM::Connection
+          include EM::HttpServer
+          attr_accessor :tweet_consumer
+
+          def process_http_request
+            Weeter.logger.info("Reconnecting Twitter stream")
+            filter_params = MultiJson.decode(@http_post_content)
+            tweet_consumer.reconnect(filter_params)
+            EM::DelegatedHttpResponse.new(self).send_response
+          end
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins/subscription/redis.rb

@@ -0,0 +1,46 @@
+require 'multi_json'
+
+module Weeter
+  module Plugins
+    module Subscription
+      class Redis
+        include Weeter::Plugins::Net::Redis
+        
+        def initialize(client_app_config)
+          @config = client_app_config
+        end
+
+        def get_initial_filters(&block)
+          redis.get(@config.subscriptions_key) do |value|
+            if value.nil?
+              raise "Expected to find subscription data at redis key #{@config.subscriptions_key}"
+            end
+            yield MultiJson.decode(value)
+          end
+        end
+
+        def listen_for_filter_update(tweet_consumer)
+          pub_sub_redis.subscribe(@config.subscriptions_changed_channel)
+          pub_sub_redis.on(:message) do |channel, message|
+            Weeter.logger.info [:message, channel, message]
+            Weeter.logger.info("Reconnecting Twitter stream")
+            get_initial_filters do |filter_params|
+              tweet_consumer.reconnect(filter_params)
+            end
+          end
+        end
+        
+      protected
+        
+        def redis
+          @redis ||= create_redis_client
+        end
+
+        def pub_sub_redis
+          @pub_sub_redis ||= create_redis_client
+        end
+        
+      end
+    end
+  end
+end

data/lib/weeter/plugins/subscription_plugin.rb

@@ -0,0 +1,26 @@
+require 'weeter/plugins/subscription/http'
+require 'weeter/plugins/subscription/redis'
+
+require 'active_support/core_ext/string/inflections'
+require 'active_support/core_ext/module/delegation'
+
+module Weeter
+  module Plugins
+    class SubscriptionPlugin
+      delegate :get_initial_filters, :to => :configured_plugin
+      delegate :listen_for_filter_update, :to => :configured_plugin
+      
+      def initialize(client_app_config)
+        @config = client_app_config
+      end
+    
+    protected
+      def configured_plugin
+        @configured_plugin ||= begin
+          Weeter.logger.info("Using #{@config.subscription_plugin} subscription plugin")
+          Subscription.const_get(@config.subscription_plugin.to_s.camelize).new(@config)
+        end
+      end
+    end
+  end
+end

data/lib/weeter/plugins.rb

@@ -0,0 +1,4 @@
+require 'weeter/plugins/lib/oauth_http'
+require 'weeter/plugins/lib/redis'
+require 'weeter/plugins/notification_plugin'
+require 'weeter/plugins/subscription_plugin'

data/lib/weeter/runner.rb

@@ -0,0 +1,40 @@
+require 'em-http'
+
+module Weeter
+  class Runner
+
+    def initialize(config)
+      @config = config
+      Weeter.logger.info("Starting weeter with configuration: #{@config.inspect}")
+    end
+
+    def start
+      EM.run {
+        subscription_plugin.get_initial_filters do |filter_params|
+          Weeter.logger.info("Connecting to twitter with initial filters")
+          tweet_consumer.connect(filter_params)
+          subscription_plugin.listen_for_filter_update(tweet_consumer)
+
+          trap('TERM') do
+            Weeter.logger.info("Stopping weeter")
+            EM.stop if EM.reactor_running?
+          end
+        end
+      }
+    end
+
+  protected
+  
+    def notification_plugin
+      @notification_plugin ||= Weeter::Plugins::NotificationPlugin.new(@config.client_app)
+    end
+
+    def subscription_plugin
+      @subscription_plugin ||= Weeter::Plugins::SubscriptionPlugin.new(@config.client_app)
+    end
+
+    def tweet_consumer
+      @tweet_consumer ||= Weeter::Twitter::TweetConsumer.new(@config.twitter, notification_plugin)
+    end
+  end
+end

data/lib/weeter/tasks.rb

@@ -0,0 +1,16 @@
+# require 'weeter/tasks'
+# will give you the weeter tasks
+
+namespace :weeter do
+  task :setup do
+    require 'weeter'
+    # extend this task to set the config path
+  end
+
+  desc "Start Weeter"
+  task :start => :setup do
+    configuration_file = ENV['WEETER_CONFIG_PATH']
+    load configuration_file
+    Weeter::Runner.new(Weeter::Configuration.instance).start
+  end
+end

data/lib/weeter/twitter/tweet_consumer.rb

@@ -0,0 +1,67 @@
+require 'twitter/json_stream'
+require 'multi_json'
+
+module Weeter
+  module Twitter
+    class TweetConsumer
+
+      def initialize(twitter_config, notifier)
+        @config = twitter_config
+        @notifier = notifier
+      end
+
+      def connect(filter_params)
+        filter_params = clean_filter_params(filter_params)
+        connect_options = {:ssl => true, :params => filter_params, :method => 'POST'}.merge(@config.auth_options)
+        @stream = ::Twitter::JSONStream.connect(connect_options)
+
+        @stream.each_item do |item|
+          begin
+            tweet_item = TweetItem.new(MultiJson.decode(item))
+
+            if tweet_item.deletion?
+              @notifier.delete_tweet(tweet_item)
+            elsif tweet_item.publishable?
+              @notifier.publish_tweet(tweet_item)
+            else
+              ignore_tweet(tweet_item)
+            end
+          rescue => ex
+            Weeter.logger.error("Twitter stream tweet exception: #{ex.class.name}: #{ex.message}")
+          end
+        end
+
+        @stream.on_error do |msg|
+          Weeter.logger.error("Twitter stream error: #{msg}. Connect options were #{connect_options.inspect}")
+        end
+
+        @stream.on_max_reconnects do |timeout, retries|
+          Weeter.logger.error("Twitter stream max-reconnects reached: timeout=#{timeout}, retries=#{retries}")
+        end
+      end
+
+      def reconnect(filter_params)
+        @stream.stop
+        connect(filter_params)
+      end
+
+    protected
+    
+      def clean_filter_params(p)
+        return {} if p.nil?
+        cleaned_params = {}
+        cleaned_params['follow'] = p['follow'] if (p['follow'] || []).any?
+        cleaned_params['track'] = p['track'] if (p['track'] || []).any?
+        cleaned_params
+      end
+    
+      def ignore_tweet(tweet_item)
+        id = tweet_item['id_str']
+        text = tweet_item['text']
+        user_id = tweet_item['user']['id_str']
+        Weeter.logger.info("Ignoring tweet #{id} from user #{user_id}: #{text}")
+      end
+
+    end
+  end
+end

data/lib/weeter/twitter/tweet_item.rb

@@ -0,0 +1,35 @@
+require 'multi_json'
+
+module Weeter
+
+  class TweetItem
+    def initialize(tweet_hash)
+      @tweet_hash = tweet_hash
+    end
+
+    def deletion?
+      !@tweet_hash['delete'].nil?
+    end
+    
+    def retweeted?
+      !@tweet_hash['retweeted_status'].nil? || @tweet_hash['text'] =~ /^RT @/i
+    end
+    
+    def reply?
+      !@tweet_hash['in_reply_to_user_id_str'].nil? || @tweet_hash['text'] =~ /^@/
+    end
+    
+    def publishable?
+      !retweeted? && !reply?
+    end
+    
+    def [](val)
+      @tweet_hash[val]
+    end
+    
+    def to_json
+      MultiJson.encode(@tweet_hash)
+    end
+  end
+
+end

data/lib/weeter/twitter.rb

@@ -0,0 +1,2 @@
+require 'weeter/twitter/tweet_consumer'
+require 'weeter/twitter/tweet_item'