tweetstream 1.1.5 → 2.0.0

data/CHANGELOG.md

@@ -1,3 +1,19 @@
+Version 2.0.0
+=============
+
+* Added Site Stream support
+* Switched to [em-twitter](https://github.com/spagalloco/em-twitter) for underlying streaming lib
+* Switched to Twitter gem objects instead of custom hashes, see [47e5cd3d21a9562b3d959bc231009af460b37567](https://github.com/intridea/tweetstream/commit/47e5cd3d21a9562b3d959bc231009af460b37567) for details (sferik)
+* Made OAuth the default authentication method
+* Removed on_interval callback
+* Removed parser configuration option
+
+Version 1.1.5
+=============
+
+* Added support for the scrub_geo response (augustj)
+* Update multi_json and twitter-stream version dependencies
+
 Version 1.1.4
 =============
 
@@ -27,8 +43,9 @@ Version 1.1.0
 * OAuth authentication
 * User Stream support
 * Removed swappable JSON backend support for MultiJson
-* Added epoll and kqueue EventMachine support
-* Added on___interval and on_inited callbacks
+* Added EventMachine epoll and kqueue support
+* Added on_interval callback
+* Added on_inited callback
 
 Version 1.0.5
 =============

data/LICENSE.md

@@ -1,4 +1,4 @@
-Copyright (c) 2011 Intridea, Inc.
+Copyright (c) 2012 Intridea, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -16,12 +16,11 @@ Using TweetStream is quite simple:
 require 'tweetstream'
 
 TweetStream.configure do |config|
-  config.consumer_key = 'abcdefghijklmnopqrstuvwxyz'
-  config.consumer_secret = '0123456789'
-  config.oauth_token = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_key       = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_secret    = '0123456789'
+  config.oauth_token        = 'abcdefghijklmnopqrstuvwxyz'
   config.oauth_token_secret = '0123456789'
-  config.auth_method = :oauth
-  config.parser   = :yajl
+  config.auth_method        = :oauth
 end
 
 # This will pull a sample of all tweets based on
@@ -48,20 +47,108 @@ TweetStream::Client.new.follow(14252, 53235) do |status|
 end
 ```
 
-The methods available to TweetStream::Client will be kept in parity
+The methods available to TweetStream::Client are kept in parity
 with the methods available on the Streaming API wiki page.
 
+## Changes in 2.0
+
+TweetStream 2.0 introduces a number of requested features and bug fixes. For the complete list refer to the [changelog](https://github.com/intridea/tweetstream/blob/master/CHANGELOG.md#version-200). Notable
+additions in 2.0 include:
+
+### OAuth
+
+OAuth is now the default authentication method.  Both userstreams and Site 
+Streams exclusively work with OAuth.  TweetStream still supports Basic Auth,
+however it is no longer the default.  If you are still using Basic Auth, you
+should plan to move to OAuth as soon as possible.
+
+### Site Stream Support
+
+Site Streams are now fully supported, including the connection management functionality.
+
+### Compatablity with the Twitter gem
+
+TweetStream now emits objects from the [Twitter gem](https://github.com/jnunemaker/twitter) instead of custom hashes. These objects are already defined in the `twitter` gem and are superior to the custom objects in the following ways:
+
+1. Object equivalence (`#==` returns true if `#id`s are the same).
+2. The `#created_at` method returns a `Date` instead of a `String`.
+3. Allows boolean methods to be called with a question mark (e.g.
+   `User#protected?`)
+
+Additionally, any new features that are added to objects in the
+`twitter` gem (e.g. identity map) will be automatically inherited by TweetStream.
+
+### em-twitter
+
+We've replaced the underlying gem that connects to the streaming API. [twitter-stream](https://github.com/voloko/twitter-stream) has been replaced with [em-twitter](https://github.com/spagalloco/em-twitter).
+It offers functionality parity with twitter-stream while also supporting several new features.
+
+### Removal of on_interval callback
+
+We have removed the `on_interval` callback.  If you require interval-based timers, it is possible to run 
+TweetStream inside an already running EventMachine reactor in which you can define `EM::Timer` or `EM::PeriodicTimer`
+for time-based operations:
+
+```ruby
+EM.run do
+  client = TweetStream::Client.new
+  
+  EM::PeriodicTimer.new(10) do
+    # do something on an interval
+  end
+end
+```
+
+### Additional Notes
+
+The parser configuration method has been removed as MultiJson automatically detects existing parsers.
+
 ## Using the Twitter Userstream
 
-Using the Twitter userstream works similarly to the regular streaming, except you use the userstream method.
+Using the Twitter userstream works similarly to regular streaming, except you use the `userstream` method.
 
 ```ruby
 # Use 'userstream' to get message from your stream
-TweetStream::Client.new.userstream do |status|
+client = TweetStream::Client.new
+
+client.userstream do |status|
   puts status.text
 end
 ```
 
+## Using Twitter Site Streams
+
+```ruby
+client = TweetStream::Client.new
+
+client.sitestream(['115192457'], :followings => true) do |status|
+  puts status.inspect
+end
+```
+
+Once connected, you can [control the Site Stream connection](https://dev.twitter.com/docs/streaming-apis/streams/site/control):
+
+```ruby
+# add users to the stream
+client.control.add_user('2039761')
+
+# remove users from the stream
+client.control.remove_user('115192457')
+
+# obtain a list of followings of users in the stream
+client.control.friends_ids('115192457') do |friends|
+  # do something
+end
+
+# obtain the current state of the stream
+client.control.info do |info| 
+  # do something
+end
+```
+
+Note that per Twitter's documentation, connection management features are not 
+immediately available when connected
+
 You also can use method hooks for both regular timeline statuses and direct messages.
 
 ```ruby
@@ -80,11 +167,10 @@ end
 client.userstream
 ```
 
-## Configuration and Changes in 1.1.0
+## Authentication
 
-As of version 1.1.0.rc1 TweetStream supports OAuth.  Please note that in order
-to support OAuth, the `TweetStream::Client` initializer no longer accepts a
-username/password.  `TweetStream::Client` now accepts a hash:
+TweetStream supports OAuth and Basic Auth.  `TweetStream::Client` now accepts 
+a hash:
 
 ```ruby
 TweetStream::Client.new(:username => 'you', :password => 'pass')
@@ -94,12 +180,11 @@ Alternatively, you can configure TweetStream via the configure method:
 
 ```ruby
 TweetStream.configure do |config|
-  config.consumer_key = 'cVcIw5zoLFE2a4BdDsmmA'
-  config.consumer_secret = 'yYgVgvTT9uCFAi2IuscbYTCqwJZ1sdQxzISvLhNWUA'
-  config.oauth_token = '4618-H3gU7mjDQ7MtFkAwHhCqD91Cp4RqDTp1AKwGzpHGL3I'
+  config.consumer_key       = 'cVcIw5zoLFE2a4BdDsmmA'
+  config.consumer_secret    = 'yYgVgvTT9uCFAi2IuscbYTCqwJZ1sdQxzISvLhNWUA'
+  config.oauth_token        = '4618-H3gU7mjDQ7MtFkAwHhCqD91Cp4RqDTp1AKwGzpHGL3I'
   config.oauth_token_secret = 'xmc9kFgOXpMdQ590Tho2gV7fE71v5OmBrX8qPGh7Y'
-  config.auth_method = :oauth
-  config.parser   = :yajl
+  config.auth_method        = :oauth
 end
 ```
 
@@ -107,31 +192,21 @@ If you are using Basic Auth:
 
 ```ruby
 TweetStream.configure do |config|
-  config.username = 'username'
-  config.password = 'password'
-  config.auth_method = :basic
-  config.parser   = :yajl
+  config.username     = 'username'
+  config.password     = 'password'
+  config.auth_method  = :basic
 end
 ```
 
 TweetStream assumes OAuth by default.  If you are using Basic Auth, it is recommended
 that you update your code to use OAuth as Twitter is likely to phase out Basic Auth
-support.
-
-## Swappable JSON Parsing
-
-As of version 1.1, TweetStream supports swappable JSON backends via MultiJson. You can
-specify a parser during configuration:
+support.  Basic Auth is only available for public streams as User Stream and Site Stream 
+functionality [only support OAuth](https://dev.twitter.com/docs/streaming-apis/connecting#Authentication).
 
-```ruby
-# Parse tweets using Yajl-Ruby
-TweetStream.configure do |config|
-  config.parser   = :yajl
-end
-```
+## Parsing JSON 
 
-Available options are `:oj`, `:yajl`, `:json_gem`, `:json_pure`, and
-`:ok_json`.
+TweetStream supports swappable JSON backends via MultiJson. Simply require your preferred 
+JSON parser and it will be used to parse responses.
 
 ## Handling Deletes and Rate Limitations
 
@@ -154,7 +229,7 @@ end
 @client.track('intridea')
 ```
 
-The on_delete and on_limit methods can also be chained, like so:
+The on_delete and on_limit methods can also be chained:
 
 ```ruby
 TweetStream::Client.new.on_delete{ |status_id, user_id|
@@ -248,10 +323,6 @@ end
 If you put the above into a script and run the script with `ruby scriptname.rb`,
 you will see a list of daemonization commands such as start, stop, and run.
 
-## TODO
-
-* SiteStream support
-
 ## Contributing
 
 * Fork the project.
@@ -262,9 +333,11 @@ you will see a list of daemonization commands such as start, stop, and run.
 
 ## Contributors
 
-* Michael Bleigh (initial gem)
-* Steve Agalloco (current maintainer)
+* [Michael Bleigh](https://github.com/mbleigh) (initial gem)
+* [Steve Agalloco](https://github.com/spagalloco) (current maintainer)
+* [Erik Michaels-Ober](https://github.com/sferik) (current maintainer)
+* [Countless others](https://github.com/intridea/tweetstream/graphs/contributors)
 
 ## Copyright
 
-Copyright (c) 2011 Intridea, Inc. (http://www.intridea.com/). See [LICENSE](https://github.com/intridea/tweetstream/blob/master/LICENSE.md) for details.
+Copyright (c) 2012 Intridea, Inc. (http://www.intridea.com/). See [LICENSE](https://github.com/intridea/tweetstream/blob/master/LICENSE.md) for details.

data/Rakefile

@@ -1,5 +1,3 @@
-#!/usr/bin/env rake
-
 require 'bundler'
 Bundler::GemHelper.install_tasks
 

data/examples/growl_daemon.rb

@@ -3,14 +3,14 @@ require 'tweetstream'
 require 'growl'
 
 tracks = 'yankees'
-puts "Starting a GrowlTweet to track: #{tracks}"
+puts "Starting a TweetStream Daemon to track: #{tracks}"
 
 TweetStream.configure do |config|
-  config.consumer_key = 'abcdefghijklmnopqrstuvwxyz'
-  config.consumer_secret = '0123456789'
-  config.oauth_token = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_key       = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_secret    = '0123456789'
+  config.oauth_token        = 'abcdefghijklmnopqrstuvwxyz'
   config.oauth_token_secret = '0123456789'
-  config.auth_method = :oauth
+  config.auth_method        = :oauth
 end
 
 TweetStream::Daemon.new('tracker').track(tracks) do |status|

data/examples/oauth.rb

@@ -2,12 +2,11 @@ require 'yajl'
 require 'tweetstream'
 
 TweetStream.configure do |config|
-  config.consumer_key = 'abcdefghijklmnopqrstuvwxyz'
-  config.consumer_secret = '0123456789'
-  config.oauth_token = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_key       = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_secret    = '0123456789'
+  config.oauth_token        = 'abcdefghijklmnopqrstuvwxyz'
   config.oauth_token_secret = '0123456789'
-  config.auth_method = :oauth
-  config.parser   = :yajl
+  config.auth_method        = :oauth
 end
 
 client = TweetStream::Client.new

data/examples/sitestream.rb

@@ -0,0 +1,40 @@
+require 'yajl'
+require 'tweetstream'
+
+TweetStream.configure do |config|
+  config.consumer_key       = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_secret    = '0123456789'
+  config.oauth_token        = 'abcdefghijklmnopqrstuvwxyz'
+  config.oauth_token_secret = '0123456789'
+  config.auth_method        = :oauth
+end
+
+EM.run do
+
+  client = TweetStream::Client.new
+
+  client.on_error do |error|
+    puts error
+  end
+
+  client.sitestream([user_id], :followings => true) do |status|
+    puts status.inspect
+  end
+
+  EM::Timer.new(60) do
+    client.control.add_user(user_id_to_add)
+    client.control.info { |i| puts i.inspect }
+  end
+
+  EM::Timer.new(75) do
+    client.control.friends_ids(user_id) do |friends|
+      puts friends.inspect
+    end
+  end
+
+  EM::Timer.new(90) do
+    client.control.remove_user(user_id_to_remove)
+    client.control.info { |i| puts i.inspect }
+  end
+
+end

data/examples/userstream.rb

@@ -2,12 +2,11 @@ require 'yajl'
 require 'tweetstream'
 
 TweetStream.configure do |config|
-  config.consumer_key = 'abcdefghijklmnopqrstuvwxyz'
-  config.consumer_secret = '0123456789'
-  config.oauth_token = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_key       = 'abcdefghijklmnopqrstuvwxyz'
+  config.consumer_secret    = '0123456789'
+  config.oauth_token        = 'abcdefghijklmnopqrstuvwxyz'
   config.oauth_token_secret = '0123456789'
-  config.auth_method = :oauth
-  config.parser   = :yajl
+  config.auth_method        = :oauth
 end
 
 client = TweetStream::Client.new

data/lib/tweetstream.rb

@@ -1,9 +1,5 @@
 require 'tweetstream/configuration'
 require 'tweetstream/client'
-require 'tweetstream/hash'
-require 'tweetstream/status'
-require 'tweetstream/direct_message'
-require 'tweetstream/user'
 require 'tweetstream/error'
 require 'tweetstream/daemon'
 

data/lib/tweetstream/client.rb

@@ -1,8 +1,9 @@
-require 'uri'
 require 'cgi'
 require 'eventmachine'
-require 'twitter/json_stream'
 require 'multi_json'
+require 'twitter'
+require 'em-twitter'
+require 'uri'
 
 module TweetStream
   # Provides simple access to the Twitter Streaming API (http://apiwiki.twitter.com/Streaming-API-Documentation)
@@ -11,7 +12,7 @@ module TweetStream
   #
   # Basic usage of the library is to call one of the provided
   # methods and provide a block that will perform actions on
-  # a yielded TweetStream::Status. For example:
+  # a yielded Twitter::Status. For example:
   #
   #     TweetStream::Client.new.track('fail') do |status|
   #       puts "[#{status.user.screen_name}] #{status.text}"
@@ -23,22 +24,16 @@ module TweetStream
 
     # @private
     attr_accessor *Configuration::VALID_OPTIONS_KEYS
-    attr_accessor :timer
+    attr_accessor :options
+    attr_reader :control_uri, :control, :stream
 
     # Creates a new API
     def initialize(options={})
-      options = TweetStream.options.merge(options)
+      self.options = options
+      merged_options = TweetStream.options.merge(options)
       Configuration::VALID_OPTIONS_KEYS.each do |key|
-        send("#{key}=", options[key])
+        send("#{key}=", merged_options[key])
       end
-
-      # Ensure the json parser can be properly loaded
-      json_parser
-    end
-
-    # Get the JSON parser class for this client.
-    def json_parser
-      parser_from(parser)
     end
 
     # Returns all public statuses. The Firehose is not a generally
@@ -124,7 +119,20 @@ module TweetStream
 
     # Make a call to the userstream api for currently authenticated user
     def userstream(&block)
-      start('', :extra_stream_parameters => {:host => "userstream.twitter.com", :path => "/2/user.json"}, &block)
+      stream_params = { :host => "userstream.twitter.com", :path => "/2/user.json" }
+      start('', :extra_stream_parameters => stream_params, &block)
+    end
+
+    # Make a call to the userstream api
+    def sitestream(user_ids = [], query_params = {}, &block)
+      stream_params = { :host => "sitestream.twitter.com", :path => '/2b/site.json' }
+      sitestream_params = {
+        :method => :post,
+        :follow => user_ids,
+        :extra_stream_parameters => stream_params
+      }
+      sitestream_params.merge!(:with => 'followings') if query_params[:followings]
+      start('', sitestream_params, &block)
     end
 
     # Set a Proc to be run when a deletion notice is received
@@ -258,7 +266,7 @@ module TweetStream
     # processing of the stream.
     #
     #     @client = TweetStream::Client.new
-    #     @client.on_timeline_message do |status|
+    #     @client.on_timeline_status do |status|
     #       # do something with the status
     #     end
     #
@@ -308,21 +316,19 @@ module TweetStream
       end
     end
 
-    # Set a Proc to be run on a regular interval
-    # independent of timeline status updates
+    # Set a Proc to be run when no data is received from the server
+    # and a stall occurs.  Twitter defines this to be 90 seconds.
     #
     #     @client = TweetStream::Client.new
-    #     @client.on_interval(20) do
-    #       # do something every 20 seconds
+    #     @client.on_no_data do
+    #       # Make note of no data, possi
     #     end
-    #
-    def on_interval(time_interval=nil, &block)
+    def on_no_data_received(&block)
       if block_given?
-        @on_interval_time = time_interval
-        @on_interval_proc = block
+        @on_no_data = block
         self
       else
-        [@on_interval_time, @on_interval_proc]
+        @on_no_data
       end
     end
 
@@ -352,6 +358,7 @@ module TweetStream
       direct_message_proc = query_parameters.delete(:direct_message) || self.on_direct_message
       timeline_status_proc = query_parameters.delete(:timeline_status) || self.on_timeline_status
       anything_proc = query_parameters.delete(:anything) || self.on_anything
+      no_data_proc = query_parameters.delete(:no_data_received) || self.on_no_data_received
 
       params = normalize_filter_parameters(query_parameters)
 
@@ -360,51 +367,42 @@ module TweetStream
       uri = method == :get ? build_uri(path, params) : build_uri(path)
 
       stream_params = {
-          :path => uri,
-          :method => method.to_s.upcase,
-          :user_agent => user_agent,
-          :on_inited => inited_proc,
-          :filters => params.delete(:track),
-          :params => params,
-          :ssl => true
-      }.merge(auth_params).merge(extra_stream_parameters)
-
-      if @on_interval_proc.is_a?(Proc)
-        interval = @on_interval_time || Configuration::DEFAULT_TIMER_INTERVAL
-        @timer = EventMachine.add_periodic_timer(interval) do
-          EventMachine.defer do
-            @on_interval_proc.call
-          end
-        end
-      end
-
-      @stream = Twitter::JSONStream.connect(stream_params)
-      @stream.each_item do |item|
+        :path => uri,
+        :method => method.to_s.upcase,
+        :user_agent => user_agent,
+        :on_inited => inited_proc,
+        :params => params
+      }.merge(extra_stream_parameters).merge(auth_params)
+
+      @stream = EM::Twitter::Client.connect(stream_params)
+      @stream.each do |item|
         begin
-          raw_hash = json_parser.decode(item)
+          hash = MultiJson.decode(item)
         rescue MultiJson::DecodeError
           error_proc.call("MultiJson::DecodeError occured in stream: #{item}") if error_proc.is_a?(Proc)
           next
         end
 
-        unless raw_hash.is_a?(::Hash)
+        unless hash.is_a?(::Hash)
           error_proc.call("Unexpected JSON object in stream: #{item}") if error_proc.is_a?(Proc)
           next
         end
 
-        hash = TweetStream::Hash.new(raw_hash)
-        if hash[:delete] && hash[:delete][:status]
-          delete_proc.call(hash[:delete][:status][:id], hash[:delete][:status][:user_id]) if delete_proc.is_a?(Proc)
-        elsif hash[:scrub_geo] && hash[:scrub_geo][:up_to_status_id]
-          scrub_geo_proc.call(hash[:scrub_geo][:up_to_status_id], hash[:scrub_geo][:user_id]) if scrub_geo_proc.is_a?(Proc)
-        elsif hash[:limit] && hash[:limit][:track]
-          limit_proc.call(hash[:limit][:track]) if limit_proc.is_a?(Proc)
-
-        elsif hash[:direct_message]
-          yield_message_to direct_message_proc, TweetStream::DirectMessage.new(hash[:direct_message])
-
-        elsif hash[:text] && hash[:user]
-          @last_status = TweetStream::Status.new(hash)
+        if hash['control'] && hash['control']['control_uri']
+          @control_uri = hash['control']['control_uri']
+          require 'tweetstream/site_stream_client'
+          @control = TweetStream::SiteStreamClient.new(@control_uri, options)
+          @control.on_error(&self.on_error)
+        elsif hash['delete'] && hash['delete']['status']
+          delete_proc.call(hash['delete']['status']['id'], hash['delete']['status']['user_id']) if delete_proc.is_a?(Proc)
+        elsif hash['scrub_geo'] && hash['scrub_geo']['up_to_status_id']
+          scrub_geo_proc.call(hash['scrub_geo']['up_to_status_id'], hash['scrub_geo']['user_id']) if scrub_geo_proc.is_a?(Proc)
+        elsif hash['limit'] && hash['limit']['track']
+          limit_proc.call(hash['limit']['track']) if limit_proc.is_a?(Proc)
+        elsif hash['direct_message']
+          yield_message_to direct_message_proc, Twitter::DirectMessage.new(hash['direct_message'])
+        elsif hash['text'] && hash['user']
+          @last_status = Twitter::Status.new(hash)
           yield_message_to timeline_status_proc, @last_status
 
           if block_given?
@@ -417,6 +415,19 @@ module TweetStream
                 yield @last_status, self
             end
           end
+        elsif hash['for_user']
+          @message = hash
+
+          if block_given?
+            # Give the block the option to receive either one
+            # or two arguments, depending on its arity.
+            case block.arity
+            when 1
+              yield @message
+            when 2
+              yield @message, self
+            end
+          end
         end
 
         yield_message_to anything_proc, hash
@@ -434,6 +445,10 @@ module TweetStream
         raise TweetStream::ReconnectError.new(timeout, retries)
       end
 
+      @stream.on_no_data_received do
+        no_data_proc.call if no_data_proc.is_a?(Proc)
+      end
+
       @stream
     end
 
@@ -454,11 +469,6 @@ module TweetStream
 
     protected
 
-    def parser_from(parser)
-      MultiJson.adapter = parser
-      MultiJson
-    end
-
     def build_uri(path, query_parameters = {}) #:nodoc:
       URI.parse("/1/#{path}.json#{build_query_parameters(query_parameters)}")
     end
@@ -488,19 +498,29 @@ module TweetStream
     end
 
     def auth_params
-      case auth_method
-        when :basic
-          return :auth => "#{username}:#{password}"
-        when :oauth
-          return :oauth => {
-              :consumer_key => consumer_key,
-              :consumer_secret => consumer_secret,
-              :access_key => oauth_token,
-              :access_secret => oauth_token_secret
-          }
+      if auth_method == :basic
+        { :basic => basic_auth_params }
+      else
+        { :oauth => oauth_params }
       end
     end
 
+    def basic_auth_params
+      {
+        :username => username,
+        :password => password
+      }
+    end
+
+    def oauth_params
+      {
+        :consumer_key => consumer_key,
+        :consumer_secret => consumer_secret,
+        :token => oauth_token,
+        :token_secret => oauth_token_secret
+      }
+    end
+
     def yield_message_to(procedure, message)
       if procedure.is_a?(Proc)
         case procedure.arity