jls-tweetstream 2.5.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,6 @@
1
+ --markup markdown
2
+ -
3
+ CHANGELOG.md
4
+ CONTRIBUTING.md
5
+ LICENSE.md
6
+ README.md
@@ -0,0 +1,76 @@
1
+ Version 2.4.0
2
+ =============
3
+ * Revert "use extract_options! from the Twitter gem"
4
+ * Reorganize development and test dependencies
5
+
6
+ Version 2.3.0
7
+ =============
8
+ * Added support for Site Stream friends list
9
+ * Update paths for API 1.1
10
+ * Added stall warning handling
11
+
12
+ Version 2.2.0
13
+ =============
14
+ * Change method to request_method
15
+
16
+ Version 2.1.0
17
+ =============
18
+ * Disable identity map to reduce memory usage
19
+ * Added options support to UserStreams
20
+
21
+ Version 2.0.1
22
+ =============
23
+ * Fixed Twitter gem objects
24
+ * Added on_unauthorized callback method (koenpunt)
25
+
26
+ Version 2.0.0
27
+ =============
28
+ * Added Site Stream support
29
+ * Switched to [em-twitter](https://github.com/spagalloco/em-twitter) for underlying streaming lib
30
+ * Switched to Twitter gem objects instead of custom hashes, see [47e5cd3d21a9562b3d959bc231009af460b37567](https://github.com/intridea/tweetstream/commit/47e5cd3d21a9562b3d959bc231009af460b37567) for details (sferik)
31
+ * Made OAuth the default authentication method
32
+ * Removed on_interval callback method
33
+ * Removed parser configuration option
34
+
35
+ Version 1.1.5
36
+ =============
37
+ * Added support for the scrub_geo response (augustj)
38
+ * Update multi_json and twitter-stream version dependencies
39
+
40
+ Version 1.1.4
41
+ =============
42
+ * Added Client#connect to start streaming inside an EM reactor (pelle)
43
+ * Added shutdown_stream to cleanly stop the stream (lud)
44
+ * Loosened multi_json dependency for Rails 3.2 compatibiltiy
45
+
46
+ Version 1.1.3
47
+ =============
48
+ * Added on_reconnect callback method
49
+
50
+ Version 1.1.2
51
+ =============
52
+ * Added support for statuses/links
53
+ * Client now checks that specified json_parser can be loaded during initialization
54
+
55
+ Version 1.1.1
56
+ =============
57
+ * Fix for 1.8.6 compatibility
58
+
59
+ Version 1.1.0
60
+ =============
61
+ * OAuth authentication
62
+ * User Stream support
63
+ * Removed swappable JSON backend support for MultiJson
64
+ * Added EventMachine epoll and kqueue support
65
+ * Added on_interval callback
66
+ * Added on_inited callback
67
+
68
+ Version 1.0.5
69
+ =============
70
+ * Force SSL to comply with
71
+
72
+ Version 1.0.0
73
+ =============
74
+ * Swappable JSON backend support
75
+ * Switches to use EventMachine instead of Yajl for the HTTP Stream
76
+ * Support reconnect and on_error
@@ -0,0 +1,9 @@
1
+ ## Contributing
2
+ * Fork the project.
3
+ * Make your feature addition or bug fix.
4
+ * Add tests for it. This is important so I don't break it in a future version
5
+ unintentionally.
6
+ * Commit, do not mess with rakefile, version, or history. (if you want to have
7
+ your own version, that is fine but bump version in a commit by itself I can
8
+ ignore when I pull)
9
+ * Send me a pull request. Bonus points for topic branches.
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2012-2013 Intridea, Inc.
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,392 @@
1
+ # TweetStream
2
+
3
+ [![Gem Version](https://badge.fury.io/rb/tweetstream.png)][gem]
4
+ [![Build Status](https://secure.travis-ci.org/intridea/tweetstream.png?branch=master)][travis]
5
+ [![Dependency Status](https://gemnasium.com/intridea/tweetstream.png?travis)][gemnasium]
6
+ [![Code Climate](https://codeclimate.com/github/intridea/tweetstream.png)][codeclimate]
7
+ [![Coverage Status](https://coveralls.io/repos/intridea/tweetstream/badge.png?branch=master)][coveralls]
8
+
9
+ [gem]: https://rubygems.org/gems/tweetstream
10
+ [travis]: http://travis-ci.org/intridea/tweetstream
11
+ [gemnasium]: https://gemnasium.com/intridea/tweetstream
12
+ [codeclimate]: https://codeclimate.com/github/intridea/tweetstream
13
+ [coveralls]: https://coveralls.io/r/intridea/tweetstream
14
+
15
+ TweetStream provides simple Ruby access to [Twitter's Streaming API](https://dev.twitter.com/docs/streaming-api).
16
+
17
+ **Note:** TweetStream does not currently work with Ruby 2.0, this is a [known issue](https://github.com/intridea/tweetstream/issues/117).
18
+
19
+ ## Installation
20
+
21
+ gem install tweetstream
22
+
23
+ ## Usage
24
+
25
+ Using TweetStream is quite simple:
26
+
27
+ ```ruby
28
+ require 'tweetstream'
29
+
30
+ TweetStream.configure do |config|
31
+ config.consumer_key = 'abcdefghijklmnopqrstuvwxyz'
32
+ config.consumer_secret = '0123456789'
33
+ config.oauth_token = 'abcdefghijklmnopqrstuvwxyz'
34
+ config.oauth_token_secret = '0123456789'
35
+ config.auth_method = :oauth
36
+ end
37
+
38
+ # This will pull a sample of all tweets based on
39
+ # your Twitter account's Streaming API role.
40
+ TweetStream::Client.new.sample do |status|
41
+ # The status object is a special Hash with
42
+ # method access to its keys.
43
+ puts "#{status.text}"
44
+ end
45
+ ```
46
+
47
+ You can also use it to track keywords or follow a given set of
48
+ user ids:
49
+
50
+ ```ruby
51
+ # Use 'track' to track a list of single-word keywords
52
+ TweetStream::Client.new.track('term1', 'term2') do |status|
53
+ puts "#{status.text}"
54
+ end
55
+
56
+ # Use 'follow' to follow a group of user ids (integers, not screen names)
57
+ TweetStream::Client.new.follow(14252, 53235) do |status|
58
+ puts "#{status.text}"
59
+ end
60
+ ```
61
+
62
+ The methods available to TweetStream::Client are kept in parity
63
+ with the methods available on the Streaming API wiki page.
64
+
65
+ ## Changes in 2.0
66
+
67
+ TweetStream 2.0 introduces a number of requested features and bug fixes. For
68
+ the complete list refer to the [changelog](CHANGELOG.md#version-200). Notable
69
+ additions in 2.0 include:
70
+
71
+ ### OAuth
72
+
73
+ OAuth is now the default authentication method. Both userstreams and Site
74
+ Streams exclusively work with OAuth. TweetStream still supports Basic Auth,
75
+ however it is no longer the default. If you are still using Basic Auth, you
76
+ should plan to move to OAuth as soon as possible.
77
+
78
+ ### Site Stream Support
79
+
80
+ Site Streams are now fully supported, including the connection management functionality.
81
+
82
+ ### Compatability with the Twitter gem
83
+
84
+ TweetStream now emits objects from the [Twitter gem](https://github.com/sferik/twitter) instead of custom hashes. These objects are already defined in the `twitter` gem and are superior to the custom objects in the following ways:
85
+
86
+ 1. Object equivalence (`#==` returns true if `#id`s are the same).
87
+ 2. The `#created_at` method returns a `Date` instead of a `String`.
88
+ 3. Allows boolean methods to be called with a question mark (e.g.
89
+ `User#protected?`)
90
+
91
+ Additionally, any new features that are added to objects in the
92
+ `twitter` gem (e.g. identity map) will be automatically inherited by TweetStream.
93
+
94
+ ### em-twitter
95
+
96
+ We've replaced the underlying gem that connects to the streaming API. [twitter-stream](https://github.com/voloko/twitter-stream) has been replaced with [em-twitter](https://github.com/spagalloco/em-twitter).
97
+ It offers functionality parity with twitter-stream while also supporting several new features.
98
+
99
+ ### Removal of on_interval callback
100
+
101
+ We have removed the `on_interval` callback. If you require interval-based timers, it is possible to run
102
+ TweetStream inside an already running EventMachine reactor in which you can define `EM::Timer` or `EM::PeriodicTimer`
103
+ for time-based operations:
104
+
105
+ ```ruby
106
+ EM.run do
107
+ client = TweetStream::Client.new
108
+
109
+ EM::PeriodicTimer.new(10) do
110
+ # do something on an interval
111
+ end
112
+ end
113
+ ```
114
+
115
+ ### Additional Notes
116
+
117
+ The parser configuration method has been removed as MultiJson automatically detects existing parsers.
118
+
119
+ ## Using the Twitter Userstream
120
+
121
+ Using the Twitter userstream works similarly to regular streaming, except you use the `userstream` method.
122
+
123
+ ```ruby
124
+ # Use 'userstream' to get message from your stream
125
+ client = TweetStream::Client.new
126
+
127
+ client.userstream do |status|
128
+ puts status.text
129
+ end
130
+ ```
131
+
132
+ ## Using Twitter Site Streams
133
+
134
+ ```ruby
135
+ client = TweetStream::Client.new
136
+
137
+ client.sitestream(['115192457'], :followings => true) do |status|
138
+ puts status.inspect
139
+ end
140
+ ```
141
+
142
+ Once connected, you can [control the Site Stream connection](https://dev.twitter.com/docs/streaming-apis/streams/site/control):
143
+
144
+ ```ruby
145
+ # add users to the stream
146
+ client.control.add_user('2039761')
147
+
148
+ # remove users from the stream
149
+ client.control.remove_user('115192457')
150
+
151
+ # obtain a list of followings of users in the stream
152
+ client.control.friends_ids('115192457') do |friends|
153
+ # do something
154
+ end
155
+
156
+ # obtain the current state of the stream
157
+ client.control.info do |info|
158
+ # do something
159
+ end
160
+ ```
161
+
162
+ Note that per Twitter's documentation, connection management features are not
163
+ immediately available when connected
164
+
165
+ You also can use method hooks for both regular timeline statuses and direct messages.
166
+
167
+ ```ruby
168
+ client = TweetStream::Client.new
169
+
170
+ client.on_direct_message do |direct_message|
171
+ puts "direct message"
172
+ puts direct_message.text
173
+ end
174
+
175
+ client.on_timeline_status do |status|
176
+ puts "timeline status"
177
+ puts status.text
178
+ end
179
+
180
+ client.userstream
181
+ ```
182
+
183
+ ## Authentication
184
+
185
+ TweetStream supports OAuth and Basic Auth. `TweetStream::Client` now accepts
186
+ a hash:
187
+
188
+ ```ruby
189
+ TweetStream::Client.new(:username => 'you', :password => 'pass')
190
+ ```
191
+
192
+ Alternatively, you can configure TweetStream via the configure method:
193
+
194
+ ```ruby
195
+ TweetStream.configure do |config|
196
+ config.consumer_key = 'cVcIw5zoLFE2a4BdDsmmA'
197
+ config.consumer_secret = 'yYgVgvTT9uCFAi2IuscbYTCqwJZ1sdQxzISvLhNWUA'
198
+ config.oauth_token = '4618-H3gU7mjDQ7MtFkAwHhCqD91Cp4RqDTp1AKwGzpHGL3I'
199
+ config.oauth_token_secret = 'xmc9kFgOXpMdQ590Tho2gV7fE71v5OmBrX8qPGh7Y'
200
+ config.auth_method = :oauth
201
+ end
202
+ ```
203
+
204
+ If you are using Basic Auth:
205
+
206
+ ```ruby
207
+ TweetStream.configure do |config|
208
+ config.username = 'username'
209
+ config.password = 'password'
210
+ config.auth_method = :basic
211
+ end
212
+ ```
213
+
214
+ TweetStream assumes OAuth by default. If you are using Basic Auth, it is recommended
215
+ that you update your code to use OAuth as Twitter is likely to phase out Basic Auth
216
+ support. Basic Auth is only available for public streams as User Stream and Site Stream
217
+ functionality [only support OAuth](https://dev.twitter.com/docs/streaming-apis/connecting#Authentication).
218
+
219
+ ## Parsing JSON
220
+
221
+ TweetStream supports swappable JSON backends via MultiJson. Simply require your preferred
222
+ JSON parser and it will be used to parse responses.
223
+
224
+ ## Handling Deletes and Rate Limitations
225
+
226
+ Sometimes the Streaming API will send messages other than statuses.
227
+ Specifically, it does so when a status is deleted or rate limitations
228
+ have caused some tweets not to appear in the stream. To handle these,
229
+ you can use the on_delete, on_limit and on_enhance_your_calm methods. Example:
230
+
231
+ ```ruby
232
+ @client = TweetStream::Client.new
233
+
234
+ @client.on_delete do |status_id, user_id|
235
+ Tweet.delete(status_id)
236
+ end
237
+
238
+ @client.on_limit do |skip_count|
239
+ # do something
240
+ end
241
+
242
+ @client.on_enhance_your_calm do
243
+ # do something
244
+ end
245
+
246
+ @client.track('intridea')
247
+ ```
248
+
249
+ The on_delete and on_limit methods can also be chained:
250
+
251
+ ```ruby
252
+ TweetStream::Client.new.on_delete{ |status_id, user_id|
253
+ Tweet.delete(status_id)
254
+ }.on_limit { |skip_count|
255
+ # do something
256
+ }.track('intridea') do |status|
257
+ # do something with the status like normal
258
+ end
259
+ ```
260
+
261
+ You can also provide `:delete` and/or `:limit`
262
+ options when you make your method call:
263
+
264
+ ```ruby
265
+ TweetStream::Client.new.track('intridea',
266
+ :delete => Proc.new{ |status_id, user_id| # do something },
267
+ :limit => Proc.new{ |skip_count| # do something }
268
+ ) do |status|
269
+ # do something with the status like normal
270
+ end
271
+ ```
272
+
273
+ Twitter recommends honoring deletions as quickly as possible, and
274
+ you would likely be wise to integrate this functionality into your
275
+ application.
276
+
277
+ ## Errors and Reconnecting
278
+
279
+ TweetStream uses EventMachine to connect to the Twitter Streaming
280
+ API, and attempts to honor Twitter's guidelines in terms of automatic
281
+ reconnection. When Twitter becomes unavailable, the block specified
282
+ by you in `on_error` will be called. Note that this does not
283
+ indicate something is actually wrong, just that Twitter is momentarily
284
+ down. It could be for routine maintenance, etc.
285
+
286
+ ```ruby
287
+ TweetStream::Client.new.on_error do |message|
288
+ # Log your error message somewhere
289
+ end.track('term') do |status|
290
+ # Do things when nothing's wrong
291
+ end
292
+ ```
293
+
294
+ However, if the maximum number of reconnect attempts has been reached,
295
+ TweetStream will raise a `TweetStream::ReconnectError` with
296
+ information about the timeout and number of retries attempted.
297
+
298
+ On reconnect, the block specified by you in `on_reconnect` will be called:
299
+
300
+ ```ruby
301
+ TweetStream::Client.new.on_reconnect do |timeout, retries|
302
+ # Do something with the reconnect
303
+ end.track('term') do |status|
304
+ # Do things when nothing's wrong
305
+ end
306
+ ```
307
+
308
+ ## Terminating a TweetStream
309
+
310
+ It is often the case that you will need to change the parameters of your
311
+ track or follow tweet streams. In the case that you need to terminate
312
+ a stream, you may add a second argument to your block that will yield
313
+ the client itself:
314
+
315
+ ```ruby
316
+ # Stop after collecting 10 statuses
317
+ @statuses = []
318
+ TweetStream::Client.new.sample do |status, client|
319
+ @statuses << status
320
+ client.stop if @statuses.size >= 10
321
+ end
322
+ ```
323
+
324
+ When `stop` is called, TweetStream will return from the block
325
+ the last successfully yielded status, allowing you to make note of
326
+ it in your application as necessary.
327
+
328
+ ## Daemonizing
329
+
330
+ It is also possible to create a daemonized script quite easily
331
+ using the TweetStream library:
332
+
333
+ ```ruby
334
+ # The third argument is an optional process name
335
+ TweetStream::Daemon.new('tracker').track('term1', 'term2') do |status|
336
+ # do something in the background
337
+ end
338
+ ```
339
+
340
+ If you put the above into a script and run the script with `ruby scriptname.rb`,
341
+ you will see a list of daemonization commands such as start, stop, and run.
342
+
343
+ A frequent use case is to use TweetStream along with ActiveRecord to insert new
344
+ statuses to a database. The library TweetStream uses the `daemons` gem for
345
+ daemonization which forks a new process when the daemon is created. After forking,
346
+ you'll need to reconnect to the database:
347
+
348
+ ```ruby
349
+ ENV["RAILS_ENV"] ||= "production"
350
+
351
+ root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
352
+ require File.join(root, "config", "environment")
353
+
354
+ daemon = TweetStream::Daemon.new('tracker', :log_output => true)
355
+ daemon.on_inited do
356
+ ActiveRecord::Base.connection.reconnect!
357
+ ActiveRecord::Base.logger = Logger.new(File.open('log/stream.log', 'w+'))
358
+ end
359
+ daemon.track('term1') do |tweet|
360
+ Status.create_from_tweet(tweet)
361
+ end
362
+ ```
363
+
364
+ ## Proxy Support
365
+
366
+ TweetStream supports a configurable proxy:
367
+
368
+ ```ruby
369
+ TweetStream.configure do |config|
370
+ config.proxy = { :uri => 'http://myproxy:8081' }
371
+ end
372
+ ```
373
+
374
+ Your proxy will now be used for all connections.
375
+
376
+ ## REST
377
+
378
+ To access the Twitter REST API, we recommend the [Twitter][] gem.
379
+
380
+ [twitter]: https://github.com/sferik/twitter
381
+
382
+ ## Contributors
383
+
384
+ * [Michael Bleigh](https://github.com/mbleigh) (initial gem)
385
+ * [Steve Agalloco](https://github.com/spagalloco) (current maintainer)
386
+ * [Erik Michaels-Ober](https://github.com/sferik) (current maintainer)
387
+ * [Countless others](https://github.com/intridea/tweetstream/graphs/contributors)
388
+
389
+ ## Copyright
390
+
391
+ Copyright (c) 2012-2013 Intridea, Inc. (http://www.intridea.com/). See
392
+ [LICENSE](LICENSE.md) for details.