tweetabout 0.0.5.0 → 0.0.6.0

data/README.md

@@ -1,12 +1,10 @@
-## Tweets About Gem
+## Tweet About Gem
 
-This gem takes a twitter username and returns a list of words.
+Takes a twitter username and returns a list of words.  Can also return a Hash with the count for each word.  Options to not include retweets, to set the number of tweets to be processed, and to return all words, without removing the useless ones.  See `Options Hash`.
 
-The words are are ordered list of the most frequently tweeted words based on the users last 1,000 tweets.
-* Retweets are included 
+The words are are ordered list of the most frequently tweeted words.
 * The casing of words doesn't matter (the = The = THE).  Output is downcased.
-* URLs are removed, because that makes sense.
-
+* URLs are removed.
 
 ## Installation
 
@@ -14,92 +12,61 @@ The words are are ordered list of the most frequently tweeted words based on the
 
 ### Gemfile
 
-    gem "tweetabout", "~> 0.0.4.1"
+    gem 'tweetabout'
 
 ### Dependencies
 
-`httparty` because httparty > NET::HTTP 
+`httparty`
 
 ## Useage
-
-example: Let's use `@jack`:
+example: For `@jack`:
 ```ruby
-#args: a single string
-
-#Invoke the TweetAbout Module:
-#call the tweetabout method
-
-@words = TweetAbout.tweetabout("jack")
-
-#returns an array of words sorted from most frequent to least
-#returns an empty array if the user doesn't have any tweets or:
-                        #   400 (most likely twitter api limit is exceeded) https://dev.twitter.com/docs/rate-limiting
-                        #   404 (most likely user doesn't exist, or twitter's api is down)
-
+#args: a single string, optional options hash
 ```
 
-## TODO
-
-* Test coverage.  Actually this a requirement before using this gem in production. And I know I am a bad Rubyist for not writing tests first but I promise to get better.
-* Make this Gem useful.  Right now it returns a list of all words, it might be interesting to strip out articles and pronouns from the list, that might actually give you an interesting insight into a person's interests. 
-* More accurate response messages.  Currently empty arrays get returned for three reasons:
-  1. Twitter api is down (404)
-  2. User does not exist (404)
-  3. Client has exceeded API limit (400).  This happens when > 150 requests are made within an hour.
-
-##Twitter API
-
-The Twitter API imposes a restriction on requests for users' timelines.  Each request can only receive a maximum of 200 tweets.  To get 1,000 tweets, that means we have to make 5 round trips to the api server.  Let's see how longs these requests take.  This is the measured response time of the GET request for 5 different twitter usernames:
-
-ex: GET `http://api.twitter.com/1/statuses/user_timeline.json?screen_name=#{user}&include_rts=true&count=200`
-(time in ms)
-                
-                trial1    trial2    trial3    trial4    trial5
-    request1    806.521   740.214   1363.33   490.090   331.253
-    request2    720.767   537.374   673.249   532.168   478.668
-    request3    492.608   733.955   560.918   547.887   380.81
-    request4    480.757   945.29    645.733   605.972   340.256
-    request5    575.621   469.731   707.737   826.423   169.244
-
-Based on this small test we can see that response times from the api vary from a few hundred milliseconds up to a full second.  Of course this is all influnced by time of day, network connection, and a variety of factors but it's good to know that if we have to make 5 trips in a row to the twitter api server we can't really count on it being very fast.  In fact, 5 requests could easily take 2.5 to 3 seconds to complete.  No doubt this is the slowest part of this gem.
-
-##Speed
-All these measurments are for processing the maximum of 1,000 tweets.  If the user has less than 1,000 tweets, obviously these processes will be faster.
-
-To see speeds yourself, checkout the speed branch and watch the server output.
-
-### get_tweets method
-This method does 5 GET requests to the Twitter api and stores them all in `@responses` These are the different time measurements:
-
-    4985.406 ms
-    3566.376 
-    4071.794 
-    7759.329 
-    3656.680
-    4510.602
+##Returns
+```ruby
+#default
+{:status => :ok, :words => ["fisbee", "golf",...]}
+```
+```ruby
+# {:with_count => true}
+{:status => :ok, :words => {"fisbee" => 5, 
+                            "golf"   => 3, ...} }`
+```
+```ruby
+#errors
+{:status => :api_limit_exceeded }
+{:status => :invalid_username }
+{:status => :no_tweets }
+```
+##Options Hash:
+```ruby
+#TweetAbout::TweetWords.new("@username").sort_words(:options => {})
+tweets:      n      (default 200)   #the number of tweets to process
+include_rts: t/f    (default true)  #include re-tweets by default
+with_count:  t/f    (default false) #if true, returns a Hash in the form of {word: n}, otherwise an Array of words is returned
+all_words:   t/f    (default false) #see `Junk Words`.  If set to true, no words are left out
+```
 
-### Processing responses (tweets)
-`@responses.each do |tweet|` block
-This method essentially takes the @responses variable, which is all the tweets, splits the words apart, removes punctuation and creates a hash of keys and values, keys are words, values are the number of times that word has shown up.  (the `bad_key` method below is part of this block.
+#Examples
+```ruby
+#returns an Array of words sorted by most frequent.
+TweetAbout::TweetWords.new("jack").sort
 
-    107.844 ms
-    117.764
-    78.989
-    87.12
-    137.528
-    134.256    
+#returns a Hash of words with each count sorted by most frequent.
+TweetAbout::TweetWords.new("jack").sort(with_count: true)
 
-#### bad_key method:
+#returns an Array of words sorted by most frequent for last 1,000 tweets
+TweetAbout::TweetWords.new("jack").sort(tweets: 1000)
+```
 
-    .006ms - .013ms each word
+##Junk Words
+These words are removed.  If you don't want them removed, pass the option `all_words => true`
 
-#### Sorting
-    
-    4.245 ms
-    4.016
-    4.098
-    5.509
-    3.994
+articles: "the", "a", "an"
+pronouns: "he", "him", "her", "she", "i", "you", "they", "them", "it"
+other_junk: "for", "from", "not", "but"
 
 
 

data/Rakefile

@@ -1,2 +1,10 @@
-#!/usr/bin/env rake
-require "bundler/gem_tasks"
+# !/usr/bin/env rake
+# require "bundler/gem_tasks"
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << 'test'
+end
+
+desc "Run tests"
+task :default => :test

data/lib/tweetabout.rb

@@ -2,6 +2,7 @@ require "tweetabout/version"
 require "httparty"
 
 module TweetAbout
-  require 'tweetabout/sorted_tweets'
-
-end
+  require_relative 'tweetabout/tweet_words'
+  require_relative 'tweetabout/twit_api'
+  require_relative 'tweetabout/words_hash'
+end

data/lib/tweetabout/tweet_words.rb

@@ -0,0 +1,47 @@
+module TweetAbout  
+  
+  class TweetWords
+    attr_accessor :status
+    attr_accessor :words
+
+    def initialize(username)
+      @username = username
+    end
+
+    def sort_words(options={})
+      to_i = options[:tweets].to_i
+      tweets_needed = to_i > 0? to_i : 200
+      options[:tweets] = tweets_needed
+      get_words
+      if self.status == :ok
+        words = self.words
+        sorted_words = words.sort
+        sorted_words = options[:with_count]? sorted_words : sorted_words.keys
+        self.words = sorted_words
+      end
+      self
+    end
+
+    private
+
+    def get_words(options={})
+      api_options = { :query => {:screen_name =>  @username, :include_rts => options[:include_rts]} }
+
+      twit_api = TwitApi.new(options[:tweets], api_options)
+
+      if tweets = twit_api.tweets
+        words = WordsHash.new(0)
+
+        tweets.each do |tweet|
+          tweet.split.each do |key|
+            words[key.gsub(/\W/, "").strip.downcase] += 1 unless key.start_with?('http') || key.empty?
+          end
+        end
+
+        words = words.sanitize unless options[:all_words]
+        self.words = words
+      end
+      self.status = twit_api.status
+    end
+  end
+end

data/lib/tweetabout/twit_api.rb

@@ -0,0 +1,41 @@
+module TweetAbout
+  class TwitApi
+    include HTTParty
+
+    attr_accessor :status
+    attr_accessor :tweets
+
+    BASE_URL = "http://api.twitter.com/1/statuses/user_timeline.json?"
+    TWEETS_NEEDED = 200
+    
+    def initialize(tweets_needed, api_options)
+      self.tweets = []
+      tweets_needed ||= TWEETS_NEEDED
+      begin
+        query_amt = tweets_needed <= 200? tweets_needed : 200
+
+        api_options[:query].merge!({:count => query_amt})
+        response = HTTParty.get(BASE_URL, api_options)
+
+      case response.code
+        when 200
+          self.status = :ok
+        when 400
+          self.status = :no_tweets
+        when 404
+          self.status = :invalid_username
+        else
+          self.status = :error
+      end
+
+      tweets_needed -= response.count
+      api_options[:query].merge!({:max_id => response.last["id"] + 1 })
+      response.each do |response|
+        self.tweets << response["text"]
+      end
+
+      end while tweets_needed > 0
+    end
+  end
+
+end

data/lib/tweetabout/version.rb

@@ -1,3 +1,3 @@
 module Tweetabout
-  VERSION = "0.0.5.0"
+  VERSION = "0.0.6.0"
 end

data/lib/tweetabout/words_hash.rb

@@ -0,0 +1,27 @@
+module TweetAbout
+  class WordsHash < Hash 
+    def sanitize
+      junk_words
+    end
+
+    def sort
+      WordsHash[self.sort_by { |key, frequency| frequency }.reverse]
+    end
+
+    private
+
+    def junk_words
+      articles = ["the", "a", "an"]
+      pronouns = ["we", "he", "him", "his", "her", "she", "i", "you", "they", "them", "it", "your",
+                  "our", "us", "my", "this", "their"]
+      other_junk = ["for", "from", "not", "but", "is", "in", "and", "so",
+                    "of", "if", "at", "rt", "", "all", "to", "that",
+                    "are", "can", "by", "on", "as", "or", "as"]
+
+      all_junk = articles << pronouns << other_junk
+      all_junk.flatten!
+
+      self.delete_if { |key, value| all_junk.include?(key) }
+    end
+  end
+end

data/test/test_tweetabout.rb

@@ -0,0 +1,60 @@
+require 'test/unit'
+require 'minitest/spec'
+require 'mocha/setup'
+require 'tweetabout'
+
+tw = TweetAbout::TweetWords.new("jack")
+
+describe "the defaults" do
+  response = tw.sort_words(:tweets => 5)
+
+  it "should be an instance of TweetWords" do
+    assert_equal TweetAbout::TweetWords, tw.class
+  end
+
+  it "should be successful and should be an Array" do
+    assert_equal :ok, response.status
+    assert_equal Array, response.words.class
+  end
+end
+
+describe "the {:with_count => true} option" do
+  response = tw.sort_words(:with_count => true, :tweets => 5)
+  words = response.words
+
+  it "should be successful and should be a Hash" do
+    assert_equal :ok, response.status
+    assert_equal TweetAbout::WordsHash, words.class
+  end
+end
+
+describe ":with_count option" do
+  it "should handle with count true" do
+    response = tw.sort_words(:with_count => true, :tweets => 5)
+    assert_equal :ok, response.status
+  end
+
+  it "should handle with count false" do
+    response = tw.sort_words(:with_count => false, :tweets => 5)
+    assert_equal :ok, response.status
+  end
+end
+
+describe "a number string to tweets should work" do
+  it "should handle a number string for tweets" do
+    response = tw.sort_words(:tweets => "5")
+    assert_equal :ok, response.status
+  end
+end
+
+describe "include_rts options should work" do
+  it "should handle include rts true" do
+    response = tw.sort_words(:include_rts => true)
+    assert_equal :ok, response.status
+  end
+
+  it "should handle include rts true" do
+    response = tw.sort_words(:include_rts => false)
+    assert_equal :ok, response.status
+  end
+end

data/test/test_wordhash.rb

@@ -0,0 +1,27 @@
+require 'test/unit'
+require 'minitest/spec'
+require 'mocha/setup'
+require 'tweetabout'
+
+words_hash = TweetAbout::WordsHash[{"of" => 1, "compassion" => 2, "love" => 4, "humanity" => 5,
+                                    "together" => 4, "free" => 4}]
+
+describe "#sanitize" do
+  it "should remove a useless word" do
+    words_hash = words_hash.sanitize
+    assert_equal true, !words_hash.has_key?("of")
+  end
+end
+
+describe "#sort" do
+  it "should sort the words" do
+    words_hash = words_hash.sort
+    values = words_hash.values
+    trues = []
+    values.each_index do |i|
+      val = values[i] >= values[i.next] if values[i.next]
+      trues << val if val
+    end
+    assert_equal false, trues.include?(false)
+  end
+end

data/tweetabout.gemspec

@@ -5,10 +5,11 @@ Gem::Specification.new do |gem|
   gem.authors       = ["Dylan Jhaveri"]
   gem.email         = ["dylanjhaveri@gmail.com"]
   gem.description   = %q{Returns a list of frequently tweeted words}
-  gem.summary       = %q{Takes a twitter username and outputs the most frequently tweeted words in their last 1,000 tweets.  Includes Re-tweets and excludes links.}
+  gem.summary       = %q{Takes a twitter username and returns a list of words.  Can also return a Hash with the count for each word.  Has options to not include retweets and set the number of tweets to process}
   gem.homepage      = ""
 
   gem.add_dependency "httparty"
+  gem.add_development_dependency('mocha', "~> 0.9.9")
 
   gem.files         = `git ls-files`.split($\)
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tweetabout
 version: !ruby/object:Gem::Version
-  version: 0.0.5.0
+  version: 0.0.6.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-08-22 00:00:00.000000000 Z
+date: 2013-01-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: httparty
@@ -27,6 +27,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.9
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.9
 description: Returns a list of frequently tweeted words
 email:
 - dylanjhaveri@gmail.com
@@ -40,8 +56,12 @@ files:
 - README.md
 - Rakefile
 - lib/tweetabout.rb
-- lib/tweetabout/sorted_tweets.rb
+- lib/tweetabout/tweet_words.rb
+- lib/tweetabout/twit_api.rb
 - lib/tweetabout/version.rb
+- lib/tweetabout/words_hash.rb
+- test/test_tweetabout.rb
+- test/test_wordhash.rb
 - tweetabout.gemspec
 homepage: ''
 licenses: []
@@ -66,6 +86,9 @@ rubyforge_project:
 rubygems_version: 1.8.24
 signing_key: 
 specification_version: 3
-summary: Takes a twitter username and outputs the most frequently tweeted words in
-  their last 1,000 tweets.  Includes Re-tweets and excludes links.
-test_files: []
+summary: Takes a twitter username and returns a list of words.  Can also return a
+  Hash with the count for each word.  Has options to not include retweets and set
+  the number of tweets to process
+test_files:
+- test/test_tweetabout.rb
+- test/test_wordhash.rb

data/lib/tweetabout/sorted_tweets.rb

@@ -1,84 +0,0 @@
-module TweetAbout  
-  
-  class SortedTweets
-    include HTTParty
-    
-    def initialize(username)
-      @username = username
-    end
-
-    def sort_tweets
-      get_tweets
-      hash = {}
-
-      @responses.each do |tweet|
-        tweet.split(" ").each do |key|
-          key = key.gsub(/\W/, "").downcase
-          if hash.has_key?(key)
-            hash["#{key}"] += 1
-          else
-            hash.merge!({"#{key}" => 1}) unless bad_key(key)
-          end
-        end
-      end
-      sorted_array = hash.sort_by { |keyword, frequency| frequency }.reverse
-      sorted_array.map { |keyword| keyword[0] }
-    end
-
-    def get_tweets
-      options = { :query => {:screen_name => @username, :include_rts => true, :count => 200} }
-
-      base_url = "http://api.twitter.com/1/statuses/user_timeline.json?"
-
-      @responses = []
-      response1 = HTTParty.get("#{base_url}", options)
-      return if response1.code == 404             #invalid username/bad url
-      return if response1.count == 0              #no tweets!
-      return if response1.code == 400             #api limt exceeded
-      start_at_1 = response1.last["id"]
-      response1.each do |object|
-        @responses << object["text"]
-      end
-
-      response2 = HTTParty.get("#{base_url}&max_id=#{start_at_1-1}", options)
-      return if response2.count == 0
-      return if response2.code == 400
-      start_at_2 = response2.last["id"]
-      response2.each do |object|
-        @responses << object["text"]
-      end
-      
-      response3 = HTTParty.get("#{base_url}&max_id=#{start_at_2-1}", options)
-      return if response3.count == 0
-      return if response3.code == 400
-      start_at_3 = response3.last["id"]
-      response3.each do |object|
-        @responses << object["text"]
-      end
-      
-      response4 = HTTParty.get("#{base_url}&max_id=#{start_at_3-1}", options)
-      return if response4.count == 0
-      return if response4.code == 400
-      start_at_4 = response4.last["id"]
-      response4.each do |object|
-        @responses << object["text"]
-      end
-
-      response5 = HTTParty.get("#{base_url}&max_id=#{start_at_4-1}", options)
-      return if response5.count == 0
-      return if response5.code == 400
-      start_at_5 = response4.last["id"]
-      response5.each do |object|
-        @responses << object["text"]
-      end
-    end
-
-    def bad_key(key)
-      return true if key.empty?
-      return true if key.start_with?('http')
-    end
-
-    def to_a
-    end
-  end
-end