PageRankr 1.7.1 → 2.0.0

data/CHANGELOG.md

@@ -1,4 +1,13 @@
 # Change Log
+
+## Version 2.0.0
+* URL validation
+* Parallel requests = way faster!
+* Not tracked returns nil
+* Alexa US and Global are treated as separate trackers and returned results are a single level hash.
+* Removed Altavista and AllTheWeb because they now direct to yahoo.
+* Changed some classes to modules so that it wasn't necessary to specify them when opening the class.
+
 ## Version 1.7.1
 * Catches exception thrown when doing compete rank lookup with url not in the form "google.com".
 

data/Gemfile.lock

@@ -1,9 +1,11 @@
 PATH
   remote: .
   specs:
-    PageRankr (1.7.0)
+    PageRankr (1.7.1)
       json (>= 1.4.6)
       nokogiri (>= 1.4.1)
+      public_suffix_service (~> 0.8.1)
+      typhoeus (~> 0.2.1)
 
 GEM
   remote: http://rubygems.org/
@@ -14,7 +16,9 @@ GEM
       rspec-instafail (~> 0.1.4)
       ruby-progressbar (~> 0.0.9)
     json (1.5.1)
+    mime-types (1.16)
     nokogiri (1.4.4)
+    public_suffix_service (0.8.1)
     rspec (2.1.0)
       rspec-core (~> 2.1.0)
       rspec-expectations (~> 2.1.0)
@@ -25,6 +29,8 @@ GEM
     rspec-instafail (0.1.5)
     rspec-mocks (2.1.0)
     ruby-progressbar (0.0.9)
+    typhoeus (0.2.1)
+      mime-types
 
 PLATFORMS
   ruby

data/PageRankr.gemspec

@@ -15,8 +15,10 @@ Gem::Specification.new do |s|
   s.add_development_dependency "bundler", ">= 1.0.0"
   s.add_development_dependency "fuubar",  ">= 0.0.1"
 
-  s.add_runtime_dependency "nokogiri", ">= 1.4.1"
-  s.add_runtime_dependency "json",     ">= 1.4.6"
+  s.add_runtime_dependency "nokogiri",              ">= 1.4.1"
+  s.add_runtime_dependency "json",                  ">= 1.4.6"
+  s.add_runtime_dependency "public_suffix_service", "~> 0.8.1"
+  s.add_runtime_dependency "typhoeus",              "~> 0.2.1"
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")

data/README.md

@@ -17,7 +17,7 @@ Check out a little [web app][1] I wrote up that uses it or look at the [source][
 
 ### Backlinks
 
-Backlinks are the result of doing a search with a query like "link:www.google.com". The number of returned results indicates how many sites point to that url.
+Backlinks are the result of doing a search with a query like "link:www.google.com". The number of returned results indicates how many sites point to that url. If a site is not tracked then `nil` is returned.
 
     PageRankr.backlinks('www.google.com', :google, :bing) #=> {:google=>161000, :bing=>208000000}
     PageRankr.backlinks('www.google.com', :yahoo)         #=> {:yahoo=>256300062}
@@ -26,20 +26,21 @@ If you don't specify a search engine, then all of them are used.
 
     # this
     PageRankr.backlinks('www.google.com')
-        #=> {:google=>23000, :bing=>215000000, :yahoo=>250522337, :altavista=>137000000, :alltheweb=>74500000, :alexa=>727036}
+        #=> {:google=>23000, :bing=>215000000, :yahoo=>250522337, :alexa=>727036}
 
     # is equivalent to
-    PageRankr.backlinks('www.google.com', :google, :bing, :yahoo, :altavista, :alltheweb, :alexa)
-        #=> {:google=>23000, :bing=>215000000, :yahoo=>250522337, :altavista=>137000000, :alltheweb=>74500000, :alexa=>727036}
+    PageRankr.backlinks('www.google.com', :google, :bing, :yahoo, :alexa)
+        #=> {:google=>23000, :bing=>215000000, :yahoo=>250522337, :alexa=>727036}
 
 You can also use the alias `backlink` instead of `backlinks`.
-Valid search engines are: `:google, :bing, :yahoo, :altavista, :alltheweb, :alexa`. To get this list you can do:
 
-    PageRankr.backlink_trackers #=> [:alexa, :alltheweb, :altavista, :bing, :google, :yahoo]
+Valid search engines are: `:google, :bing, :yahoo, :alexa` (altavista and alltheweb now redirect to yahoo). To get this list you can do:
+
+    PageRankr.backlink_trackers #=> [:alexa, :bing, :google, :yahoo]
 
 ### Indexes
 
-Indexes are the result of doing a search with a query like "site:www.google.com". The number of returned results indicates how many pages of a domain are indexed by a particular search engine.
+Indexes are the result of doing a search with a query like "site:www.google.com". The number of returned results indicates how many pages of a domain are indexed by a particular search engine. If the site is not indexed `nil` is returned.
 
     PageRankr.indexes('www.google.com', :google)       #=> {:google=>4860000}
     PageRankr.indexes('www.google.com', :bing)         #=> {:bing=>2120000}
@@ -55,36 +56,43 @@ If you don't specify a search engine, then all of them are used.
         #=> {:bing=>2120000, :google=>4860000}
 
 You can also use the alias `index` instead of `indexes`.
+
 Valid search engines are: `:google, :bing`. To get this list you can do:
 
-    PageRankr.index_trackers #=> [:alexa, :alltheweb, :altavista, :bing, :google, :yahoo]
+    PageRankr.index_trackers #=> [:bing, :google]
 
 ### Ranks
 
-__Note: Compete only accepts urls in the form of "google.com" and not "www.google.com". If the latter is given, a value of 0 will be returned. This will be fixed in version 2 of PageRankr.__
+Ranks are ratings assigned to specify how popular a site is. The most famous example of this is the google page rank.
+
+    PageRankr.ranks('www.google.com', :google)        #=> {:google=>10}
 
-    PageRankr.ranks('google.com', :alexa, :google, :compete) #=> {:alexa=>{:us=>1, :global=>1}, :google=>10, :compete=>1}
+If you don't specify a rank provider, then all of them are used.
+
+    PageRankr.ranks('www.google.com', :alexa_us, :alexa_global, :compete, :google)
+        #=> {:alexa_us=>1, :alexa_global=>1, :google=>10, :compete=>1}
 
     # this also gives the same result
-    PageRankr.ranks('google.com')                            #=> {:alexa=>{:us=>1, :global=>1}, :google=>10, :compete=>1}
+    PageRankr.ranks('www.google.com')
+        #=> {:alexa_us=>1, :alexa_global=>1, :google=>10, :compete=>1}
 
 You can also use the alias `rank` instead of `ranks`.
-There are three valid rank trackers supported: `:alexa, :google, :compete`. To get this you can do:
 
-    PageRankr.rank_trackers #=> [:alexa, :google, :compete]
+Valid rank trackers are: `:alexa_us, :alexa_global, :compete, :google`. To get this you can do:
+
+    PageRankr.rank_trackers #=> [:alexa_global, :alexa_us, :compete, :google]
 
-Alexa ranks are descending where 1 is the most popular. If a site has an alexa rank of 0 then the site is unranked.
-Google page ranks are in the range 0-10 where 10 is the most popular. If a site is unindexed then the rank will be -1.
+Alexa and Compete ranks are descending where 1 is the most popular. Google page ranks are in the range 0-10 where 10 is the most popular. If a site is unindexed then the rank will be nil.
 
 ## Fix it!
 
 If you ever find something is broken it should now be much easier to fix it with version >= 1.3.0. For example, if the xpath used to lookup a backlink is broken, just override the method for that class to provide the correct xpath.
 
     module PageRankr
-      class Backlinks < Tracker
-        class Google < Backlink
+      class Backlinks
+        class Bing
           def xpath
-            "my new awesome xpath"
+            "//my/new/awesome/@xpath"
           end
         end
       end
@@ -92,13 +100,16 @@ If you ever find something is broken it should now be much easier to fix it with
 
 ## Extend it!
 
-If you ever come across a site that provides a rank or backlinks you can hook that class up to automatically be use with PageRankr.
+If you ever come across a site that provides a rank or backlinks you can hook that class up to automatically be use with PageRankr. PageRankr does this by looking up all the classes namespaced under Backlinks, Indexes, and Ranks.
 
     module PageRankr
-      class Backlinks < Tracker
-        class Foo < Backlink
-          def url(site)
-            "http://example.com/?q=#{site}"
+      class Backlinks
+        class Foo
+          include Backlink
+
+          def request
+            @request ||= Typhoeus::Request.new("http://example.com/",
+                :params => {:q => @site.to_s})
           end
 
           def xpath
@@ -125,15 +136,16 @@ Then, just make sure you require the class and PageRankr and whenever you call P
   (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
-## TODO Version 2
+## TODO Version 3
 * Use API's where possible
-* Use [Typhoeus](https://github.com/pauldix/typhoeus) to improve speed when requesting multiple ranks and/or backlinks
 * Configuration
     * Optionally use API keys
+    * Maybe allow API key cycling to get around query limits
+* Google search API is deprecated
 
 ## Contributors
 * [Druwerd](https://github.com/Druwerd) - Use Google Search API instead of scraping.
-* [Iteration Labs](https://github.com/iterationlabs) - Compete rank tracker and domain indexes.
+* [Iteration Labs, LLC](https://github.com/iterationlabs) - Compete rank tracker and domain indexes.
 
 ## Shout Out
 Gotta give credit where credits due!

data/lib/page_rankr.rb

@@ -2,31 +2,32 @@ require File.join(File.dirname(__FILE__), "page_rankr", "tracker")
 require File.join(File.dirname(__FILE__), "page_rankr", "backlinks")
 require File.join(File.dirname(__FILE__), "page_rankr", "ranks")
 require File.join(File.dirname(__FILE__), "page_rankr", "indexes")
+require File.join(File.dirname(__FILE__), "page_rankr", "site")
 
 module PageRankr
   class << self
     def backlinks(site, *search_engines)
-      Backlinks.new.lookup site, *search_engines
+      Backlinks.new.lookup(Site.new(site), *search_engines)
     end
     alias_method :backlink, :backlinks
+    
+    def backlink_trackers
+      Backlinks.new.backlink_trackers
+    end
 
     def ranks(site, *rank_trackers)
-      Ranks.new.lookup site, *rank_trackers
+      Ranks.new.lookup(Site.new(site), *rank_trackers)
     end
     alias_method :rank, :ranks
     
-    def indexes(site, *index_trackers)
-      Indexes.new.lookup site, *index_trackers
-    end
-    alias_method :index, :indexes
-    
     def rank_trackers
       Ranks.new.rank_trackers
     end
     
-    def backlink_trackers
-      Backlinks.new.backlink_trackers
+    def indexes(site, *index_trackers)
+      Indexes.new.lookup(Site.new(site), *index_trackers)
     end
+    alias_method :index, :indexes
     
     def index_trackers
       Indexes.new.index_trackers

data/lib/page_rankr/backlink.rb

@@ -3,12 +3,18 @@ require 'nokogiri'
 require 'open-uri'
 
 module PageRankr
-  class Backlink
+  module Backlink
     attr_reader :backlinks
     alias_method :tracked, :backlinks
     
     def initialize(site)
-      @backlinks = clean Nokogiri::HTML(open url(site)).at(xpath).to_s
+      @site = site
+      
+      request.on_complete do |response|
+        html = Nokogiri::HTML(response.body)
+        @backlinks = clean(html.at(xpath).to_s)
+        @backlinks = nil if @backlinks.zero?
+      end
     end
     
     def clean(backlink_count)

data/lib/page_rankr/backlinks.rb

@@ -1,13 +1,13 @@
 require File.join(File.dirname(__FILE__), "backlink")
 require File.join(File.dirname(__FILE__), "backlinks", "alexa")
-require File.join(File.dirname(__FILE__), "backlinks", "alltheweb")
-require File.join(File.dirname(__FILE__), "backlinks", "altavista")
 require File.join(File.dirname(__FILE__), "backlinks", "bing")
 require File.join(File.dirname(__FILE__), "backlinks", "google")
 require File.join(File.dirname(__FILE__), "backlinks", "yahoo")
 
 module PageRankr
-  class Backlinks < Tracker
+  class Backlinks
+    include Tracker
+    
     alias_method :backlink_trackers, :site_trackers
   end
 end

data/lib/page_rankr/backlinks/alexa.rb

@@ -1,10 +1,13 @@
-require 'cgi'
+require 'typhoeus'
 
 module PageRankr
-  class Backlinks < Tracker
-    class Alexa < Backlink
-      def url(site)
-        "http://data.alexa.com/data?cli=10&dat=snbamz&url=#{CGI.escape(site)}"
+  class Backlinks
+    class Alexa
+      include Backlink
+      
+      def request
+        @request ||= Typhoeus::Request.new("http://data.alexa.com/data",
+            :params => {:cli => 10, :dat => "snbamz", :url => @site.to_s}, :method => :get)
       end
       
       def xpath

data/lib/page_rankr/backlinks/bing.rb

@@ -1,10 +1,13 @@
-require 'cgi'
+require 'typhoeus'
 
 module PageRankr
-  class Backlinks < Tracker
-    class Bing < Backlink
-      def url(site)
-        "http://www.bing.com/search?q=link%3A#{CGI.escape(site)}"
+  class Backlinks
+    class Bing
+      include Backlink
+      
+      def request
+        @request ||= Typhoeus::Request.new("http://www.bing.com/search", 
+            :params => {:q => "link:#{@site.to_s}"}, :method => :get)
       end
       
       def xpath

data/lib/page_rankr/backlinks/google.rb

@@ -1,18 +1,24 @@
-require 'cgi'
 require 'json'
 
 module PageRankr
-  class Backlinks < Tracker
-    class Google < Backlink
+  class Backlinks
+    class Google
+      include Backlink
       
       # overloaded to use Google's AJAX search API
       # http://code.google.com/apis/ajaxsearch/documentation/
       def initialize(site)
-        @backlinks = clean JSON.parse( open( url(site)).read )["responseData"]["cursor"]["estimatedResultCount"].to_s
+        @site = site
+        request.on_complete do |response|
+          json = JSON.parse(response.body)
+          @backlinks = clean(json["responseData"]["cursor"]["estimatedResultCount"].to_s)
+          @backlinks = nil if @backlinks.zero?
+        end
       end
       
-      def url(site)
-        "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=1&q=link%3A#{CGI.escape(site)}"
+      def request
+        @request ||= Typhoeus::Request.new("http://ajax.googleapis.com/ajax/services/search/web", 
+            :params => {:v => "1.0", :rsz => 1, :q => "link:#{@site.to_s}"}, :method => :get)
       end
     end
   end

data/lib/page_rankr/backlinks/yahoo.rb

@@ -1,10 +1,13 @@
-require 'cgi'
+require 'typhoeus'
 
 module PageRankr
-  class Backlinks < Tracker
-    class Yahoo < Backlink
-      def url(site)
-        "http://siteexplorer.search.yahoo.com/search?p=#{CGI.escape(site)}"
+  class Backlinks
+    class Yahoo
+      include Backlink
+      
+      def request
+        @request ||= Typhoeus::Request.new("http://siteexplorer.search.yahoo.com/search",
+            :params => {:p => "#{@site.to_s}"}, :method => :get)
       end
       
       def xpath

data/lib/page_rankr/index.rb

@@ -2,12 +2,17 @@ require 'nokogiri'
 require 'open-uri'
 
 module PageRankr
-  class Index
+  module Index
     attr_reader :indexes
     alias_method :tracked, :indexes
     
     def initialize(site)
-      @indexes = clean Nokogiri::HTML(open url(site)).at(xpath).to_s
+      @site = site
+      request.on_complete do |response|
+        html = Nokogiri::HTML(response.body)
+        @indexes = clean(html.at(xpath).to_s)
+        @indexes = nil if @indexes.zero?
+      end
     end
     
     def clean(backlink_count)

data/lib/page_rankr/indexes.rb

@@ -3,7 +3,9 @@ require File.join(File.dirname(__FILE__), "indexes", "bing")
 require File.join(File.dirname(__FILE__), "indexes", "google")
 
 module PageRankr
-  class Indexes < Tracker
+  class Indexes
+    include Tracker
+    
     alias_method :index_trackers, :site_trackers
   end
 end

data/lib/page_rankr/indexes/bing.rb

@@ -1,10 +1,13 @@
-require 'cgi'
+require 'typhoeus'
 
 module PageRankr
-  class Indexes < Tracker
-    class Bing < Index
-      def url(site)
-        "http://www.bing.com/search?q=site%3A#{CGI.escape(site)}"
+  class Indexes
+    class Bing
+      include Index
+      
+      def request
+        @request ||= Typhoeus::Request.new("http://www.bing.com/search",
+            :params => {:q => "site:#{@site.to_s}"})
       end
       
       def xpath

data/lib/page_rankr/indexes/google.rb

@@ -1,18 +1,26 @@
-require 'cgi'
+require 'typhoeus'
 require 'json'
 
 module PageRankr
-  class Indexes < Tracker
-    class Google < Index
+  class Indexes
+    class Google
+      include Index
       
       # overloaded to use Google's AJAX search API
       # http://code.google.com/apis/ajaxsearch/documentation/
       def initialize(site)
-        @indexes = clean JSON.parse( open( url(site)).read )["responseData"]["cursor"]["estimatedResultCount"].to_s
+        @site = site
+        
+        request.on_complete do |response|
+          json = JSON.parse(response.body)
+          @indexes = clean(json["responseData"]["cursor"]["estimatedResultCount"].to_s)
+          @indexes = nil if @indexes.zero?
+        end
       end
       
-      def url(site)
-        "http://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=1&q=site%3A#{CGI.escape(site)}"
+      def request
+        @request ||= Typhoeus::Request.new("http://ajax.googleapis.com/ajax/services/search/web",
+            :params => {:v => "1.0", :rsz => 1, :q => "site:#{@site.to_s}"}, :method => :get)
       end
     end
   end

data/lib/page_rankr/rank.rb

@@ -1,6 +1,20 @@
 module PageRankr
-  class Rank
+  module Rank
     attr_reader :rank
     alias_method :tracked, :rank
+    
+    def initialize(site)
+      @site = site
+      
+      request.on_complete do |response|
+        html = Nokogiri::HTML(response.body)
+        @rank = clean(html.search(xpath))
+        @rank = nil if @rank.zero?
+      end
+    end
+    
+    def clean(rank)
+      rank.to_s.to_i
+    end
   end
 end

data/lib/page_rankr/ranks.rb

@@ -1,10 +1,13 @@
 require File.join(File.dirname(__FILE__), "rank")
-require File.join(File.dirname(__FILE__), "ranks", "alexa")
+require File.join(File.dirname(__FILE__), "ranks", "alexa_us")
+require File.join(File.dirname(__FILE__), "ranks", "alexa_global")
 require File.join(File.dirname(__FILE__), "ranks", "google")
 require File.join(File.dirname(__FILE__), "ranks", "compete")
 
 module PageRankr
-  class Ranks < Tracker
+  class Ranks
+    include Tracker
+    
     alias_method :rank_trackers, :site_trackers
   end
 end

data/lib/page_rankr/ranks/alexa_global.rb

@@ -0,0 +1,19 @@
+require 'typhoeus'
+require 'nokogiri'
+
+module PageRankr
+  class Ranks
+    class AlexaGlobal
+      include Rank
+      
+      def xpath
+        "//popularity/@text"
+      end
+      
+      def request
+        @request ||= Typhoeus::Request.new("http://data.alexa.com/data",
+            :params => {:cli => 10, :dat => "snbamz", :url => @site.to_s})
+      end
+    end
+  end
+end

data/lib/page_rankr/ranks/alexa_us.rb

@@ -0,0 +1,19 @@
+require 'typhoeus'
+require 'nokogiri'
+
+module PageRankr
+  class Ranks
+    class AlexaUs
+      include Rank
+      
+      def xpath
+        "//reach/@rank"
+      end
+      
+      def request
+         @request ||= Typhoeus::Request.new("http://data.alexa.com/data",
+              :params => {:cli => 10, :dat => "snbamz", :url => @site.to_s})
+      end
+    end
+  end
+end

data/lib/page_rankr/ranks/compete.rb

@@ -1,22 +1,22 @@
-require 'open-uri'
+require 'typhoeus'
 require 'cgi'
 require 'nokogiri'
 
 module PageRankr
-  class Ranks < Tracker
-    class Compete < Rank
-      def initialize(site)
-        @rank = Nokogiri::HTML(open(url(site))).search(xpath).to_s.gsub(',', '').to_i
-      rescue
-        @rank = 0 #compete only accepts urls without http:// and www, will be fixed in 2.0
-      end
+  class Ranks
+    class Compete
+      include Rank
       
       def xpath
         "//div[@id='rank']/div[@class='number value']/text()"
       end
       
-      def url(site)
-        "http://siteanalytics.compete.com/#{CGI.escape(site)}/"
+      def request
+        @request ||= Typhoeus::Request.new("http://siteanalytics.compete.com/#{CGI.escape(@site.domain)}/")
+      end
+      
+      def clean(rank)
+        rank.to_s.gsub(',', '').to_i
       end
     end
   end

data/lib/page_rankr/ranks/google.rb

@@ -1,15 +1,21 @@
-require "open-uri"
+require 'typhoeus'
 require File.join(File.dirname(__FILE__), "google", "checksum")
 
 module PageRankr
-  class Ranks < Tracker
-    class Google < Rank
+  class Ranks
+    class Google
+      include Rank
+      
       def initialize(site)
-        checksum = Checksum.generate(site)
-        @rank = begin
-          open(url(site, checksum)) {|io| io.read.scan(regex)[0][0].to_i}
-        rescue
-          -1
+        @site = site
+        @checksum = Checksum.generate(@site.to_s)
+        
+        request.on_complete do |response|
+          @rank = if response.body =~ regex
+            clean($1)
+          else
+            nil
+          end
         end
       end
       
@@ -17,8 +23,9 @@ module PageRankr
         /Rank_\d+:\d+:(\d+)/
       end
 
-      def url(site, checksum)
-        "http://toolbarqueries.google.com/search?client=navclient-auto&ch=#{checksum}&features=Rank&q=info:#{site}"
+      def request
+        @request ||= Typhoeus::Request.new("http://toolbarqueries.google.com/search",
+              :params => {:client => "navclient-auto", :ch => @checksum, :features => "Rank", :q => "info:#{@site.to_s}"})
       end
     end
   end

data/lib/page_rankr/ranks/google/checksum.rb

@@ -1,6 +1,6 @@
 module PageRankr
-  class Ranks < Tracker
-    class Google < Rank
+  class Ranks
+    class Google
       class Checksum
         class << self
           def generate(site)

data/lib/page_rankr/site.rb

@@ -0,0 +1,21 @@
+require 'public_suffix_service'
+require 'delegate'
+
+module PageRankr
+  class DomainInvalid < StandardError; end
+  
+  class Site < DelegateClass(PublicSuffixService::Domain)
+    def initialize(site)
+      super(PublicSuffixService.parse(clean(site)))
+      valid? or raise DomainInvalid, "The domain provided is invalid."
+    end
+    
+    private
+    
+    def clean(site)
+      site = site.split("://").last # remove protocol
+      site = site.split("/").first  # remove path
+      site.split("?").first         # remove params
+    end
+  end
+end

data/lib/page_rankr/tracker.rb

@@ -1,23 +1,48 @@
+require 'typhoeus'
+
 module PageRankr
-  class Tracker
+  module Tracker
     attr_accessor :site_trackers
     
     def initialize
-      @site_trackers = self.class.constants.collect{|tracker| tracker.to_s.downcase.to_sym}
+      @site_trackers = self.class.constants.collect{|tracker| symbol_for(tracker)}
     end
 
     def lookup(site, *trackers)
       trackers = site_trackers if trackers.empty?
       
       tracked = {}
+      hydra = Typhoeus::Hydra.new
       trackers.each do |tracker|
-        name, klass = tracker.to_s.capitalize, self.class
+        name, klass = constant_name(tracker), self.class
         
         next unless klass.const_defined? name
 
-        tracked[tracker] = klass.const_get(name).new(site).tracked
+        tracked[tracker] = klass.const_get(name).new(site)
+        hydra.queue tracked[tracker].request
+      end
+      hydra.run
+      
+      tracked.keys.each do |tracker|
+        tracked[tracker] = tracked[tracker].tracked
       end
+      
       tracked
     end
+    
+    private
+    
+    def symbol_for(klass)
+      word = klass.to_s.dup
+      word.gsub!(/([A-Z]+)([A-Z][a-z])/){|match| "#{$1}_#{$2}" }
+      word.gsub!(/([a-z\d])([A-Z])/){|match| "#{$1}_#{$2}" }
+      word.tr!("-", "_")
+      word.downcase!
+      word.to_sym
+    end
+    
+    def constant_name(sym)
+      sym.to_s.split('_').collect{|str| str.capitalize}.join
+    end
   end
 end

data/lib/page_rankr/version.rb

@@ -1,3 +1,3 @@
 module PageRankr
-  VERSION = "1.7.1"
+  VERSION = "2.0.0"
 end

data/spec/page_rankr_spec.rb

@@ -14,47 +14,39 @@ describe PageRankr do
   describe "#rank_trackers" do
     subject{ PageRankr.rank_trackers }
     
-    it{ should include(:alexa) }
+    it{ should include(:alexa_us) }
+    it{ should include(:alexa_global) }
+    it{ should include(:compete) }
     it{ should include(:google) }
   end
 
   describe "#ranks" do
     describe "success" do
-      subject{ PageRankr.ranks("google.com") }
+      subject{ PageRankr.ranks("www.google.com") }
   
-      it{ should have_key(:alexa) }
+      it{ should have_key(:alexa_us) }
+      it{ should have_key(:alexa_global) }
       it{ should have_key(:google) }
+      it{ should have_key(:compete) }
     
-      let(:alexa_rank) { subject[:alexa] }
-      it{ alexa_rank.should have_key(:us) }
-      it{ alexa_rank.should have_key(:global) }
-    
-      let(:alexa_us_rank) { alexa_rank[:us] }
-      it{ alexa_us_rank.should >= 1 }
-      let(:alexa_global_rank) { alexa_rank[:global] }
-      it{ alexa_global_rank.should >= 1 }
-    
-      let(:google_rank) { subject[:google] }
-      it{ google_rank.should be_in(0..10)}
+      it{ subject[:alexa_us].should >= 1 }
+      it{ subject[:alexa_global].should >= 1 }
+      it{ subject[:compete].should >= 1 }
+      it{ subject[:google].should be_in(0..10) }
     end
     
     describe "failure" do
       subject{ PageRankr.ranks("please-dont-register-a-site-that-breaks-this-test.com") }
       
-      it{ should have_key(:alexa) }
+      it{ should have_key(:alexa_us) }
+      it{ should have_key(:alexa_global) }
       it{ should have_key(:google) }
-      
-      let(:alexa_rank) { subject[:alexa] }
-      it{ alexa_rank.should have_key(:us) }
-      it{ alexa_rank.should have_key(:global) }
-    
-      let(:alexa_us_rank) { alexa_rank[:us] }
-      it{ alexa_us_rank.should == 0 }
-      let(:alexa_global_rank) { alexa_rank[:global] }
-      it{ alexa_global_rank.should == 0 }
+      it{ should have_key(:compete) }
     
-      let(:google_rank) { subject[:google] }
-      it{ google_rank.should == -1 }
+      it{ subject[:alexa_us].should be_nil }
+      it{ subject[:alexa_global].should be_nil }
+      it{ subject[:compete].should be_nil }
+      it{ subject[:google].should be_nil }
     end
   end
   
@@ -62,8 +54,6 @@ describe PageRankr do
     subject{ PageRankr.backlink_trackers }
     
     it{ should include(:alexa) }
-    it{ should include(:alltheweb) }
-    it{ should include(:altavista) }
     it{ should include(:bing) }
     it{ should include(:google) }
     it{ should include(:yahoo) }
@@ -71,11 +61,11 @@ describe PageRankr do
   
   describe "#backlinks" do
     describe "success" do
-      subject{ PageRankr.backlinks("google.com") }
+      subject{ PageRankr.backlinks("www.google.com") }
     
       PageRankr.backlink_trackers.each do |tracker|
         it{ should have_key(tracker) }
-        it{ subject[tracker].should >= 0 }
+        it{ subject[tracker].should > 0 }
       end
     end
     
@@ -84,7 +74,7 @@ describe PageRankr do
       
       PageRankr.backlink_trackers.each do |tracker|
         it{ should have_key(tracker) }
-        it{ subject[tracker].should == 0 }
+        it{ subject[tracker].should be_nil }
       end
     end
   end
@@ -98,11 +88,11 @@ describe PageRankr do
   
   describe "#indexes" do
     describe "success" do
-      subject{ PageRankr.indexes("google.com") }
+      subject{ PageRankr.indexes("www.google.com") }
       
       PageRankr.index_trackers.each do |tracker|
         it{ should have_key(tracker) }
-        it{ subject[tracker].should >= 0 }
+        it{ subject[tracker].should > 0 }
       end
     end
     
@@ -111,7 +101,7 @@ describe PageRankr do
       
       PageRankr.index_trackers.each do |tracker|
         it{ should have_key(tracker) }
-        it{ subject[tracker].should == 0 }
+        it{ subject[tracker].should be_nil }
       end
     end
   end

metadata

@@ -3,10 +3,10 @@ name: PageRankr
 version: !ruby/object:Gem::Version 
   prerelease: false
   segments: 
-  - 1
-  - 7
-  - 1
-  version: 1.7.1
+  - 2
+  - 0
+  - 0
+  version: 2.0.0
 platform: ruby
 authors: 
 - Allen Madsen
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-02-01 00:00:00 -05:00
+date: 2011-02-02 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -92,6 +92,36 @@ dependencies:
         version: 1.4.6
   type: :runtime
   version_requirements: *id005
+- !ruby/object:Gem::Dependency 
+  name: public_suffix_service
+  prerelease: false
+  requirement: &id006 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        - 8
+        - 1
+        version: 0.8.1
+  type: :runtime
+  version_requirements: *id006
+- !ruby/object:Gem::Dependency 
+  name: typhoeus
+  prerelease: false
+  requirement: &id007 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        - 2
+        - 1
+        version: 0.2.1
+  type: :runtime
+  version_requirements: *id007
 description: Easy way to retrieve Google Page Rank, Alexa Rank, and backlink counts
 email: 
 - blatyo@gmail.com
@@ -115,8 +145,6 @@ files:
 - lib/page_rankr/backlink.rb
 - lib/page_rankr/backlinks.rb
 - lib/page_rankr/backlinks/alexa.rb
-- lib/page_rankr/backlinks/alltheweb.rb
-- lib/page_rankr/backlinks/altavista.rb
 - lib/page_rankr/backlinks/bing.rb
 - lib/page_rankr/backlinks/google.rb
 - lib/page_rankr/backlinks/yahoo.rb
@@ -126,10 +154,12 @@ files:
 - lib/page_rankr/indexes/google.rb
 - lib/page_rankr/rank.rb
 - lib/page_rankr/ranks.rb
-- lib/page_rankr/ranks/alexa.rb
+- lib/page_rankr/ranks/alexa_global.rb
+- lib/page_rankr/ranks/alexa_us.rb
 - lib/page_rankr/ranks/compete.rb
 - lib/page_rankr/ranks/google.rb
 - lib/page_rankr/ranks/google/checksum.rb
+- lib/page_rankr/site.rb
 - lib/page_rankr/tracker.rb
 - lib/page_rankr/version.rb
 - spec/page_rankr_spec.rb

data/lib/page_rankr/backlinks/alltheweb.rb

@@ -1,15 +0,0 @@
-require 'cgi'
-
-module PageRankr
-  class Backlinks < Tracker
-    class Alltheweb < Backlink
-      def url(site)
-        "http://www.alltheweb.com/search?q=link%3A#{CGI.escape(site)}"
-      end
-      
-      def xpath
-        "//span[@class='ofSoMany']/text()"
-      end
-    end
-  end
-end

data/lib/page_rankr/backlinks/altavista.rb

@@ -1,15 +0,0 @@
-require 'cgi'
-
-module PageRankr
-  class Backlinks < Tracker
-    class Altavista < Backlink
-      def url(site)
-        "http://www.altavista.com/web/results?q=link%3A#{CGI.escape(site)}"
-      end
-      
-      def xpath
-        "//a[@class='lbl']/text()"
-      end
-    end
-  end
-end

data/lib/page_rankr/ranks/alexa.rb

@@ -1,26 +0,0 @@
-require 'open-uri'
-require 'cgi'
-require 'nokogiri'
-
-module PageRankr
-  class Ranks < Tracker
-    class Alexa < Rank
-      def initialize(site)
-        @rank = {}
-        xpath.each_key do |key|
-          @rank[key] = Nokogiri::HTML(open(url(site))).search(xpath[key]).to_s.to_i
-        end
-        @rank
-      end
-      
-      def xpath
-        { :us     => "//reach/@rank", 
-          :global => "//popularity/@text" }
-      end
-      
-      def url(site)
-        "http://data.alexa.com/data?cli=10&dat=snbamz&url=#{CGI.escape(site)}"
-      end
-    end
-  end
-end