john_stamos 0.0.1 → 0.0.2

data/README.md

@@ -1,26 +1,35 @@
-# JohnStamos 
+# JohnStamos
 
 The greasiest Pinterest client. JohnStamos exposes an interface you can use to query Pinterest for data about individual Pins, Pinners and Pin searches.  Have mercy.
 
 [![Build Status](https://travis-ci.org/mariozig/john_stamos.png?branch=master)](https://travis-ci.org/mariozig/john_stamos)
+[![Code Climate](https://codeclimate.com/github/mariozig/john_stamos.png)](https://codeclimate.com/github/mariozig/john_stamos)
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'john_stamos'
+```
+gem 'john_stamos'
+```
 
 And then execute:
 
-    $ bundle
+```
+$ bundle
+```
 
 Or install it yourself as:
 
-    $ gem install john_stamos
+```
+$ gem install john_stamos
+```
 
 Or finally, if none of that stuff applies to you, do what it takes to install the gem and add this to your script:
 
-    require 'john_stamos'
+```ruby
+require 'john_stamos'
+```
 
 ## Usage
 
@@ -28,95 +37,102 @@ Or finally, if none of that stuff applies to you, do what it takes to install th
 
 Here's a typical use-case: You want to search Pinterest for some pins.  Once you have those pins, you want to access information about each individual pin as well as information about the person who pinned it.
 
-    client = JohnStamos::Client.new
-    
-    # Search for some sweet Pins of "pallet gardens"
-    pins = client.search_pins("pallet gardens", limit: 10)
-    # Returns an Array of Pin objects
-    
-    # The first Pin will be fun to investigate.
-    first_pallet_garden_pin = pallet_search.first
-    
-    # Print some info about the first Pin description
-    puts first_pallet_garden_pin.description
-    # => "A cool pallet garden"
-    
-    # the full sized image
-    puts first_pallet_garden_pin.image
-    # => "http://media-cache-ak0.pinimg.com/originals....whatever.jpg"
-    
-    # Pinterest's ID for this Pin
-    puts first_pallet_garden_pin.id
-    # => "123456789"
-    
-    # URL to the Pin
-    puts first_pallet_garden_pin.url
-    # => "http://pinterest.com/pin/123456789"
-    
-    # We can even get at the user who Pinned this Pin
-    first_pallet_garden_pin.pinner
-    
-    # Print the Pinner's full name
-    puts first_pallet_garden_pin.pinner.full_name
-    # => "Little Bobby Tables"
+```ruby
+client = JohnStamos::Client.new
+
+# Search for some sweet Pins of "pallet gardens"
+pins = client.search_pins("pallet gardens", limit: 10)
+# Returns an Array of Pin objects
+
+# The first Pin will be fun to investigate.
+first_pallet_garden_pin = pallet_search.first
+
+# Print some info about the first Pin description
+puts first_pallet_garden_pin.description
+# => "A cool pallet garden"
+
+# the full sized image
+puts first_pallet_garden_pin.image
+# => "http://media-cache-ak0.pinimg.com/originals....whatever.jpg"
+
+# Pinterest's ID for this Pin
+puts first_pallet_garden_pin.id
+# => "123456789"
+
+# URL to the Pin
+puts first_pallet_garden_pin.url
+# => "http://pinterest.com/pin/123456789"
+
+# We can even get at the user who Pinned this Pin
+first_pallet_garden_pin.pinner
+
+# Print the Pinner's full name
+puts first_pallet_garden_pin.pinner.full_name
+# => "Little Bobby Tables"
+```
 
 ### Using the Client
 
-The Client is the single entry point into all things JohnStamos. 
+The Client is the single entry point into all things JohnStamos.
 
-    # Create a new Client
-    client = JohnStamos::Client.new
-    
-    # The Client also supports using a proxy
-    client_with_proxy = JohnStamos::Client.new(proxy: 'http://127.1.1.1:4747')
-    
-    # Note: If you're hosted on Heroku, you will want to use a proxy.
+```ruby
+# Create a new Client
+client = JohnStamos::Client.new
 
+# The Client also supports using a proxy
+client_with_proxy = JohnStamos::Client.new(proxy: 'http://127.1.1.1:4747')
+
+# Note: If you're hosted on Heroku, you will want to use a proxy.
+```
 
 ### Accessing Individual Pin Data
 
 There are times you might want to access an individual Pin's information.  All you need is the ID assigned to the Pin by Pinterest to get going.
 
-    client = JohnStamos::Client.new
-    
-    # Get a magnificent Pin using it's ID
-    perfect_strangers_pin = client.pin(161848180328631541)
-    
-    puts perfect_strangers_pin.description
-    # => "Balki Bartokomous and Cousin Larry!!! PERFECT STRANGERS...loved that show!"
-    
-    puts perfect_strangers_pin.image
-    # => "http://media-cache-ec0.pinimg.com/originals/4b/10/d3/4b10d3bd1c172c52934eb2d79ff0031f.jpg"
-    
-    # If you didn't look at that image above, please do it before reading further
-    
-    puts perfect_strangers_pin.url
-    # => "http://pinterest.com/pin/161848180328631541/"
-    
-    puts perfect_strangers_pin.repin_count
-    # => 1
+```ruby
+client = JohnStamos::Client.new
+
+# Get a magnificent Pin using it's ID
+perfect_strangers_pin = client.pin(161848180328631541)
+
+puts perfect_strangers_pin.description
+# => "Balki Bartokomous and Cousin Larry!!! PERFECT STRANGERS...loved that show!"
+
+puts perfect_strangers_pin.image
+# => "http://media-cache-ec0.pinimg.com/originals/4b/10/d3/4b10d3bd1c172c52934eb2d79ff0031f.jpg"
+
+# If you didn't look at that image above, please do it before reading further
+
+puts perfect_strangers_pin.url
+# => "http://pinterest.com/pin/161848180328631541/"
+
+puts perfect_strangers_pin.repin_count
+# => 1
+```
 
 ### Accessing Pinner Data
 
 You can also query Pinterest for information about specific Pinners. Just pass in the Pinner's user name and you will be good to go.
 
-    client = JohnStamos::Client.new
-    
-    # Get Michelle Obama's account
-    michelle_obama = client.pinner("michelleobama")
-    
-    # Print some of the Pinner's information
-    puts michelle_obama.last_name
-    # => "Obama"
-    
-    puts michelle_obama.first_name
-    # => "Michelle"
-    
-    puts michelle_obama.pin_count
-    # => "81"
-    
-    puts michelle_obama.website_url
-    # => "http://www.barackobama.com"
+```ruby
+client = JohnStamos::Client.new
+
+# Get Michelle Obama's account
+michelle_obama = client.pinner("michelleobama")
+
+# Print some of the Pinner's information
+puts michelle_obama.last_name
+# => "Obama"
+
+puts michelle_obama.first_name
+# => "Michelle"
+
+puts michelle_obama.pin_count
+# => "81"
+
+puts michelle_obama.website_url
+# => "http://www.barackobama.com"
+```
 
 ## Contributing
 

data/john_stamos.gemspec

@@ -25,8 +25,9 @@ Gem::Specification.new do |gem|
   gem.add_development_dependency "vcr"
   gem.add_development_dependency "guard-rspec"
 
-  gem.add_dependency "mechanize"
-  gem.add_dependency "rest-client"
+  gem.add_dependency "nokogiri"
+  gem.add_dependency "faraday"
+  gem.add_dependency "typhoeus"
   gem.add_dependency "json"
   gem.add_dependency "launchy"
 end

data/lib/john_stamos/client.rb

@@ -1,3 +1,5 @@
+require 'logger'
+
 class JohnStamos::Client
   attr_accessor :proxy
 
@@ -23,35 +25,48 @@ class JohnStamos::Client
   end
 
   def page_content(url)
-    begin
-      page = mechanize_agent.get(url)
-    rescue Mechanize::ResponseReadError => e
-      $stderr.puts "#{e.class}: #{e.message}"
-      $stderr.puts "An error occured; attempting to force_parse"
-      page = e.force_parse
-    end
+    response = make_request(url)
 
-    page
+    Nokogiri::HTML(response)
   end
 
   def json_content(url, params)
-    RestClient.proxy = @proxy
-    response = RestClient.get(url, params: params, accept: :json, "X-Requested-With" => "XMLHttpRequest")
+    response = make_json_request(url, params)
 
     JSON.parse(response)
   end
 
   private
-    def mechanize_agent
-      agent = Mechanize.new
-      agent.user_agent_alias = 'Mac Safari'
+    def make_request(url, params={}, accept_json=false)
+      request_headers = build_request_headers(accept_json)
 
-      if @proxy
-        proxy_uri = URI.parse(@proxy)
-        agent.set_proxy(proxy_uri.hostname, proxy_uri.port)
+      response = pinterest_connection.get do |req|
+        req.url url, params
+        req.headers = request_headers
       end
 
-      agent
+      response.body
+    end
+
+    def make_json_request(url, params={})
+      make_request(url, params, true)
+    end
+
+    def build_request_headers(accept_json=false)
+      headers = {}
+
+      if accept_json
+        headers['Accept'] = 'application/json'
+        headers['X-Requested-With'] = 'XMLHttpRequest'
+      end
+      headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36'
+
+      headers
+    end
+
+    def pinterest_connection
+      pinterest_url = 'http://pinterest.com'
+      @pinterest_connection ||= @proxy.nil? ? Faraday.new(url: pinterest_url) : Faraday.new(url: pinterest_url, proxy: @proxy)
     end
 
 end

data/lib/john_stamos/pin_search.rb

@@ -25,11 +25,11 @@ class JohnStamos::PinSearch
 
   def first_retrieval_url
     raise JohnStamos::MissingSearchText if @search_text.nil?
-    "http://pinterest.com/search/pins/?q=#{URI::encode(@search_text)}"
+    "/search/pins/?q=#{URI::encode(@search_text)}"
   end
 
   def subsequent_retrieval_url
-    'http://pinterest.com/resource/SearchResource/get/'
+    '/resource/SearchResource/get/'
   end
 
   def first_retrieval!
@@ -134,7 +134,8 @@ class JohnStamos::PinSearch
 
       url_params = {
                     source_url: "/search/pins/?q=#{URI::encode(search_text)}",
-                    "_" => 1375127302668,
+                   # Pass in the current time in miliseconds (like pinterest does) to prevent caching
+                    "_" => (Time.now.to_f * 1000.0).to_i,
                     data: data_json.to_json
                    }
       url_params

data/lib/john_stamos/pinner.rb

@@ -54,7 +54,7 @@ class JohnStamos::Pinner
 
   private
     def page
-      @page ||= @client.page_content(url)
+      @page ||= @client.page_content("/#{@username}/")
     end
 
     def embedded_pinner_json

data/lib/john_stamos/version.rb

@@ -1,3 +1,3 @@
 module JohnStamos
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/lib/john_stamos.rb

@@ -1,6 +1,7 @@
-require 'mechanize'
+require 'faraday'
+require 'typhoeus/adapters/faraday'
+require 'nokogiri'
 require 'json'
-require 'rest_client'
 require 'launchy'
 
 require 'john_stamos/version'
@@ -13,6 +14,8 @@ module JohnStamos
   class MissingSearchText < Exception; end
   class MissingNextBookmark < Exception; end
 
+  Faraday.default_adapter = :typhoeus
+
   private
     def self.uncle_jesse
       Launchy.open('http://www.youtube.com/watch?v=bLqAqIj8Rdc')

data/spec/lib/john_stamos/client_spec.rb

@@ -2,7 +2,7 @@ require 'spec_helper'
 
 describe JohnStamos::Client, :vcr do
   subject(:client) { JohnStamos::Client.new }
-  let(:pinterest_url) { 'https://pinterest.com' }
+  let(:pinterest_url) { 'http://pinterest.com' }
 
   describe '#search_pins' do
     it 'responds to #search_pins' do
@@ -49,99 +49,88 @@ describe JohnStamos::Client, :vcr do
     end
   end
 
-  describe '#json_content' do
-    it 'responds to #json_content' do
-      expect(client).to respond_to(:json_content)
+  describe '#page_content' do
+    it 'responds to #page_content' do
+      expect(client).to respond_to(:page_content)
     end
 
-    it 'returns a Hash' do
-      RestClient.stub(:get).and_return('{"json":"json"}')
-
-      expect(client.json_content('url', 'params')).to be_a(Hash)
+    it 'returns a Nokogiri::HTML::Document object' do
+      expect(client.page_content(pinterest_url)).to be_a(Nokogiri::HTML::Document)
     end
   end
 
-  describe '#mechanize_agent' do
-    let(:agent) { client.send(:mechanize_agent) }
-
-    context 'when using a proxy' do
-      before(:each) do
-        client.proxy = 'http://proxy.com:4747'
-      end
-
-      it 'sets the correct hostname' do
-        expect(agent.proxy_addr).to eq('proxy.com')
-      end
+  describe '#json_content' do
+    let(:fake_json) { '{ "key":"value" }' }
 
-      it 'sets the correct port' do
-        expect(agent.proxy_port).to eq(4747)
-      end
+    it 'responds to #json_content' do
+      expect(client).to respond_to(:json_content)
     end
 
-    context 'when not using a proxy' do
-      before(:each) do
-        client.proxy = nil
-      end
-
-      it 'does not set a hostname' do
-        expect(agent.proxy_addr).to be_nil
-      end
-
-      it 'does not set a port' do
-        expect(agent.proxy_port).to be_nil
-      end
+    it 'returns a Hash representation of the JSON' do
+      client.stub(:make_request).and_return(fake_json)
+      expect(client.json_content(nil, nil)).to be_a(Hash)
     end
   end
 
-  describe '#page_content' do
-    it 'responds to #page_content' do
-      expect(client).to respond_to(:page_content)
+  describe '#make_request' do
+    it 'returns a string representation of the response body' do
+      response = client.send(:make_request, pinterest_url, {})
+      expect(response).to be_a(String)
     end
+  end
+
+  describe '#make_json_request' do
+    it 'calls #make_request with the correct params' do
+      client.stub(:make_request).and_return(nil)
+      expect(client).to receive(:make_request).with('a', 'b', true)
 
-    it 'returns a Mechanize::Page' do
-      expect(client.page_content(pinterest_url)).to be_a(Mechanize::Page)
+      client.send(:make_json_request, 'a', 'b')
     end
   end
 
-  describe '#json_content' do
-    let(:fake_json) { '{"json":"json"}' }
-
-    it 'responds to #json_content' do
-      expect(client).to respond_to(:json_content)
+  describe '#pinterest_connection' do
+    context 'when using a proxy' do
+      it 'creates a new Faraday proxied instance' do
+        client.proxy = 'http://fakeproxy.com:1234'
+        expect(Faraday).to receive(:new).with({ url: pinterest_url, proxy: client.proxy })
+        client.send(:pinterest_connection)
+      end
     end
 
     context 'when not using a proxy' do
-      before(:each) do
-        allow(RestClient).to receive(:get).and_return(fake_json)
+      it 'creates a new Faraday instance' do
+        client.proxy = nil
+        expect(Faraday).to receive(:new).with({ url: pinterest_url })
+        client.send(:pinterest_connection)
       end
+    end
+  end
 
-      let(:json_content) { client.json_content('url', 'params') }
-
-      it 'returns a Hash representation of the JSON' do
-        expect(json_content).to be_a(Hash)
-      end
+  describe '#build_request_headers' do
+    it 'has a valid User-Agent' do
+      headers = client.send(:build_request_headers)
+      expect(headers['User-Agent']).to eq('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36')
+    end
 
-      it 'does not set RestClient proxy properties' do
-        json_content
-        expect(RestClient.proxy).to be_nil
+    context 'non-JSON requests' do
+      it 'do not contain JSON request related headers' do
+        headers = client.send(:build_request_headers)
+        expect(headers).not_to include('Accept', 'X-Requested-With')
       end
     end
 
-    context 'when using a proxy' do
-      let(:proxy) { 'http://proxy.com:31337' }
-      let(:proxy_client) { JohnStamos::Client.new({ proxy: proxy }) }
-
-      before(:each) do
-        allow(proxy_client).to receive(:proxy).and_return(proxy)
-        allow(RestClient).to receive(:get).and_return(fake_json)
+    context 'JSON requests' do
+      let(:json_headers) do
+        headers = {}
+        headers['Accept'] = 'application/json'
+        headers['X-Requested-With'] = 'XMLHttpRequest'
+        headers
       end
 
-      it 'sets RestClient proxy properties' do
-        proxy_client.json_content('url', 'params')
-        expect(RestClient.proxy).to eq(proxy_client.proxy)
+      it 'contain JSON request related headers' do
+        headers = client.send(:build_request_headers, true)
+        expect(headers).to include(json_headers)
       end
     end
-
-
   end
 end

data/spec/lib/john_stamos/pin_search_spec.rb

@@ -89,7 +89,7 @@ describe JohnStamos::PinSearch, :vcr do
 
     it 'returns the correct URL with the search text URL encoded' do
       scraper.search_text = search_text
-      expect(scraper.first_retrieval_url).to eq('http://pinterest.com/search/pins/?q=coffee%20roasting')
+      expect(scraper.first_retrieval_url).to eq('/search/pins/?q=coffee%20roasting')
     end
   end
 
@@ -102,15 +102,15 @@ describe JohnStamos::PinSearch, :vcr do
         expect { scraper.first_retrieval! }.to change{ scraper.next_bookmark }.from(nil)
       end
 
-      it 'sets next_bookmark with a hash for the next 50 results' do
+      it 'sets next_bookmark with a hash for the next batch of results' do
         scraper.first_retrieval!
         decrypted_hash = Base64.strict_decode64(scraper.next_bookmark)
         next_page_results_start_position = decrypted_hash.split('|')[0]
-        next_page_results_start_position.should eq("oo50")
+        next_page_results_start_position.should eq("oo25")
       end
 
       it 'should set pin_ids' do
-        expect { scraper.first_retrieval! }.to change{ scraper.pin_ids.length }.from(0).to(50)
+        expect { scraper.first_retrieval! }.to change{ scraper.pin_ids.length }.from(0).to(25)
       end
     end
 
@@ -133,7 +133,7 @@ describe JohnStamos::PinSearch, :vcr do
 
   describe '#subsequent_retrieval_url' do
     it 'returns the correct "resource" url' do
-      expect(scraper.subsequent_retrieval_url).to eq('http://pinterest.com/resource/SearchResource/get/')
+      expect(scraper.subsequent_retrieval_url).to eq('/resource/SearchResource/get/')
     end
   end
 
@@ -163,18 +163,18 @@ describe JohnStamos::PinSearch, :vcr do
         expect { scraper.subsequent_retrieval! }.to change{ scraper.next_bookmark }
       end
 
-      it 'sets next_bookmark with a hash for the next 50 results' do
+      it 'sets next_bookmark with a hash for the next batch of results' do
         scraper.subsequent_retrieval!
         decrypted_hash = Base64.strict_decode64(scraper.next_bookmark)
         next_page_results_start_position = decrypted_hash.split('|')[0]
-        expect(next_page_results_start_position).to eq("oo100")
+        expect(next_page_results_start_position).to eq("oo50")
       end
 
       it 'should append the new pin_ids' do
-        # The expected count can change based on Pinterest. Sometimes they give 99, sometimes 100.
+        # The expected count can change based on Pinterest. Sometimes they give 1 less than expected.
         # When you update the VCR cassettes it may cause this test to fail... please update it.
         scraper.subsequent_retrieval!
-        expect(scraper.pin_ids.length).to eq(100)
+        expect(scraper.pin_ids.length).to eq(50)
       end
     end
   end

data/spec/support/vcr_cassettes/JohnStamos_Client/_make_request/returns_a_string_representation_of_the_response_body.yml

@@ -0,0 +1,59 @@
+---
+http_interactions:
+- request:
+    method: get
+    uri: http://pinterest.com/
+    body:
+      encoding: US-ASCII
+      string: ''
+    headers:
+      User-Agent:
+      - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML,
+        like Gecko) Chrome/28.0.1500.95 Safari/537.36
+  response:
+    status:
+      code: 302
+      message: FOUND
+    headers:
+      Accept-Ranges:
+      - bytes
+      Age:
+      - '0'
+      Cache-Control:
+      - no-cache
+      Content-Type:
+      - text/html; charset=utf-8
+      Date:
+      - Thu, 22 Aug 2013 23:38:02 GMT
+      Location:
+      - https://pinterest.com/
+      Pinterest-Breed:
+      - CORGI
+      Pinterest-Generated-By:
+      - ngapp-1b3b34b2
+      Pinterest-Version:
+      - 5bdef68
+      Server:
+      - nginx
+      Set-Cookie:
+      - _pinterest_sess="eJzzNvE19PDKjbKoSql0NTLwCApxTPZPystO8yi3tY8vycxNtfUN8TX2q/Ks8HOJNPFNt7VVK04tLs5MsfXM8jSODMk28HPxyvY1iqzwCwk0jQyJNPXNSq/ydwnKisoNrfDL8gLSYTmeQH0AYikjhQ==";
+        Domain=.pinterest.com; expires=Sun, 17-Aug-2014 23:38:02 GMT; Max-Age=31103999;
+        Path=/
+      - csrftoken=3pu9GXdpS9N8ilNUApaOtopW0nV0tOOn; Domain=.pinterest.com; expires=Thu,
+        21-Aug-2014 23:38:02 GMT; Max-Age=31449600; Path=/
+      Vary:
+      - Cookie
+      Via:
+      - 1.1 varnish
+      X-Varnish:
+      - '2111788339'
+      Content-Length:
+      - '0'
+      Connection:
+      - keep-alive
+    body:
+      encoding: UTF-8
+      string: ''
+    http_version: 
+  recorded_at: Thu, 22 Aug 2013 23:38:02 GMT
+recorded_with: VCR 2.5.0