spidey 0.0.3 → 0.0.4

data/README.md

@@ -1,32 +1,46 @@
 Spidey
 ======
 
-Spidey provides a bare-bones framework for crawling and scraping web sites.
+Spidey provides a bare-bones framework for crawling and scraping web sites. Its goal is to keep boilerplate scraping logic out of your code.
 
 
 Example
 -------
 
-This [non-working] example _spider_ crawls the ebay.com home page, follows links to auction pages, and finally records a few scraped item details as a _result_.
+This example _spider_ crawls an eBay page, follows links to category pages, continues to auction detail pages, and finally records a few scraped item details as a _result_.
 
-    class EbaySpider < Spidey::AbstractSpider
-      handle "http://www.ebay.com", :process_home
+    class EbayPetSuppliesSpider < Spidey::AbstractSpider
+      handle "http://pet-supplies.shop.ebay.com", :process_home
       
       def process_home(page, default_data = {})
-        page.links_with(href: /auction\.aspx/).each do |link|
-          handle resolve_url(link.href, page), :process_auction, auction_title: link.text
+        page.search("#AllCats a[role=menuitem]").each do |a|
+          handle resolve_url(a.attr('href'), page), :process_category, category: a.text.strip
+        end
+      end
+      
+      def process_category(page, default_data = {})
+        page.search("#ResultSetItems table.li td.dtl a").each do |a|
+          handle resolve_url(a.attr('href'), page), :process_auction, default_data.merge(title: a.text.strip)
         end
       end
       
       def process_auction(page, default_data = {})
-        record default_data.merge(sale_price: page.search('.sale_price').text)
+        image_el = page.search('div.vi-ipic1 img').first
+        price_el = page.search('span[itemprop=price]').first
+        record default_data.merge(
+          image_url: (image_el.attr('src') if image_el),
+          price: price_el.text.strip
+        )
       end
+    
     end
     
-    spider = EbaySpider.new verbose: true
+    spider = EbayPetSuppliesSpider.new verbose: true
     spider.crawl max_urls: 100
+    
+    spider.results  # => [{category: "Aquarium & Fish", title: "5 Gal. Fish Tank"...
 
-Implement a _spider_ class extending `Spidey::AbstractSpider` for each target site. The class can declare starting URLs with class-level calls to `handle`. Spidey invokes each of the methods specified in those calls, passing in the resulting `page` (a [Mechanize](http://mechanize.rubyforge.org/) [Page](http://mechanize.rubyforge.org/Mechanize/Page.html) object) and, optionally, some scraped data. The methods can do whatever processing of the page is necessary, calling `handle` with additional URLs to crawl and/or `record` with scraped results.
+Implement a _spider_ class extending `Spidey::AbstractSpider` for each target site. The class can declare starting URLs by calling `handle` at the class level. Spidey invokes each of the methods specified in those calls, passing in the resulting `page` (a [Mechanize](http://mechanize.rubyforge.org/) [Page](http://mechanize.rubyforge.org/Mechanize/Page.html) object) and, optionally, some scraped data. The methods can do whatever processing of the page is necessary, calling `handle` with additional URLs to crawl and/or `record` with scraped results.
 
 
 Storage Strategies
@@ -38,12 +52,17 @@ By default, the lists of URLs being crawled, results scraped, and errors encount
     spider.results  # => [{auction_title: "...", sale_price: "..."}, ...]
     spider.errors   # => [{url: "...", handler: :process_home, error: FooException}, ...]
 
-Add the [spidey-mongo](https://github.com/joeyAghion/spidey-mongo) gem and include `Spidey::Strategies::Mongo` in your spider to instead use MongoDB to persist these data. [See the docs](https://github.com/joeyAghion/spidey-mongo) for more information.
+Add the [spidey-mongo](https://github.com/joeyAghion/spidey-mongo) gem and include `Spidey::Strategies::Mongo` in your spider to instead use MongoDB to persist these data. [See the docs](https://github.com/joeyAghion/spidey-mongo) for more information. Or, you can implement your own strategy by overriding the appropriate methods from `AbstractSpider`.
+
+
+Contributing
+------------
+
+Spidey is very much a work in progress. Pull requests welcome.
 
 
 To Do
 -----
-* Add working examples
 * Spidey works well for crawling public web pages, but since little effort is undertaken to preserve the crawler's state across requests, it works less well when particular cookies or sequences of form submissions are required. [Mechanize](http://mechanize.rubyforge.org/) supports this quite well, though, so Spidey could grow in that direction.
 
 

data/examples/ebay_pet_supplies_spider.rb

@@ -0,0 +1,25 @@
+class EbayPetSuppliesSpider < Spidey::AbstractSpider
+  handle "http://pet-supplies.shop.ebay.com", :process_home
+  
+  def process_home(page, default_data = {})
+    page.search("#AllCats a[role=menuitem]").each do |a|
+      handle resolve_url(a.attr('href'), page), :process_category, category: a.text.strip
+    end
+  end
+  
+  def process_category(page, default_data = {})
+    page.search("#ResultSetItems table.li td.dtl a").each do |a|
+      handle resolve_url(a.attr('href'), page), :process_auction, default_data.merge(title: a.text.strip)
+    end
+  end
+  
+  def process_auction(page, default_data = {})
+    image_el = page.search('div.vi-ipic1 img').first
+    price_el = page.search('span[itemprop=price]').first
+    record default_data.merge(
+      image_url: (image_el.attr('src') if image_el),
+      price: price_el.text.strip
+    )
+  end
+  
+end

data/lib/spidey/version.rb

@@ -1,3 +1,3 @@
 module Spidey
-  VERSION = "0.0.3"
+  VERSION = "0.0.4"
 end

data/spidey.gemspec

@@ -21,6 +21,7 @@ Gem::Specification.new do |s|
   
   s.add_development_dependency "rake"
   s.add_development_dependency "rspec"
+  s.add_development_dependency "ruby-debug19"
   
   s.add_runtime_dependency "mechanize"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spidey
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.4
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-06-27 00:00:00.000000000Z
+date: 2012-12-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70359848145740 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,31 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70359848145740
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70359848144760 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: ruby-debug19
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +53,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70359848144760
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: mechanize
-  requirement: &70359848143880 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,7 +69,12 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70359848143880
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: A loose framework for crawling and scraping web sites.
 email:
 - joey@aghion.com
@@ -56,6 +87,7 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
+- examples/ebay_pet_supplies_spider.rb
 - lib/spidey.rb
 - lib/spidey/abstract_spider.rb
 - lib/spidey/version.rb
@@ -77,7 +109,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1311415898264218872
+      hash: -3162500508741796001
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -86,10 +118,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1311415898264218872
+      hash: -3162500508741796001
 requirements: []
 rubyforge_project: spidey
-rubygems_version: 1.8.10
+rubygems_version: 1.8.24
 signing_key: 
 specification_version: 3
 summary: A loose framework for crawling and scraping web sites.