spidey 0.0.2 → 0.0.3

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Joey Aghion, Art.sy Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,27 +1,52 @@
 Spidey
 ======
 
+Spidey provides a bare-bones framework for crawling and scraping web sites.
 
 
 Example
 -------
 
-
-Strategies
+This [non-working] example _spider_ crawls the ebay.com home page, follows links to auction pages, and finally records a few scraped item details as a _result_.
+
+    class EbaySpider < Spidey::AbstractSpider
+      handle "http://www.ebay.com", :process_home
+      
+      def process_home(page, default_data = {})
+        page.links_with(href: /auction\.aspx/).each do |link|
+          handle resolve_url(link.href, page), :process_auction, auction_title: link.text
+        end
+      end
+      
+      def process_auction(page, default_data = {})
+        record default_data.merge(sale_price: page.search('.sale_price').text)
+      end
+    end
+    
+    spider = EbaySpider.new verbose: true
+    spider.crawl max_urls: 100
+
+Implement a _spider_ class extending `Spidey::AbstractSpider` for each target site. The class can declare starting URLs with class-level calls to `handle`. Spidey invokes each of the methods specified in those calls, passing in the resulting `page` (a [Mechanize](http://mechanize.rubyforge.org/) [Page](http://mechanize.rubyforge.org/Mechanize/Page.html) object) and, optionally, some scraped data. The methods can do whatever processing of the page is necessary, calling `handle` with additional URLs to crawl and/or `record` with scraped results.
+
+
+Storage Strategies
 ----------
 
-### In-Memory
+By default, the lists of URLs being crawled, results scraped, and errors encountered are stored as simple arrays in the spider (i.e., in memory):
 
-### Mongo
+    spider.urls     # => ["http://www.ebay.com", "http://www.ebay.com/...", ...]
+    spider.results  # => [{auction_title: "...", sale_price: "..."}, ...]
+    spider.errors   # => [{url: "...", handler: :process_home, error: FooException}, ...]
 
+Add the [spidey-mongo](https://github.com/joeyAghion/spidey-mongo) gem and include `Spidey::Strategies::Mongo` in your spider to instead use MongoDB to persist these data. [See the docs](https://github.com/joeyAghion/spidey-mongo) for more information.
 
-Contributing
-------------
 
 To Do
 -----
-* Add examples
+* Add working examples
+* Spidey works well for crawling public web pages, but since little effort is undertaken to preserve the crawler's state across requests, it works less well when particular cookies or sequences of form submissions are required. [Mechanize](http://mechanize.rubyforge.org/) supports this quite well, though, so Spidey could grow in that direction.
 
 
-License
--------
+Copyright
+---------
+Copyright (c) 2012 Joey Aghion, Art.sy Inc. See [LICENSE.txt](LICENSE.txt) for further details.

data/lib/spidey/version.rb

@@ -1,3 +1,3 @@
 module Spidey
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

data/spidey.gemspec

@@ -10,6 +10,7 @@ Gem::Specification.new do |s|
   s.homepage    = "https://github.com/joeyAghion/spidey"
   s.summary     = %q{A loose framework for crawling and scraping web sites.}
   s.description = %q{A loose framework for crawling and scraping web sites.}
+  s.license     = 'MIT'
   
   s.rubyforge_project = "spidey"
   

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spidey
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-06-26 00:00:00.000000000Z
+date: 2012-06-27 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70136044260380 !ruby/object:Gem::Requirement
+  requirement: &70359848145740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70136044260380
+  version_requirements: *70359848145740
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70136044256960 !ruby/object:Gem::Requirement
+  requirement: &70359848144760 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70136044256960
+  version_requirements: *70359848144760
 - !ruby/object:Gem::Dependency
   name: mechanize
-  requirement: &70136044255140 !ruby/object:Gem::Requirement
+  requirement: &70359848143880 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,7 +43,7 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70136044255140
+  version_requirements: *70359848143880
 description: A loose framework for crawling and scraping web sites.
 email:
 - joey@aghion.com
@@ -53,6 +53,7 @@ extra_rdoc_files: []
 files:
 - .gitignore
 - Gemfile
+- LICENSE.txt
 - README.md
 - Rakefile
 - lib/spidey.rb
@@ -62,7 +63,8 @@ files:
 - spec/spidey/abstract_spider_spec.rb
 - spidey.gemspec
 homepage: https://github.com/joeyAghion/spidey
-licenses: []
+licenses:
+- MIT
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -75,7 +77,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 3586547064482976006
+      hash: -1311415898264218872
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -84,7 +86,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 3586547064482976006
+      hash: -1311415898264218872
 requirements: []
 rubyforge_project: spidey
 rubygems_version: 1.8.10