scrapes 0.2.0

data/LICENSE

@@ -0,0 +1,22 @@
+== License
+
+Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,123 @@
+= Scrapes
+
+Scrapes is a framework for crawling and scraping multi-page web sites.
+
+Unlike other scraping frameworks, Scrapes is designed to work with "dirty"
+web sites.  That is, web sites that were not designed to have their data
+extracted programmatically.
+
+It includes features for both the initial development of a scraper, and the
+continued maintenance of that scraper.  These features include:
+
+* Rule based selection and extraction of data that can use CSS selectors or
+  pseudo XPath expressions
+* Caching system so that during development you don't have to continuously
+  download pages from a web server while you experiment with your selectors and
+  extractors
+* Validation system that helps detect web site changes that would
+  otherwise invalidate your extraction rules
+* Support for initiating a session with the web server, and passing session
+  cookies back to the web server
+* When all else fails, you can run a web page through the xsltproc XSLT
+  processor to generate an XML document that can then be run through your
+  rule based parser
+* Useful set of post-processing methods such as normalize_name
+
+
+== Installing Scrapes
+
+ gem install scrapes --include-dependencies
+
+
+== Dependencies
+
+* Hpricot: http://code.whytheluckystiff.net/hpricot/wiki/AnHpricotShowcase
+* Rextra: http://rubyforge.org/projects/rextra2/
+
+
+== Quick Start
+
+You start by writing a class for parsing a single page:
+
+ # process the Google.com index.html page
+ class GoogleMain < Scrapes::Page
+   # make sure that the :about_link rule matched the web page
+   validates_presence_of(:about_link)
+ 
+   # extract the link to the about page
+   rule(:about_link, 'a[@href*="about"]', '@href', 1)
+ end
+
+ # process the Google.com about page
+ class GoogleAbout < Scrapes::Page
+   # ensure the :title rule below matches the web page
+   validates_presence_of(:title)
+ 
+   # extract the text inside the <title></title> tag
+   rule(:title, 'title', 'text()', 1)
+ end
+
+Then you start a scraping session and use those classes to process the web
+site:
+
+ Scrapes::Session.start do |session|
+   session.page(GoogleMain, 'http://google.com') do |main_page|
+     session.page(GoogleAbout, main_page.about_link) do |about_page|
+       puts about_page.title + ': ' + session.absolute_uri(main_page.about_link)
+     end
+   end
+ end
+
+On my machine, this code produces:
+ About Google: http://www.google.com/intl/en/about.html
+
+For more information, please review the following classes:
+* Scrapes::Session 
+* Scrapes::Page
+* Scrapes::RuleParser
+* Scrapes::Hpricot::Extractors
+
+
+== Development Tips
+
+=== Add something like this to your .irbrc:
+
+  require 'rubygems'
+  require 'yaml'
+  require 'open-uri'
+  require 'hpricot'
+  require 'scrapes'
+  def h(url) Hpricot(open(url)) end
+Then use like this in irb to understand how Hpricot selectors work:
+  doc = h 'http://www.foobar.com/'
+  links = doc.search('table/a[@href]')  # for example
+To understand the text extractors:
+  texts(links)
+  word(links.first)  # etc..
+
+
+=== Converting normal Xpath to Hpricot Xpath, sort of:
+
+There are various add-ons to firefox, for example, that display the Xpath to a
+selected node.  Hpricot uses a different sytanx however,
+(http://code.whytheluckystiff.net/hpricot/wiki/SupportedXpathExpressions).  The
+following method is a first try at the conversion:
+  def xpath_to_hpricot path
+    path.split('/').reject{|e|e=~/^(html|tbody)$/ or e.blank?}.map do |e|
+      res = e.sub(/\[/,':eq(').sub(/\]/,')')
+      res.sub(/\d+/, (/(\d+)/.match(res).to_s.to_i - 1).to_s)
+    end.join('//')
+  end
+
+=== Hpricot bugs
+
+* This selector will hang, 'a[href="this"]' and this one won't, 'a[@href="this"]'.
+  Just make sure you have the '@' in front of the attribute name.
+
+
+== Credits
+
+* Peter Jones, author and maintainer
+* Michael Garriss, author and maintainer
+* Bob Showalter, continuous improvements and maintenance
+* Assaf Arkin, rule inspiration from http://trac.labnotes.org/cgi-bin/trac.cgi/wiki/Ruby/MicroformatParser

data/demo/demo.rb

@@ -0,0 +1,33 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+require 'scrapes'
+
+Scrapes::Session.start do |session|
+  session.page(GoogleMain, 'http://google.com') do |main_page|
+    session.page(GoogleAbout, main_page.about_link) do |about_page|
+      puts about_page.title + ': ' + session.absolute_uri(main_page.about_link)
+    end
+  end
+end

data/demo/pages/about.rb

@@ -0,0 +1,32 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+class GoogleAbout < Scrapes::Page
+  # ensure the :title rule below matches the web page
+  validates_presence_of(:title)
+
+  # extract the text inside the <title></title> tag
+  rule(:title, 'title', 'text()', 1)
+
+end

data/demo/pages/main.rb

@@ -0,0 +1,32 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+class GoogleMain < Scrapes::Page
+  # make sure that the :about_link rule matched the web page
+  validates_presence_of(:about_link)
+
+  # extract the link to the about page
+  rule(:about_link, 'a[@href*="about"]', '@href', 1)
+
+end

data/lib/scrapes.rb

@@ -0,0 +1,41 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+require 'rubygems'
+################################################################################
+require 'scrapes/initializer'
+require 'scrapes/to_proxy'
+require 'scrapes/page'
+require 'scrapes/cookies'
+require 'scrapes/cache'
+require 'scrapes/crawler'
+require 'scrapes/session'
+require 'scrapes/cookbook'
+################################################################################
+include Scrapes::Hpricot::Extractors
+################################################################################
+Scrapes::Initializer.run do |initializer|
+  initializer.process
+end
+################################################################################

data/lib/scrapes/cache.rb

@@ -0,0 +1,110 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+require 'ostruct'
+require 'digest/md5'
+################################################################################
+module Scrapes
+  ################################################################################
+  # Cache web pages
+  class Cache
+    ################################################################################
+    # Enable/disable caching
+    attr_accessor :enabled
+
+    ################################################################################
+    # Set the directory to use for caching
+    attr_accessor :directory
+
+    ################################################################################
+    # Set to a proc that given an URI, translates it to a file system name
+    # default is to MD5 the URL
+    attr_accessor :uri_translator
+
+    ################################################################################
+    def initialize
+      @directory = File.expand_path('cache')
+      @enabled = false
+      @uri_translator = nil
+    end
+
+    ################################################################################
+    # Disables caching while the given block is active.
+    def without_cache
+      state = @enabled
+      @enabled = false
+      yield if block_given?
+      @enabled = state
+    end
+
+    ################################################################################
+    # Checks the cache to see if there is a match for the given URI
+    def check (uri)
+      return nil unless @enabled
+
+      # FIXME check some time limits around this
+      cache_name = translate_uri(uri)
+      
+      file = File.join(@directory, cache_name)
+      File.exist?(file) and File.open(file) do |f|
+        return OpenStruct.new(:body => f.read, :cache_file => file)
+      end
+    end
+
+    ################################################################################
+    # Updates the cache by placing the data for the give URI on the file system.
+    def update (uri, data)
+      return nil unless @enabled
+
+      cache_name = translate_uri(uri)
+      mkdir # FIXME include cache_name to build all necessary directories 
+
+      File.open(File.join(@directory, cache_name), 'w') do |f|
+        f << data
+      end
+    end
+
+    ################################################################################
+    # helper method to translate a URL to a MD5
+    def uri_to_md5 (uri)
+      Digest::MD5.hexdigest(uri.to_s)
+    end
+
+    ################################################################################
+    private
+
+    ################################################################################
+    def mkdir
+      Dir.mkdir(@directory) unless File.exist?(@directory)
+    end
+    
+    ################################################################################
+    def translate_uri (uri)
+      @uri_translator ? @uri_translator.call(uri) : uri_to_md5(uri)
+    end
+
+  end
+  ################################################################################
+end
+################################################################################

data/lib/scrapes/cookbook.rb

@@ -0,0 +1,53 @@
+################################################################################
+#
+# Copyright (C) 2006 Peter J Jones (pjones@pmade.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+# 
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+# 
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+################################################################################
+module Scrapes
+  ################################################################################
+  module RE
+    EMAIL = /[a-zA-Z]([.]?([[:alnum:]_-]+)*)?@([[:alnum:]\-_]+\.)+[a-zA-Z]{2,4}/
+    US_PHONE_NUMBER = /\(?\d{3}[-) ]\s{0,3}\d{3}[- ]\d{4}/
+  end
+  ################################################################################
+  def self.normalize_name names
+    require 'unicode'
+    result = names.map do |part|
+      result = ""
+      Unicode.normalize_D(part).each_byte{|byte| (result << byte) if byte < 128}
+      result = result.strip.chomp('.')
+      result.upcase! if result =~ /^(jr|sr|ii+|iv|vi*)$/i
+      result = nil if result =~ /^Rev|Ph\.D$/i
+      result
+    end.compact
+    if result[-2] =~ /,$/
+      extra = result.pop
+      result.unshift result.pop
+      result << extra
+      result.join(' ')
+    else                                                                                  
+      result.unshift(result.pop + ',').join(' ')
+    end
+  end
+  ################################################################################
+end
+################################################################################