grubby 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 88c8ecc06ffba254ee9e9de3d42f868c0692b244
-  data.tar.gz: 8cd3445f33c9f7db05550947d686293ceee620c1
+SHA256:
+  metadata.gz: 7528791ce5da4ca182e8258cf5bc8920345470ee76ee50de44cf89adac7ffec6
+  data.tar.gz: 3b3dad255ae1841583abb2c61345fbefffb268231906968619000b95005044be
 SHA512:
-  metadata.gz: d9dc7435763425d54d82f4930935c913cd67eb15a1250c58c7ccf3b0e419eeb6ceb87289ed3c22386faa96e3ae3584a9f39cf8e671af8b56b52bb3e2c8257e4d
-  data.tar.gz: 2c5c96993c8a673274a4acc34c3f82a8719fbea508c1dffb5ce8cff1d4a13cd90c9fcbf8a3c743500a31f08419c687c24adf5f2aee464a8a4f1935b2b302b184
+  metadata.gz: 295c2957f708d86b596a4c062fcdf31d9c5083d26d15989de31feb174316ee17430e19d1746ad0f389d599560cc419e740835b1bfba6b0f57627e633f1a0ecf1
+  data.tar.gz: e8bc4ecb3ce277436be91ee4e8cf9c187c1f0bbf5ee170bc7a4e3f221f94d678e0e24d2f7dd878427c7b2b0e0b2fe1baa485556bdf21cd89e05a7ff222a9dc53

data/CHANGELOG.md

@@ -0,0 +1,17 @@
+## 1.1.0
+* Added `Grubby#ok?`.
+* Added `Grubby::PageScraper.scrape_file` and `Grubby::JsonScraper.scrape_file`.
+* Added `Mechanize::Parser#save_to` and `Mechanize::Parser#save_to!`,
+  which are inherited by `Mechanize::Download` and `Mechanize::File`.
+* Added `URI#basename`.
+* Added `URI#query_param`.
+* Added utility methods from [ryoba](https://rubygems.org/gems/ryoba).
+* Added `Grubby::Scraper::Error#scraper` and `Grubby::Scraper#errors`
+  for interactive debugging with e.g. byebug.
+* Improved log messages and error formatting.
+* Fixed compatibility with net-http-persistent gem v3.0.
+
+
+## 1.0.0
+
+* Initial release

data/README.md

@@ -60,6 +60,7 @@ puts hn.items.take(10).map(&:title) # your scraping logic goes here
 
 - [Grubby](http://www.rubydoc.info/gems/grubby/Grubby)
   - [#get_mirrored](http://www.rubydoc.info/gems/grubby/Grubby:get_mirrored)
+  - [#ok?](http://www.rubydoc.info/gems/grubby/Grubby:ok%3F)
   - [#singleton](http://www.rubydoc.info/gems/grubby/Grubby:singleton)
   - [#time_between_requests](http://www.rubydoc.info/gems/grubby/Grubby:time_between_requests)
 - [Scraper](http://www.rubydoc.info/gems/grubby/Grubby/Scraper)
@@ -69,37 +70,89 @@ puts hn.items.take(10).map(&:title) # your scraping logic goes here
   - [#source](http://www.rubydoc.info/gems/grubby/Grubby/Scraper:source)
   - [#to_h](http://www.rubydoc.info/gems/grubby/Grubby/Scraper:to_h)
 - [PageScraper](http://www.rubydoc.info/gems/grubby/Grubby/PageScraper)
+  - [.scrape_file](http://www.rubydoc.info/gems/grubby/Grubby/PageScraper.scrape_file)
   - [#page](http://www.rubydoc.info/gems/grubby/Grubby/PageScraper:page)
 - [JsonScraper](http://www.rubydoc.info/gems/grubby/Grubby/JsonScraper)
+  - [.scrape_file](http://www.rubydoc.info/gems/grubby/Grubby/JsonScraper.scrape_file)
   - [#json](http://www.rubydoc.info/gems/grubby/Grubby/JsonScraper:json)
-- Nokogiri::XML::Searchable
-  - [#at!](http://www.rubydoc.info/gems/grubby/Nokogiri/XML/Searchable:at%21)
-  - [#search!](http://www.rubydoc.info/gems/grubby/Nokogiri/XML/Searchable:search%21)
+- Mechanize::Download
+  - [#save_to](http://www.rubydoc.info/gems/grubby/Mechanize/Parser:save_to)
+  - [#save_to!](http://www.rubydoc.info/gems/grubby/Mechanize/Parser:save_to%21)
+- Mechanize::File
+  - [#save_to](http://www.rubydoc.info/gems/grubby/Mechanize/Parser:save_to)
+  - [#save_to!](http://www.rubydoc.info/gems/grubby/Mechanize/Parser:save_to%21)
 - Mechanize::Page
   - [#at!](http://www.rubydoc.info/gems/grubby/Mechanize/Page:at%21)
   - [#search!](http://www.rubydoc.info/gems/grubby/Mechanize/Page:search%21)
 - Mechanize::Page::Link
   - [#to_absolute_uri](http://www.rubydoc.info/gems/grubby/Mechanize/Page/Link#to_absolute_uri)
+- URI
+  - [#basename](https://www.rubydoc.info/gems/grubby/URI:basename)
+  - [#query_param](https://www.rubydoc.info/gems/grubby/URI:query_param)
 
 
 ## Supplemental API
 
-*grubby* uses several gems which extend core Ruby objects with
-convenience methods.  When you import *grubby* you automatically make
-these methods available.  See each gem below for its specific API
-documentation:
+*grubby* includes several gems which extend Ruby objects with
+convenience methods.  When you load *grubby* you automatically make
+these methods available.  The included gems are listed below, along with
+**a few** of the methods each provides.  See each gem's documentation
+for a complete API listing.
 
 - [Active Support](https://rubygems.org/gems/activesupport)
   ([docs](http://www.rubydoc.info/gems/activesupport/))
+  - [Enumerable#index_by](https://www.rubydoc.info/gems/activesupport/Enumerable:index_by)
+  - [File.atomic_write](https://www.rubydoc.info/gems/activesupport/File:atomic_write)
+  - [NilClass#try](https://www.rubydoc.info/gems/activesupport/NilClass:try)
+  - [Object#presence](https://www.rubydoc.info/gems/activesupport/Object:presence)
+  - [String#blank?](https://www.rubydoc.info/gems/activesupport/String:blank%3F)
+  - [String#squish](https://www.rubydoc.info/gems/activesupport/String:squish)
 - [casual_support](https://rubygems.org/gems/casual_support)
   ([docs](http://www.rubydoc.info/gems/casual_support/))
+  - [Enumerable#index_to](http://www.rubydoc.info/gems/casual_support/Enumerable:index_to)
+  - [String#after](http://www.rubydoc.info/gems/casual_support/String:after)
+  - [String#after_last](http://www.rubydoc.info/gems/casual_support/String:after_last)
+  - [String#before](http://www.rubydoc.info/gems/casual_support/String:before)
+  - [String#before_last](http://www.rubydoc.info/gems/casual_support/String:before_last)
+  - [String#between](http://www.rubydoc.info/gems/casual_support/String:between)
+  - [Time#to_hms](http://www.rubydoc.info/gems/casual_support/Time:to_hms)
+  - [Time#to_ymd](http://www.rubydoc.info/gems/casual_support/Time:to_ymd)
 - [gorge](https://rubygems.org/gems/gorge)
   ([docs](http://www.rubydoc.info/gems/gorge/))
+  - [Pathname#file_crc32](http://www.rubydoc.info/gems/gorge/Pathname:file_crc32)
+  - [Pathname#file_md5](http://www.rubydoc.info/gems/gorge/Pathname:file_md5)
+  - [Pathname#file_sha1](http://www.rubydoc.info/gems/gorge/Pathname:file_sha1)
+  - [String#crc32](http://www.rubydoc.info/gems/gorge/String:crc32)
+  - [String#md5](http://www.rubydoc.info/gems/gorge/String:md5)
+  - [String#sha1](http://www.rubydoc.info/gems/gorge/String:sha1)
 - [mini_sanity](https://rubygems.org/gems/mini_sanity)
   ([docs](http://www.rubydoc.info/gems/mini_sanity/))
+  - [Array#assert_length!](http://www.rubydoc.info/gems/mini_sanity/Array:assert_length%21)
+  - [Enumerable#refute_empty!](http://www.rubydoc.info/gems/mini_sanity/Enumerable:refute_empty%21)
+  - [Object#assert_equal!](http://www.rubydoc.info/gems/mini_sanity/Object:assert_equal%21)
+  - [Object#assert_in!](http://www.rubydoc.info/gems/mini_sanity/Object:assert_in%21)
+  - [Object#refute_nil!](http://www.rubydoc.info/gems/mini_sanity/Object:refute_nil%21)
+  - [Pathname#assert_exist!](http://www.rubydoc.info/gems/mini_sanity/Pathname:assert_exist%21)
+  - [String#assert_match!](http://www.rubydoc.info/gems/mini_sanity/String:assert_match%21)
 - [pleasant_path](https://rubygems.org/gems/pleasant_path)
   ([docs](http://www.rubydoc.info/gems/pleasant_path/))
-
+  - [Pathname#dirs](http://www.rubydoc.info/gems/pleasant_path/Pathname:dirs)
+  - [Pathname#dirs_r](http://www.rubydoc.info/gems/pleasant_path/Pathname:dirs_r)
+  - [Pathname#files](http://www.rubydoc.info/gems/pleasant_path/Pathname:files)
+  - [Pathname#files_r](http://www.rubydoc.info/gems/pleasant_path/Pathname:files_r)
+  - [Pathname#make_dirname](http://www.rubydoc.info/gems/pleasant_path/Pathname:make_dirname)
+  - [Pathname#rename_basename](http://www.rubydoc.info/gems/pleasant_path/Pathname:rename_basename)
+  - [Pathname#rename_extname](http://www.rubydoc.info/gems/pleasant_path/Pathname:rename_extname)
+  - [Pathname#touch_file](http://www.rubydoc.info/gems/pleasant_path/Pathname:touch_file)
+- [ryoba](https://rubygems.org/gems/ryoba)
+  ([docs](http://www.rubydoc.info/gems/ryoba/))
+  - [Nokogiri::XML::Node#matches!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Node:matches%21)
+  - [Nokogiri::XML::Node#text!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Node:text%21)
+  - [Nokogiri::XML::Node#uri](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Node:uri)
+  - [Nokogiri::XML::Searchable#ancestor!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Searchable:ancestor%21)
+  - [Nokogiri::XML::Searchable#ancestors!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Searchable:ancestors%21)
+  - [Nokogiri::XML::Searchable#at!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Searchable:at%21)
+  - [Nokogiri::XML::Searchable#search!](http://www.rubydoc.info/gems/ryoba/Nokogiri/XML/Searchable:search%21)
 
 ## Installation
 

data/Rakefile

@@ -8,9 +8,6 @@ end
 
 desc "Launch IRB with this gem pre-loaded"
 task :irb do
-  # HACK because lib/grubby/version is prematurely loaded by bundler/gem_tasks
-  Object.send(:remove_const, :Grubby)
-
   require "grubby"
   require "irb"
   ARGV.clear

data/grubby.gemspec

@@ -5,7 +5,7 @@ require "grubby/version"
 
 Gem::Specification.new do |spec|
   spec.name          = "grubby"
-  spec.version       = Grubby::VERSION
+  spec.version       = GRUBBY_VERSION
   spec.authors       = ["Jonathan Hefner"]
   spec.email         = ["jonathan.hefner@gmail.com"]
 
@@ -27,6 +27,7 @@ Gem::Specification.new do |spec|
   spec.add_runtime_dependency "mechanize", "~> 2.7"
   spec.add_runtime_dependency "mini_sanity", "~> 1.0"
   spec.add_runtime_dependency "pleasant_path", "~> 1.1"
+  spec.add_runtime_dependency "ryoba", "~> 1.0"
 
   spec.add_development_dependency "bundler", "~> 1.15"
   spec.add_development_dependency "rake", "~> 10.0"

data/lib/grubby.rb

@@ -5,7 +5,9 @@ require "gorge"
 require "mechanize"
 require "mini_sanity"
 require "pleasant_path"
+require "ryoba"
 
+require_relative "grubby/version"
 require_relative "grubby/log"
 
 require_relative "grubby/core_ext/string"
@@ -15,22 +17,30 @@ require_relative "grubby/mechanize/download"
 require_relative "grubby/mechanize/file"
 require_relative "grubby/mechanize/link"
 require_relative "grubby/mechanize/page"
-require_relative "grubby/nokogiri/searchable"
+require_relative "grubby/mechanize/parser"
 
 
 class Grubby < Mechanize
 
+  VERSION = GRUBBY_VERSION
+
+  # The enforced minimum amount of time to wait between requests, in
+  # seconds.  If the value is a Range, a random number within the Range
+  # is chosen for each request.
+  #
   # @return [Integer, Float, Range<Integer>, Range<Float>]
-  #   The enforced minimum amount of time to wait between requests, in
-  #   seconds.  If the value is a Range, a random number within the
-  #   Range is chosen for each request.
   attr_accessor :time_between_requests
 
-  # @param singleton_journal [Pathname, String]
-  #   Optional journal file to persist the list of resources processed
-  #   by {singleton}.  Useful to ensure only-once processing across
-  #   multiple program runs.
-  def initialize(singleton_journal = nil)
+  # Journal file used to ensure only-once processing of resources by
+  # {singleton} across multiple program runs.  Set via {initialize}.
+  #
+  # @return [Pathname, nil]
+  attr_reader :journal
+
+  # @param journal [Pathname, String]
+  #   Optional journal file used to ensure only-once processing of
+  #   resources by {singleton} across multiple program runs.
+  def initialize(journal = nil)
     super()
 
     # Prevent "memory leaks", and prevent mistakenly blank urls from
@@ -58,10 +68,22 @@ class Grubby < Mechanize
     self.pre_connect_hooks << Proc.new{ self.send(:sleep_between_requests) }
     self.time_between_requests = 1.0
 
-    @journal = singleton_journal ?
-      singleton_journal.to_pathname.touch_file : Pathname::NULL
-    @seen = SingletonKey.parse_file(@journal).
-      group_by(&:purpose).transform_values{|sks| sks.map(&:key).index_to{ true } }
+    @journal = journal.try(&:to_pathname).try(&:touch_file)
+    @seen = @journal ? SingletonKey.parse_file(@journal).index_to{ true } : {}
+  end
+
+  # Calls +#head+ and returns true if the result has response code
+  # "200".  Unlike +#head+, error response codes (e.g. "404", "500")
+  # do not cause a +Mechanize::ResponseCodeError+ to be raised.
+  #
+  # @param uri [String]
+  # @return [Boolean]
+  def ok?(uri, query_params = {}, headers = {})
+    begin
+      head(uri, query_params, headers).code == "200"
+    rescue Mechanize::ResponseCodeError => e
+      false
+    end
   end
 
   # Calls +#get+ with each of +mirror_uris+ until a successful
@@ -82,8 +104,8 @@ class Grubby < Mechanize
       if i >= mirror_uris.length
         raise
       else
-        $log.info("Mirror failed with response code #{e.response_code}: #{mirror_uris[i - 1]}")
-        $log.debug("Trying next mirror: #{mirror_uris[i]}")
+        $log.debug("Mirror failed (code #{e.response_code}): #{mirror_uris[i - 1]}")
+        $log.debug("Try mirror: #{mirror_uris[i]}")
         retry
       end
     end
@@ -111,20 +133,20 @@ class Grubby < Mechanize
   def singleton(target, purpose = "")
     series = []
 
-    original_url = target.to_absolute_uri
-    return if skip_singleton?(purpose, original_url.to_s, series)
+    original_uri = target.to_absolute_uri
+    return if try_skip_singleton(original_uri, purpose, series)
 
-    url = normalize_url(original_url)
-    return if skip_singleton?(purpose, url.to_s, series)
+    normalized_uri = normalize_uri(original_uri)
+    return if try_skip_singleton(normalized_uri, purpose, series)
 
-    $log.info("Fetching #{url}")
-    resource = get(url)
-    skip = skip_singleton?(purpose, resource.uri.to_s, series) |
-      skip_singleton?(purpose, "content hash: #{resource.content_hash}", series)
+    $log.info("Fetch #{normalized_uri}")
+    resource = get(normalized_uri)
+    skip = try_skip_singleton(resource.uri, purpose, series) |
+      try_skip_singleton("content hash: #{resource.content_hash}", purpose, series)
 
     yield resource unless skip
 
-    series.map{|k| SingletonKey.new(purpose, k) }.append_to_file(@journal)
+    series.append_to_file(@journal) if @journal
 
     !skip
   end
@@ -132,22 +154,23 @@ class Grubby < Mechanize
 
   private
 
-  SingletonKey = DumbDelimited[:purpose, :key]
+  SingletonKey = DumbDelimited[:purpose, :target]
 
-  def skip_singleton?(purpose, key, series)
-    return false if series.include?(key)
-    series << key
-    already = (@seen[purpose.to_s] ||= {}).displace(key, true)
-    $log.info("Skipping #{series.first} (already seen #{series.last})") if already
-    already
+  def try_skip_singleton(target, purpose, series)
+    series << SingletonKey.new(purpose, target.to_s)
+    if series.uniq!.nil? && @seen.displace(series.last, true)
+      seen_info = series.length > 1 ? "seen #{series.last.target}" : "seen"
+      $log.info("Skip #{series.first.target} (#{seen_info})")
+      true
+    end
   end
 
-  def normalize_url(url)
-    url = url.dup
-    $log.warn("Discarding fragment in URL: #{url}") if url.fragment
-    url.fragment = nil
-    url.path = url.path.chomp("/")
-    url
+  def normalize_uri(uri)
+    uri = uri.dup
+    $log.warn("Ignore ##{uri.fragment} in #{uri}") if uri.fragment
+    uri.fragment = nil
+    uri.path = uri.path.chomp("/")
+    uri
   end
 
   def sleep_between_requests
@@ -162,7 +185,6 @@ class Grubby < Mechanize
 end
 
 
-require_relative "grubby/version"
 require_relative "grubby/json_parser"
 require_relative "grubby/scraper"
 require_relative "grubby/page_scraper"

data/lib/grubby/core_ext/uri.rb

@@ -1,5 +1,45 @@
 module URI
 
+  # Returns the basename of the URI's +path+, a la +File.basename+.
+  #
+  # @example
+  #   URI("http://example.com/foo/bar").basename  # == "bar"
+  #   URI("http://example.com/foo").basename      # == "foo"
+  #   URI("http://example.com/").basename         # == ""
+  #
+  # @return [String]
+  def basename
+    self.path == "/" ? "" : File.basename(self.path)
+  end
+
+  # Returns the value of the specified param in the URI's +query+.
+  # The specified param name must be exactly as it appears in the query
+  # string, and support for complex nested values is limited.  (See
+  # +CGI.parse+ for parsing behavior.)  If the param name includes a
+  # +"[]"+, the result will be an array of all occurrences of that param
+  # in the query string.  Otherwise, the result will be the last
+  # occurrence of that param in the query string.
+  #
+  # @example
+  #   URI("http://example.com/?foo=a").query_param("foo")          # == "a"
+  #
+  #   URI("http://example.com/?foo=a&foo=b").query_param("foo")    # == "b"
+  #   URI("http://example.com/?foo=a&foo=b").query_param("foo[]")  # == nil
+  #
+  #   URI("http://example.com/?foo[]=a&foo[]=b").query_param("foo")    # == nil
+  #   URI("http://example.com/?foo[]=a&foo[]=b").query_param("foo[]")  # == ["a", "b"]
+  #
+  #   URI("http://example.com/?foo[][x]=a&foo[][y]=b").query_param("foo[]")     # == nil
+  #   URI("http://example.com/?foo[][x]=a&foo[][y]=b").query_param("foo[][x]")  # == ["a"]
+  #
+  # @return [String, nil]
+  # @return [Array<String>, nil]
+  #   if +name+ contains +"[]"+
+  def query_param(name)
+    values = CGI.parse(self.query)[name.to_s]
+    (values.nil? || name.include?("[]")) ? values : values.last
+  end
+
   # Raises an exception if the URI is not +absolute?+.
   #
   # @return [self]

data/lib/grubby/json_parser.rb

@@ -33,8 +33,9 @@ class Grubby::JsonParser < Mechanize::File
     @json_parse_options = options
   end
 
+  # The parsed JSON data.
+  #
   # @return [Hash, Array]
-  #   The parsed JSON data.
   attr_reader :json
 
   def initialize(uri = nil, response = nil, body = nil, code = nil)

data/lib/grubby/json_scraper.rb

@@ -1,7 +1,8 @@
 class Grubby::JsonScraper < Grubby::Scraper
 
+  # The parsed JSON data being scraped.
+  #
   # @return [Hash, Array]
-  #   The parsed JSON data being scraped.
   attr_reader :json
 
   # @param source [Grubby::JsonParser]
@@ -10,4 +11,22 @@ class Grubby::JsonScraper < Grubby::Scraper
     super
   end
 
+  # Scrapes a locally-stored file.  This method is intended for use with
+  # subclasses of +Grubby::JsonScraper+.
+  #
+  # @example
+  #   class MyScraper < Grubby::JsonScraper
+  #     # ...
+  #   end
+  #
+  #   MyScraper.scrape_file("path/to/local_file.json").class  # == MyScraper
+  #
+  # @param path [String]
+  # @return [Grubby::JsonScraper]
+  def self.scrape_file(path)
+    uri = URI.join("file:///", File.expand_path(path))
+    body = File.read(path)
+    self.new(Grubby::JsonParser.new(uri, nil, body, "200"))
+  end
+
 end

data/lib/grubby/mechanize/fetch_with_retry.rb

@@ -9,9 +9,8 @@ class Mechanize::HTTP::Agent
   IDEMPOTENT_HTTP_METHODS = [:get, :head, :options, :delete]
 
   # Replacement for +Mechanize::HTTP::Agent#fetch+.  When a "too many
-  # connection resets" error is encountered, this method shuts down the
-  # persistent HTTP connection, and then retries the request (upto
-  # {MAX_CONNECTION_RESET_RETRIES} times).
+  # connection resets" error is encountered, this method retries the
+  # request (upto {MAX_CONNECTION_RESET_RETRIES} times).
   def fetch_with_retry(uri, http_method = :get, headers = {}, params = [], referer = current_page, redirects = 0)
     retry_count = 0
     begin
@@ -26,9 +25,9 @@ class Mechanize::HTTP::Agent
 
       # otherwise, shutdown the persistent HTTP connection and try again
       retry_count += 1
-      $log.warn("Possible connection reset bug.  Retry(#{retry_count}) #{http_method.to_s.upcase} #{uri}")
-      self.http.shutdown
-      sleep(retry_count) # incremental backoff in case problem is with server
+      $log.warn("#{e.message} (#{e.class}).  Retry in #{retry_count} seconds.")
+      sleep(retry_count) # incremental backoff to allow server to self-correct
+      $log.warn("Retry #{http_method.to_s.upcase} #{uri}")
       retry
     end
   end

data/lib/grubby/mechanize/parser.rb

@@ -0,0 +1,46 @@
+require "fileutils"
+
+module Mechanize::Parser
+
+  # Saves the payload to a specified directory, but using the default
+  # filename suggested by the server.  If a file with that name already
+  # exists, this method will try to find a free filename by appending
+  # numbers to the original name.  Returns the full path of the saved
+  # file.
+  #
+  # NOTE: this method expects a +#save!+ method to be defined by the
+  # class extending +Mechanize::Parser+, e.g. +Mechanize::File#save!+
+  # and +Mechanize::Download#save!+.
+  #
+  # @param directory [String]
+  # @return [String]
+  def save_to(directory)
+    raise "#{self.class}#save! is not defined" unless self.respond_to?(:save!)
+
+    FileUtils.mkdir_p(directory)
+    path = find_free_name(File.join(directory, @filename))
+    save!(path)
+    path
+  end
+
+  # Saves the payload to a specified directory, but using the default
+  # filename suggested by the server.  If a file with that name already
+  # exists, that file will be overwritten.  Returns the full path of the
+  # saved file.
+  #
+  # NOTE: this method expects a +#save!+ method to be defined by the
+  # class extending +Mechanize::Parser+, e.g. +Mechanize::File#save!+
+  # and +Mechanize::Download#save!+.
+  #
+  # @param directory [String]
+  # @return [String]
+  def save_to!(directory)
+    raise "#{self.class}#save! is not defined" unless self.respond_to?(:save!)
+
+    FileUtils.mkdir_p(directory)
+    path = File.join(directory, @filename)
+    save!(path)
+    path
+  end
+
+end

data/lib/grubby/page_scraper.rb

@@ -1,7 +1,8 @@
 class Grubby::PageScraper < Grubby::Scraper
 
+  # The Page being scraped.
+  #
   # @return [Mechanize::Page]
-  #   The Page being scraped.
   attr_reader :page
 
   # @param source [Mechanize::Page]
@@ -10,4 +11,23 @@ class Grubby::PageScraper < Grubby::Scraper
     super
   end
 
+  # Scrapes a locally-stored file.  This method is intended for use with
+  # subclasses of +Grubby::PageScraper+.
+  #
+  # @example
+  #   class MyScraper < Grubby::PageScraper
+  #     # ...
+  #   end
+  #
+  #   MyScraper.scrape_file("path/to/local_file.html").class  # == MyScraper
+  #
+  # @param path [String]
+  # @param agent [Mechanize]
+  # @return [Grubby::PageScraper]
+  def self.scrape_file(path, agent = Grubby.new)
+    uri = URI.join("file:///", File.expand_path(path))
+    body = File.read(path)
+    self.new(Mechanize::Page.new(uri, nil, body, "200", agent))
+  end
+
 end

data/lib/grubby/scraper.rb

@@ -1,8 +1,5 @@
 class Grubby::Scraper
 
-  class Error < RuntimeError
-  end
-
   # Defines an attribute reader method named by +field+.  During
   # +initialize+, the given block is called, and the attribute is set to
   # the block's return value.  By default, if the block's return value
@@ -22,38 +19,48 @@ class Grubby::Scraper
     self.fields << field
 
     define_method(field) do
+      raise "#{self.class}#initialize does not invoke `super`" unless defined?(@scraped)
       return @scraped[field] if @scraped.key?(field)
 
-      unless @errors.key?(field)
+      unless @errors[field]
         begin
           value = instance_eval(&block)
           if value.nil?
-            raise "`#{field}` cannot be nil" unless optional
-            $log.debug("Scraped nil value for #{self.class}##{field}")
+            raise FieldValueRequiredError.new(field) unless optional
+            $log.debug("#{self.class}##{field} is nil")
           end
           @scraped[field] = value
-        rescue RuntimeError => e
+        rescue RuntimeError, IndexError => e
           @errors[field] = e
         end
       end
 
-      raise "`#{field}` raised a #{@errors[field].class}" if @errors.key?(field)
+      raise FieldScrapeFailedError.new(field, @errors[field]) if @errors[field]
 
       @scraped[field]
     end
   end
 
+  # The names of all scraped values, as defined by {scrapes}.
+  #
   # @return [Array<Symbol>]
-  #   The names of all scraped values, as defined by {scrapes}.
   def self.fields
     @fields ||= []
   end
 
+  # The source being scraped.  Typically a Mechanize pluggable parser
+  # such as +Mechanize::Page+.
+  #
   # @return [Object]
-  #   The source being scraped.  Typically a Mechanize pluggable parser
-  #   such as +Mechanize::Page+.
   attr_reader :source
 
+  # Hash of errors raised by blocks passed to {scrapes}.  If
+  # {initialize} does not raise +Grubby::Scraper::Error+, this Hash will
+  # be empty.
+  #
+  # @return [Hash<Symbol, StandardError>]
+  attr_reader :errors
+
   # @param source
   # @raise [Grubby::Scraper::Error]
   #   if any scraped values result in error
@@ -65,18 +72,11 @@ class Grubby::Scraper
     self.class.fields.each do |field|
       begin
         self.send(field)
-      rescue RuntimeError
+      rescue FieldScrapeFailedError
       end
     end
 
-    unless @errors.empty?
-      listing = @errors.map do |field, error|
-        error_class = " (#{error.class})" unless error.class == RuntimeError
-        error_trace = error.backtrace.join("\n").indent(2)
-        "* #{field} -- #{error.message}#{error_class}\n#{error_trace}"
-      end
-      raise Error.new("Failed to scrape the following fields:\n#{listing.join("\n")}")
-    end
+    raise Error.new(self) unless @errors.empty?
   end
 
   # Returns the scraped value named by +field+.
@@ -96,4 +96,43 @@ class Grubby::Scraper
     @scraped.dup
   end
 
+  class Error < RuntimeError
+    BACKTRACE_CLEANER = ActiveSupport::BacktraceCleaner.new.tap do |cleaner|
+      cleaner.add_silencer do |line|
+        line.include?(__dir__) && line.include?("scraper.rb:")
+      end
+    end
+
+    # @return [Grubby::Scraper]
+    #   The Scraper that raised this error.
+    attr_accessor :scraper
+
+    def initialize(scraper)
+      self.scraper = scraper
+
+      listing = scraper.errors.
+        reject{|field, error| error.is_a?(FieldScrapeFailedError) }.
+        map do |field, error|
+          "* `#{field}` (#{error.class})\n" +
+            error.message.indent(2) + "\n\n" +
+            BACKTRACE_CLEANER.clean(error.backtrace).join("\n").indent(4) + "\n"
+        end.
+        join("\n")
+
+      super("Failed to scrape the following fields:\n#{listing}")
+    end
+  end
+
+  class FieldScrapeFailedError < RuntimeError
+    def initialize(field, field_error)
+      super("`#{field}` raised #{field_error.class}")
+    end
+  end
+
+  class FieldValueRequiredError < RuntimeError
+    def initialize(field)
+      super("`#{field}` is nil but is not marked as optional")
+    end
+  end
+
 end

data/lib/grubby/version.rb

@@ -1,3 +1 @@
-class Grubby
-  VERSION = "1.0.0"
-end
+GRUBBY_VERSION = "1.1.0"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: grubby
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Jonathan Hefner
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-09-05 00:00:00.000000000 Z
+date: 2018-07-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -108,6 +108,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: ryoba
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -173,6 +187,7 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - ".travis.yml"
+- CHANGELOG.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -189,7 +204,7 @@ files:
 - lib/grubby/mechanize/file.rb
 - lib/grubby/mechanize/link.rb
 - lib/grubby/mechanize/page.rb
-- lib/grubby/nokogiri/searchable.rb
+- lib/grubby/mechanize/parser.rb
 - lib/grubby/page_scraper.rb
 - lib/grubby/scraper.rb
 - lib/grubby/version.rb
@@ -213,7 +228,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 2.7.6
 signing_key: 
 specification_version: 4
 summary: Fail-fast web scraping

data/lib/grubby/nokogiri/searchable.rb

@@ -1,27 +0,0 @@
-module Nokogiri::XML::Searchable
-
-  # Searches the node using the given XPath or CSS queries, and returns
-  # the results.  Raises an exception if there are no results.  See also
-  # +#search+.
-  #
-  # @param queries [Array<String>]
-  # @return [Array<Nokogiri::XML::Element>]
-  # @raise [RuntimeError] if queries yield no results
-  def search!(*queries)
-    results = search(*queries)
-    raise "No elements matching #{queries.map(&:inspect).join(" OR ")}" if results.empty?
-    results
-  end
-
-  # Searches the node using the given XPath or CSS queries, and returns
-  # only the first result.  Raises an exception if there are no results.
-  # See also +#at+.
-  #
-  # @param queries [Array<String>]
-  # @return [Nokogiri::XML::Element]
-  # @raise [RuntimeError] if queries yield no results
-  def at!(*queries)
-    search!(*queries).first
-  end
-
-end