pupa 0.0.7 → 0.0.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c06be9f93589b34b71f5b7037ab75cecbfaa43da
-  data.tar.gz: 44b4ab787cc85352b84d6edf8d502adc8a7d6b79
+  metadata.gz: d4ec7210671485a2de58673a70088e415a9767b7
+  data.tar.gz: 8b2a77e3fe3c5775509fef59fb847ea2057f63cb
 SHA512:
-  metadata.gz: 5d605b3a6e98b3bdf94161e88904653b7e6facdf25ffff29d0d8d5170c4066196772d2c7d9c2736d2789325a66715423078a1b3761a7a41b065f366eba379735
-  data.tar.gz: db7166154329a552a9a8ed9d8c1486077536590cff961173f64a595513d4cdfb522db47c11a56d9d5c9faa65a9fe51d745defb4e7d0033f84484b5a8d256bf53
+  metadata.gz: b3cdcf2da535ebd8d840fe2a1f6e6dd0db68de4d92b728f6f33bf7ba80f7499331f6cc9ace81da93b7e1062e7f3b8389fbab1891d81fbc44d1c48cbbc3be8eea
+  data.tar.gz: 641956572610184f0437f0869e2ee9da3c624e493284ac3d4e0e88157395c4b8d41840ac4f1d351b63032aff1fde091862a2b83401c5ee55c1d83ac52a856a70

data/.travis.yml

@@ -3,3 +3,4 @@ rvm:
   - 2.0.0
 services:
   - mongodb
+  - redis

data/README.md

@@ -45,6 +45,56 @@ The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.htm
 
 1.  You may want more control over the method used to perform a scraping task. For example, a legislature may publish legislators before 1997 in one format and legislators after 1997 in another format. In this case, you may want to select the method used to scrape legislators according to the year. See [legislator.rb](http://opennorth.github.io/pupa-ruby/docs/legislator.html).
 
+## Performance
+
+Pupa.rb offers several ways to significantly improve performance.
+
+In an example case, reducing file I/O and skipping validation as described below reduced the time to scrape 10,000 documents from 100 cached HTTP responses from 100 seconds down to 5 seconds. Like fast tests, fast scrapers make development smoother.
+
+The `import` action's performance (when using a dependency graph) is currently limited by MongoDB.
+
+### Caching HTTP requests
+
+HTTP requests consume the most time. To avoid repeat HTTP requests while developing a scraper, cache all HTTP responses. Pupa.rb will by default use a `web_cache` directory in the same directory as your script. You can change the directory by setting the `--cache_dir` switch on the command line, for example:
+
+    ruby cat.rb --cache_dir my_cache_dir
+
+### Reducing file I/O
+
+After HTTP requests, file I/O is the slowest operation. Two types of files are written to disk: HTTP responses are written to the cache directory, and JSON documents are written to the output directory. Writing to memory is much faster than writing to disk. You may store HTTP responses in [Memcached](http://memcached.org/) like so:
+
+    ruby cat.rb --cache_dir memcached://localhost:11211
+
+And you may store JSON documents in [Redis](http://redis.io/) like so:
+
+    ruby cat.rb --output_dir redis://localhost:6379/0
+
+Note that Pupa.rb flushes the JSON documents before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa.rb and other applications. You can select a different database than the default `0` for use with Pupa.rb by passing an argument like `redis://localhost:6379/1`, where `1` is the Redis database number.
+
+### Skipping validation
+
+The `json-schema` gem is slow compared to, for example, [JSV](https://github.com/garycourt/JSV). Setting the `--no-validate` switch and running JSON Schema validations separately can further reduce a scraper's running time.
+
+### Profiling
+
+You can profile your code using [perftools.rb](https://github.com/tmm1/perftools.rb). First, install the gem:
+
+    gem install perftools.rb
+
+Then, run your script with the profiler (changing `/tmp/PROFILE_NAME` and `script.rb` as appropriate):
+
+    CPUPROFILE=/tmp/PROFILE_NAME RUBYOPT="-r`gem which perftools | tail -1`" ruby script.rb
+
+You may want to set the `CPUPROFILE_REALTIME=1` flag; however, it seems to change the behavior of the `json-schema` gem, for whatever reason.
+
+[perftools.rb](https://github.com/tmm1/perftools.rb) has several output formats. If your code is straight-forward, you can draw a graph (changing `/tmp/PROFILE_NAME` and `/tmp/PROFILE_NAME.pdf` as appropriate):
+
+    pprof.rb --pdf /tmp/PROFILE_NAME > /tmp/PROFILE_NAME.pdf
+
+## Testing
+
+**DO NOT** run this gem's specs if you are using Redis database number 15 on `localhost`!
+
 ## Bugs? Questions?
 
 This project's main repository is on GitHub: [http://github.com/opennorth/pupa-ruby](http://github.com/opennorth/pupa-ruby), where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.

data/lib/pupa.rb

@@ -1,3 +1,4 @@
+require 'fileutils'
 require 'forwardable'
 
 require 'active_support/concern'

data/lib/pupa/models/base.rb

@@ -84,9 +84,9 @@ module Pupa
         self.json_schema = if Hash === value
           value
         elsif Pathname.new(value).absolute?
-          value
+          File.read(value)
         else
-          File.expand_path(File.join('..', '..', '..', 'schemas', "#{value}.json"), __dir__)
+          File.read(File.expand_path(File.join('..', '..', '..', 'schemas', "#{value}.json"), __dir__))
         end
       end
     end
@@ -164,7 +164,7 @@ module Pupa
     # @raises [JSON::Schema::ValidationError] if the object is invalid
     def validate!
       if self.class.json_schema
-        # JSON::Validator#initialize_data runs fastest if given a hash.
+        # JSON::Validator#initialize_schema runs fastest if given a hash.
         JSON::Validator.validate!(self.class.json_schema, stringify_keys(to_h))
       end
     end

data/lib/pupa/processor.rb

@@ -6,8 +6,12 @@ require 'pupa/processor/client'
 require 'pupa/processor/dependency_graph'
 require 'pupa/processor/helper'
 require 'pupa/processor/persistence'
+require 'pupa/processor/document_store'
 require 'pupa/processor/yielder'
 
+require 'pupa/processor/document_store/file_store'
+require 'pupa/processor/document_store/redis_store'
+
 module Pupa
   # An abstract processor class from which specific processors inherit.
   class Processor
@@ -17,23 +21,26 @@ module Pupa
     class_attribute :tasks
     self.tasks = []
 
-    attr_reader :report, :client, :options
+    attr_reader :report, :store, :client, :options
 
     def_delegators :@logger, :debug, :info, :warn, :error, :fatal
 
-    # @param [String] output_dir the directory in which to dump JSON documents
-    # @param [String] cache_dir the directory in which to cache HTTP responses
+    # @param [String] output_dir the directory or Redis address
+    #   (e.g. `redis://localhost:6379`) in which to dump JSON documents
+    # @param [String] cache_dir the directory or Memcached address
+    #   (e.g. `memcached://localhost:11211`) in which to cache HTTP responses
     # @param [Integer] expires_in the cache's expiration time in seconds
+    # @param [Boolean] validate whether to validate JSON documents
     # @param [String] level the log level
     # @param [String,IO] logdev the log device
     # @param [Hash] options criteria for selecting the methods to run
-    def initialize(output_dir, cache_dir: nil, expires_in: 86400, level: 'INFO', logdev: STDOUT, options: {})
-      @output_dir = output_dir
-      @options    = options
-      @level      = level
-      @logger     = Logger.new('pupa', level: level, logdev: logdev)
-      @client     = Client.new(cache_dir: cache_dir, expires_in: expires_in, level: level)
-      @report     = {}
+    def initialize(output_dir, cache_dir: nil, expires_in: 86400, validate: true, level: 'INFO', logdev: STDOUT, options: {})
+      @store    = DocumentStore.new(output_dir)
+      @client   = Client.new(cache_dir: cache_dir, expires_in: expires_in, level: level)
+      @logger   = Logger.new('pupa', level: level, logdev: logdev)
+      @validate = validate
+      @options  = options
+      @report   = {}
     end
 
     # Retrieves and parses a document with a GET request.
@@ -213,23 +220,22 @@ module Pupa
     # @raises [Pupa::Errors::DuplicateObjectIdError]
     def dump_scraped_object(object)
       type = object.class.to_s.demodulize.underscore
-      basename = "#{type}_#{object._id.gsub(File::SEPARATOR, '_')}.json"
-      path = File.join(@output_dir, basename)
+      name = "#{type}_#{object._id.gsub(File::SEPARATOR, '_')}.json"
 
-      if File.exist?(path)
+      if @store.exist?(name)
         raise Errors::DuplicateObjectIdError, "duplicate object ID: #{object._id} (was the same objected yielded twice?)"
       end
 
-      info {"save #{type} #{object.to_s} as #{basename}"}
+      info {"save #{type} #{object.to_s} as #{name}"}
 
-      File.open(path, 'w') do |f|
-        f.write(JSON.dump(object.to_h(include_foreign_objects: true)))
-      end
+      @store.write(name, object.to_h(include_foreign_objects: true))
 
-      begin
-        object.validate!
-      rescue JSON::Schema::ValidationError => e
-        warn {e.message}
+      if @validate
+        begin
+          object.validate!
+        rescue JSON::Schema::ValidationError => e
+          warn {e.message}
+        end
       end
     end
 
@@ -238,8 +244,7 @@ module Pupa
     # @return [Hash] a hash of scraped objects keyed by ID
     def load_scraped_objects
       {}.tap do |objects|
-        Dir[File.join(@output_dir, '*.json')].each do |path|
-          data = JSON.load(File.read(path))
+        @store.read_multi(@store.entries).each do |data|
           object = data['_type'].camelize.constantize.new(data)
           objects[object._id] = object
         end
@@ -276,16 +281,15 @@ module Pupa
     # @param [Hash] objects a hash of scraped objects keyed by ID
     # @return [Hash] a mapping from an object ID to the ID of its duplicate
     def build_losers_to_winners_map(objects)
+      inverse = {}
+      objects.each do |id,object|
+        (inverse[object.to_h.except(:_id)] ||= []) << id
+      end
+
       {}.tap do |map|
-        # We don't need to iterate on the last item in the hash, but skipping
-        # the last item is more effort than running the last item.
-        objects.each_with_index do |(id1,object1),index|
-          unless map.key?(id1) # Don't search for duplicates of duplicates.
-            objects.drop(index + 1).each do |id2,object2|
-              if object1 == object2
-                map[id2] = id1
-              end
-            end
+        inverse.values.each do |ids|
+          ids.drop(1).each do |id|
+            map[id] = ids[0]
           end
         end
       end

data/lib/pupa/processor/client.rb

@@ -18,7 +18,10 @@ module Pupa
     class Client
       # Returns a configured Faraday HTTP client.
       #
-      # @param [String] cache_dir a directory in which to cache requests
+      # In order to automatically parse XML responses, you must `require 'multi_xml'`.
+      #
+      # @param [String] cache_dir a directory or a Memcached address
+      #   (e.g. `memcached://localhost:11211`) in which to cache requests
       # @param [Integer] expires_in the cache's expiration time in seconds
       # @param [String] level the log level
       # @return [Faraday::Connection] a configured Faraday HTTP client
@@ -26,20 +29,30 @@ module Pupa
         Faraday.new do |connection|
           connection.request :url_encoded
           connection.use Middleware::Logger, Logger.new('faraday', level: level)
+
           # @see http://tools.ietf.org/html/rfc2854
           # @see http://tools.ietf.org/html/rfc3236
           connection.use Middleware::ParseHtml, content_type: %w(text/html application/xhtml+xml)
+
           # @see http://tools.ietf.org/html/rfc4627
           connection.use FaradayMiddleware::ParseJson, content_type: /\bjson$/
-          # @see http://tools.ietf.org/html/rfc3023
+
           if defined?(MultiXml)
+            # @see http://tools.ietf.org/html/rfc3023
             connection.use FaradayMiddleware::ParseXml, content_type: /\bxml$/
           end
+
           if cache_dir
             connection.response :caching do
-              ActiveSupport::Cache::FileStore.new(cache_dir, expires_in: expires_in)
+              address = cache_dir[%r{\Amemcached://(.+)\z}, 1]
+              if address
+                ActiveSupport::Cache::MemCacheStore.new(address, expires_in: expires_in)
+              else
+                ActiveSupport::Cache::FileStore.new(cache_dir, expires_in: expires_in)
+              end
             end
           end
+
           connection.adapter Faraday.default_adapter # must be last
         end
       end

data/lib/pupa/processor/document_store.rb

@@ -0,0 +1,21 @@
+module Pupa
+  class Processor
+    # An JSON document store factory.
+    #
+    # Heavily inspired by `ActiveSupport::Cache::Store`.
+    class DocumentStore
+      # Returns a configured JSON document store.
+      #
+      # @param [String] argument the filesystem directory or Redis address
+      #   (e.g. `redis://localhost:6379/0`) in which to dump JSON documents
+      # @return a configured JSON document store
+      def self.new(argument)
+        if argument[%r{\Aredis://}]
+          RedisStore.new(argument)
+        else
+          FileStore.new(argument)
+        end
+      end
+    end
+  end
+end

data/lib/pupa/processor/document_store/file_store.rb

@@ -0,0 +1,83 @@
+module Pupa
+  class Processor
+    class DocumentStore
+      # Stores JSON documents on disk.
+      #
+      # @see ActiveSupport::Cache::FileStore
+      class FileStore
+        # @param [String] output_dir the directory in which to dump JSON documents
+        def initialize(output_dir)
+          @output_dir = output_dir
+          FileUtils.mkdir_p(@output_dir)
+        end
+
+        # Returns whether a file with the given name exists.
+        #
+        # @param [String] name a key
+        # @return [Boolean] whether the store contains an entry for the given key
+        def exist?(name)
+          File.exist?(namespaced_key(name))
+        end
+
+        # Returns all file names in the storage directory.
+        #
+        # @return [Array<String>] all keys in the store
+        def entries
+          Dir.chdir(@output_dir) do
+            Dir['*.json']
+          end
+        end
+
+        # Returns, as JSON, the contents of the file with the given name.
+        #
+        # @param [String] name a key
+        # @return [Hash] the value of the given key
+        def read(name)
+          File.open(namespaced_key(name)) do |f|
+            JSON.load(f)
+          end
+        end
+
+        # Returns, as JSON, the contents of the files with the given names.
+        #
+        # @param [String] names keys
+        # @return [Array<Hash>] the values of the given keys
+        def read_multi(names)
+          names.map do |name|
+            read(name)
+          end
+        end
+
+        # Writes, as JSON, the value to a file with the given name.
+        #
+        # @param [String] name a key
+        # @param [Hash] value a value
+        def write(name, value)
+          File.open(namespaced_key(name), 'w') do |f|
+            JSON.dump(value, f)
+          end
+        end
+
+        # Delete a file with the given name.
+        #
+        # @param [String] name a key
+        def delete(name)
+          File.delete(namespaced_key(name))
+        end
+
+        # Deletes all files in the storage directory.
+        def clear
+          Dir[File.join(@output_dir, '*.json')].each do |path|
+            File.delete(path)
+          end
+        end
+
+      private
+
+        def namespaced_key(name)
+          File.join(@output_dir, name)
+        end
+      end
+    end
+  end
+end

data/lib/pupa/processor/document_store/redis_store.rb

@@ -0,0 +1,77 @@
+module Pupa
+  class Processor
+    class DocumentStore
+      # Stores JSON documents in Redis.
+      #
+      # Pupa flushes the JSON document store before scraping. If you use Redis,
+      # **DO NOT** share a Redis database with Pupa and other applications. You
+      # can select a different database than the default `0` for use with Pupa
+      # by passing an argument like `redis://localhost:6379/0`.
+      #
+      # @note Redis support depends on the `redis` gem. For better performance,
+      #   use the `hiredis` gem as well.
+      class RedisStore
+        # @param [String] address the address (e.g. `redis://localhost:6379/0`)
+        #   in which to dump JSON documents
+        def initialize(address)
+          options = {}
+          if defined?(Hiredis)
+            options.update(driver: :hiredis)
+          end
+          @redis = Redis::Store::Factory.create(address, options)
+        end
+
+        # Returns whether database contains an entry for the given key.
+        #
+        # @param [String] name a key
+        # @return [Boolean] whether the store contains an entry for the given key
+        def exist?(name)
+          @redis.exists(name)
+        end
+
+        # Returns all keys in the database.
+        #
+        # @return [Array<String>] all keys in the store
+        def entries
+          @redis.keys('*')
+        end
+
+        # Returns, as JSON, the value of the given key.
+        #
+        # @param [String] name a key
+        # @return [Hash] the value of the given key
+        def read(name)
+          JSON.load(@redis.get(name))
+        end
+
+        # Returns, as JSON, the values of the given keys.
+        #
+        # @param [String] names keys
+        # @return [Array<Hash>] the values of the given keys
+        def read_multi(names)
+          @redis.mget(*names).map{|value| JSON.load(value)}
+        end
+
+        # Writes, as JSON, the value to a key.
+        #
+        # @param [String] name a key
+        # @param [Hash] value a value
+        def write(name, value)
+          @redis.set(name, JSON.dump(value))
+        end
+
+        # Delete a key.
+        #
+        # @param [String] name a key
+        def delete(name)
+          @redis.del(name)
+        end
+
+        # Deletes all keys in the database.
+        def clear
+          @redis.flushdb
+        end
+      end
+    end
+  end
+end

data/lib/pupa/refinements/faraday_middleware.rb

@@ -1,6 +1,4 @@
-# A refinement for the Faraday caching middleware to cache all requests, not
-# only GET requests. Using Ruby's refinements doesn't seem to work, possibly
-# because Faraday caches middlewares.
+# Caches all requests, not only GET requests.
 class FaradayMiddleware::Caching
   def call(env)
     # Remove if-statement to cache any request, not only GET.

data/lib/pupa/refinements/json-schema.rb

@@ -1,9 +1,9 @@
 module Pupa
-  class Refinements
+  module Refinements
     # A refinement for JSON Schema to validate "email" and "uri" formats. Using
     # Ruby's refinements doesn't seem to work, possibly because `refine` can't
     # be used with `prepend`.
-    module Format
+    module FormatAttribute
       # @see http://my.rails-royce.org/2010/07/21/email-validation-in-ruby-on-rails-without-regexp/
       def validate(current_schema, data, fragments, processor, validator, options = {})
         case current_schema.schema['format']
@@ -33,6 +33,6 @@ end
 
 class JSON::Schema::FormatAttribute
   class << self
-    prepend Pupa::Refinements::Format
+    prepend Pupa::Refinements::FormatAttribute
   end
 end

data/lib/pupa/runner.rb

@@ -1,4 +1,3 @@
-require 'fileutils'
 require 'optparse'
 require 'ostruct'
 
@@ -19,6 +18,7 @@ module Pupa
         output_dir:     File.expand_path('scraped_data', Dir.pwd),
         cache_dir:      File.expand_path('web_cache', Dir.pwd),
         expires_in:     86400, # 1 day
+        validate:       true,
         host_with_port: 'localhost:27017',
         database:       'pupa',
         dry_run:        false,
@@ -72,15 +72,18 @@ module Pupa
         opts.on('-t', '--task TASK', @processor_class.tasks, 'Select a scraping task to run (you may give this switch multiple times)', "  (#{@processor_class.tasks.join(', ')})") do |v|
           options.tasks << v
         end
-        opts.on('-o', '--output_dir PATH', 'The directory in which to dump JSON documents') do |v|
+        opts.on('-o', '--output_dir PATH', 'The directory or Redis address (e.g. redis://localhost:6379) in which to dump JSON documents') do |v|
           options.output_dir = v
         end
-        opts.on('-c', '--cache_dir PATH', 'The directory in which to cache HTTP requests') do |v|
+        opts.on('-c', '--cache_dir PATH', 'The directory or Memcached address (e.g. memcached://localhost:11211) in which to cache HTTP requests') do |v|
           options.cache_dir = v
         end
         opts.on('-e', '--expires_in SECONDS', "The cache's expiration time in seconds") do |v|
           options.expires_in = v
         end
+        opts.on('--[no-]validate', 'Validate JSON documents') do |v|
+          options.validate = v
+        end
         opts.on('-H', '--host HOST:PORT', 'The host and port to MongoDB') do |v|
           options.host_with_port = v
         end
@@ -137,7 +140,12 @@ module Pupa
         options.tasks = @processor_class.tasks
       end
 
-      processor = @processor_class.new(options.output_dir, cache_dir: options.cache_dir, expires_in: options.expires_in, level: options.level, options: Hash[*rest])
+      processor = @processor_class.new(options.output_dir,
+        cache_dir: options.cache_dir,
+        expires_in: options.expires_in,
+        validate: options.validate,
+        level: options.level,
+        options: Hash[*rest])
 
       options.actions.each do |action|
         unless action == 'scrape' || processor.respond_to?(action)
@@ -174,13 +182,7 @@ module Pupa
       Pupa.session = Moped::Session.new([options.host_with_port], database: options.database)
 
       if options.actions.delete('scrape')
-        FileUtils.mkdir_p(options.output_dir)
-        FileUtils.mkdir_p(options.cache_dir)
-
-        Dir[File.join(options.output_dir, '*.json')].each do |path|
-          FileUtils.rm(path)
-        end
-
+        processor.store.clear
         report[:scrape] = {}
         options.tasks.each do |task_name|
           report[:scrape][task_name] = processor.dump_scraped_objects(task_name)

data/lib/pupa/version.rb

@@ -1,3 +1,3 @@
 module Pupa
-  VERSION = "0.0.7"
+  VERSION = "0.0.8"
 end

data/pupa.gemspec

@@ -25,10 +25,12 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency('nokogiri', '~> 1.6.0')
 
   s.add_development_dependency('coveralls')
+  s.add_development_dependency('dalli')
   s.add_development_dependency('json', '~> 1.7.7') # to silence coveralls warning
+  s.add_development_dependency('multi_xml')
   s.add_development_dependency('octokit') # to update Popolo schema
   s.add_development_dependency('rake')
+  s.add_development_dependency('redis-store')
   s.add_development_dependency('rspec', '~> 2.10')
   s.add_development_dependency('vcr', '~> 2.5.0')
-  s.add_development_dependency('multi_xml')
 end

data/spec/cassettes/f861172f1df3bdb2052af5451f9922699d574b77.yml

@@ -0,0 +1,62 @@
+---
+http_interactions:
+- request:
+    method: get
+    uri: http://example.com/
+    body:
+      encoding: US-ASCII
+      string: ''
+    headers:
+      User-Agent:
+      - Faraday v0.8.8
+  response:
+    status:
+      code: 200
+      message: 
+    headers:
+      accept-ranges:
+      - bytes
+      cache-control:
+      - max-age=604800
+      content-type:
+      - text/html
+      date:
+      - Fri, 27 Sep 2013 00:31:23 GMT
+      etag:
+      - '"3012602696"'
+      expires:
+      - Fri, 04 Oct 2013 00:31:23 GMT
+      last-modified:
+      - Fri, 09 Aug 2013 23:54:35 GMT
+      server:
+      - ECS (mdw/13C6)
+      x-cache:
+      - HIT
+      x-ec-custom-error:
+      - '1'
+      content-length:
+      - '1270'
+      connection:
+      - close
+    body:
+      encoding: UTF-8
+      string: "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n
+        \   <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html;
+        charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width,
+        initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color:
+        #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: \"Open
+        Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n
+        \   div {\n        width: 600px;\n        margin: 5em auto;\n        padding:
+        50px;\n        background-color: #fff;\n        border-radius: 1em;\n    }\n
+        \   a:link, a:visited {\n        color: #38488f;\n        text-decoration:
+        none;\n    }\n    @media (max-width: 700px) {\n        body {\n            background-color:
+        #fff;\n        }\n        div {\n            width: auto;\n            margin:
+        0 auto;\n            border-radius: 0;\n            padding: 1em;\n        }\n
+        \   }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n
+        \   <p>This domain is established to be used for illustrative examples in
+        documents. You may use this\n    domain in examples without prior coordination
+        or asking for permission.</p>\n    <p><a href=\"http://www.iana.org/domains/example\">More
+        information...</a></p>\n</div>\n</body>\n</html>\n"
+    http_version: 
+  recorded_at: Fri, 27 Sep 2013 00:31:23 GMT
+recorded_with: VCR 2.5.0

data/spec/fixtures/bar.json

@@ -0,0 +1 @@
+{"name":"bar"}

data/spec/fixtures/baz.json

@@ -0,0 +1 @@
+{"name":"baz"}

data/spec/fixtures/foo.json

@@ -0,0 +1 @@
+{"name":"foo"}

data/spec/models/base_spec.rb

@@ -18,9 +18,15 @@ describe Pupa::Base do
           },
         },
       }
-      attr_accessor :name, :label, :founding_date, :inactive, :label_id, :manager_id, :links
+
+      attr_accessor :label, :founding_date, :inactive, :label_id, :manager_id, :links
+      attr_reader :name
       foreign_key :label_id, :manager_id
       foreign_object :label
+
+      def name=(name)
+        @name = name
+      end
     end
   end
 
@@ -32,25 +38,33 @@ describe Pupa::Base do
     Music::Band.new(properties)
   end
 
-  describe '#attr_accessor' do
+  describe '.attr_accessor' do
+    it 'should add properties' do
+      [:_id, :_type, :extras, :label, :founding_date, :inactive, :label_id, :manager_id, :links].each do |property|
+        Music::Band.properties.to_a.should include(property)
+      end
+    end
+  end
+
+  describe '.attr_reader' do
     it 'should add properties' do
-      Music::Band.properties.to_a.should == [:_id, :_type, :extras, :name, :label, :founding_date, :inactive, :label_id, :manager_id, :links]
+      Music::Band.properties.to_a.should include(:name)
     end
   end
 
-  describe '#foreign_key' do
+  describe '.foreign_key' do
     it 'should add foreign keys' do
       Music::Band.foreign_keys.to_a.should == [:label_id, :manager_id]
     end
   end
 
-  describe '#foreign_object' do
+  describe '.foreign_object' do
     it 'should add foreign objects' do
       Music::Band.foreign_objects.to_a.should == [:label]
     end
   end
 
-  describe '#schema=' do
+  describe '.schema=' do
     let :klass_with_absolute_path do
       Class.new(Pupa::Base) do
         self.schema = '/path/to/schema.json'
@@ -82,11 +96,13 @@ describe Pupa::Base do
     end
 
     it 'should accept an absolute path' do
-      klass_with_absolute_path.json_schema.should == '/path/to/schema.json'
+      File.should_receive(:read).and_return('{}')
+      klass_with_absolute_path.json_schema.should == '{}'
     end
 
     it 'should accept a relative path' do
-      klass_with_relative_path.json_schema.should == File.expand_path(File.join('..', '..', 'schemas', 'schema.json'), __dir__)
+      File.should_receive(:read).and_return('{}')
+      klass_with_relative_path.json_schema.should == '{}'
     end
   end
 

data/spec/processor/client_spec.rb

@@ -1,4 +1,15 @@
 require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
 
 describe Pupa::Processor::Client do
+  describe '.new' do
+    it 'should use the filesystem' do
+      ActiveSupport::Cache::FileStore.should_receive(:new).and_call_original
+      Pupa::Processor::Client.new(cache_dir: '/tmp', level: 'UNKNOWN').get('http://example.com/')
+    end
+
+    it 'should use Memcached' do
+      ActiveSupport::Cache::MemCacheStore.should_receive(:new).and_call_original
+      Pupa::Processor::Client.new(cache_dir: 'memcached://localhost', level: 'UNKNOWN').get('http://example.com/')
+    end
+  end
 end

data/spec/processor/document_store/file_store_spec.rb

@@ -0,0 +1,65 @@
+require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
+
+describe Pupa::Processor::DocumentStore::FileStore do
+  let :store do
+    Pupa::Processor::DocumentStore::FileStore.new(File.expand_path(File.join('..', '..', 'fixtures'), __dir__))
+  end
+
+  describe '#exist?' do
+    it 'should return true if the store contains an entry for the given key' do
+      store.exist?('foo.json').should == true
+    end
+
+    it 'should return false if the store does not contain an entry for the given key' do
+      store.exist?('nonexistent').should == false
+    end
+  end
+
+  describe '#entries' do
+    it 'should return all keys in the store' do
+      store.entries.sort.should == %w(bar.json baz.json foo.json)
+    end
+  end
+
+  describe '#read' do
+    it 'should return the value of the given key' do
+      store.read('foo.json').should == {'name' => 'foo'}
+    end
+  end
+
+  describe '#read_multi' do
+    it 'should return the values of the given keys' do
+      store.read_multi(%w(foo.json bar.json)).should == [{'name' => 'foo'}, {'name' => 'bar'}]
+    end
+  end
+
+  describe '#write' do
+    it 'should write an entry with the given value for the given key' do
+      store.exist?('new.json').should == false
+      store.write('new.json', {'name' => 'new'})
+      store.read('new.json').should == {'name' => 'new'}
+      store.delete('new.json') # cleanup
+    end
+  end
+
+  describe '#delete' do
+    it 'should delete an entry with the given key from the store' do
+      store.write('new.json', {'name' => 'new'})
+      store.exist?('new.json').should == true
+      store.delete('new.json')
+      store.exist?('new.json').should == false
+    end
+  end
+
+  describe '#clear' do
+    it 'should delete all entries from the store' do
+      store.entries.sort.should == %w(bar.json baz.json foo.json)
+      store.clear
+      store.entries.should == []
+
+      %w(bar baz foo).each do |name| # cleanup
+        store.write("#{name}.json", {'name' => name})
+      end
+    end
+  end
+end

data/spec/processor/document_store/redis_store_spec.rb

@@ -0,0 +1,71 @@
+require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
+
+describe Pupa::Processor::DocumentStore::RedisStore do
+  def store
+    Pupa::Processor::DocumentStore::RedisStore.new('redis://localhost/15')
+  end
+
+  before :all do
+    %w(foo bar baz).each do |name|
+      store.write("#{name}.json", {'name' => name})
+    end
+  end
+
+  describe '#exist?' do
+    it 'should return true if the store contains an entry for the given key' do
+      store.exist?('foo.json').should == true
+    end
+
+    it 'should return false if the store does not contain an entry for the given key' do
+      store.exist?('nonexistent').should == false
+    end
+  end
+
+  describe '#entries' do
+    it 'should return all keys in the store' do
+      store.entries.sort.should == %w(bar.json baz.json foo.json)
+    end
+  end
+
+  describe '#read' do
+    it 'should return the value of the given key' do
+      store.read('foo.json').should == {'name' => 'foo'}
+    end
+  end
+
+  describe '#read_multi' do
+    it 'should return the values of the given keys' do
+      store.read_multi(%w(foo.json bar.json)).should == [{'name' => 'foo'}, {'name' => 'bar'}]
+    end
+  end
+
+  describe '#write' do
+    it 'should write an entry with the given value for the given key' do
+      store.exist?('new.json').should == false
+      store.write('new.json', {'name' => 'new'})
+      store.read('new.json').should == {'name' => 'new'}
+      store.delete('new.json') # cleanup
+    end
+  end
+
+  describe '#delete' do
+    it 'should delete an entry with the given key from the store' do
+      store.write('new.json', {'name' => 'new'})
+      store.exist?('new.json').should == true
+      store.delete('new.json')
+      store.exist?('new.json').should == false
+    end
+  end
+
+  describe '#clear' do
+    it 'should delete all entries from the store' do
+      store.entries.sort.should == %w(bar.json baz.json foo.json)
+      store.clear
+      store.entries.should == []
+
+      %w(bar baz foo).each do |name| # cleanup
+        store.write("#{name}.json", {'name' => name})
+      end
+    end
+  end
+end

data/spec/processor/document_store_spec.rb

@@ -0,0 +1,15 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::Processor::DocumentStore do
+  describe '.new' do
+    it 'should use the filesystem' do
+      Pupa::Processor::DocumentStore::FileStore.should_receive(:new).with('/tmp').and_call_original
+      Pupa::Processor::DocumentStore.new('/tmp')
+    end
+
+    it 'should use Redis' do
+      Pupa::Processor::DocumentStore::RedisStore.should_receive(:new).with('redis://localhost').and_call_original
+      Pupa::Processor::DocumentStore.new('redis://localhost')
+    end
+  end
+end

data/spec/processor/persistence_spec.rb

@@ -11,7 +11,7 @@ describe Pupa::Processor::Persistence do
     Pupa.session[:people].insert(_type: 'pupa/person', name: 'non-unique')
   end
 
-  describe '#find' do
+  describe '.find' do
     it 'should return nil if no matches' do
       Pupa::Processor::Persistence.find(_type: 'pupa/person', name: 'nonexistent').should == nil
     end

data/spec/processor_spec.rb

@@ -31,6 +31,10 @@ describe Pupa::Processor do
     PersonProcessor.new('/tmp', level: 'WARN', logdev: io)
   end
 
+  let :novalidate do
+    PersonProcessor.new('/tmp', level: 'WARN', logdev: io, validate: false)
+  end
+
   describe '#get' do
     it 'should send a GET request' do
       processor.get('http://httpbin.org/get', 'foo=bar')['args'].should == {'foo' => 'bar'}
@@ -51,7 +55,7 @@ describe Pupa::Processor do
     end
   end
 
-  describe '#add_scraping_task' do
+  describe '.add_scraping_task' do
     it 'should add a scraping task and define a lazy method' do
       PersonProcessor.tasks.should == [:people]
       processor.should respond_to(:people)
@@ -64,9 +68,9 @@ describe Pupa::Processor do
     end
 
     it 'should not overwrite an existing file' do
-      FileUtils.touch(path)
+      File.open(path, 'w') {}
       expect{processor.dump_scraped_objects(:people)}.to raise_error(Pupa::Errors::DuplicateObjectIdError)
-      FileUtils.rm(path)
+      File.delete(path)
     end
 
     it 'should dump a JSON document' do
@@ -80,6 +84,12 @@ describe Pupa::Processor do
       processor.dump_scraped_objects(:people)
       io.string.should match('http://popoloproject.com/schemas/person.json')
     end
+
+    it 'should not validate the object' do
+      novalidate.make_person_invalid
+      novalidate.dump_scraped_objects(:people)
+      io.string.should_not match('http://popoloproject.com/schemas/person.json')
+    end
   end
 
   describe '#import' do

data/spec/spec_helper.rb

@@ -3,6 +3,8 @@ require 'rubygems'
 require 'coveralls'
 Coveralls.wear!
 
+require 'multi_xml'
+require 'redis-store'
 require 'rspec'
 require 'vcr'
 require File.dirname(__FILE__) + '/../lib/pupa'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupa
 version: !ruby/object:Gem::Version
-  version: 0.0.7
+  version: 0.0.8
 platform: ruby
 authors:
 - Open North
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-09-26 00:00:00.000000000 Z
+date: 2013-09-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -122,6 +122,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: dalli
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: json
   requirement: !ruby/object:Gem::Requirement
@@ -136,6 +150,20 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 1.7.7
+- !ruby/object:Gem::Dependency
+  name: multi_xml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: octokit
   requirement: !ruby/object:Gem::Requirement
@@ -165,47 +193,47 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: rspec
+  name: redis-store
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '2.10'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - '>='
       - !ruby/object:Gem::Version
-        version: '2.10'
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: vcr
+  name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.5.0
+        version: '2.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.5.0
+        version: '2.10'
 - !ruby/object:Gem::Dependency
-  name: multi_xml
+  name: vcr
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 2.5.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 2.5.0
 description: 
 email:
 - info@opennorth.ca
@@ -240,6 +268,9 @@ files:
 - lib/pupa/processor.rb
 - lib/pupa/processor/client.rb
 - lib/pupa/processor/dependency_graph.rb
+- lib/pupa/processor/document_store.rb
+- lib/pupa/processor/document_store/file_store.rb
+- lib/pupa/processor/document_store/redis_store.rb
 - lib/pupa/processor/helper.rb
 - lib/pupa/processor/middleware/logger.rb
 - lib/pupa/processor/middleware/parse_html.rb
@@ -264,6 +295,10 @@ files:
 - spec/cassettes/ce69ff734ce852d2bfaa482bbf55d7ffb4762e87.yml
 - spec/cassettes/da629b01e0836deda8a5540a4e6a08783dd7aef9.yml
 - spec/cassettes/e398f35bea86b3d4c87a6934bae1eb7fca8744f9.yml
+- spec/cassettes/f861172f1df3bdb2052af5451f9922699d574b77.yml
+- spec/fixtures/bar.json
+- spec/fixtures/baz.json
+- spec/fixtures/foo.json
 - spec/logger_spec.rb
 - spec/models/base_spec.rb
 - spec/models/concerns/contactable_spec.rb
@@ -280,6 +315,9 @@ files:
 - spec/models/post_spec.rb
 - spec/processor/client_spec.rb
 - spec/processor/dependency_graph_spec.rb
+- spec/processor/document_store/file_store_spec.rb
+- spec/processor/document_store/redis_store_spec.rb
+- spec/processor/document_store_spec.rb
 - spec/processor/helper_spec.rb
 - spec/processor/middleware/logger_spec.rb
 - spec/processor/middleware/parse_html_spec.rb
@@ -319,6 +357,10 @@ test_files:
 - spec/cassettes/ce69ff734ce852d2bfaa482bbf55d7ffb4762e87.yml
 - spec/cassettes/da629b01e0836deda8a5540a4e6a08783dd7aef9.yml
 - spec/cassettes/e398f35bea86b3d4c87a6934bae1eb7fca8744f9.yml
+- spec/cassettes/f861172f1df3bdb2052af5451f9922699d574b77.yml
+- spec/fixtures/bar.json
+- spec/fixtures/baz.json
+- spec/fixtures/foo.json
 - spec/logger_spec.rb
 - spec/models/base_spec.rb
 - spec/models/concerns/contactable_spec.rb
@@ -335,6 +377,9 @@ test_files:
 - spec/models/post_spec.rb
 - spec/processor/client_spec.rb
 - spec/processor/dependency_graph_spec.rb
+- spec/processor/document_store/file_store_spec.rb
+- spec/processor/document_store/redis_store_spec.rb
+- spec/processor/document_store_spec.rb
 - spec/processor/helper_spec.rb
 - spec/processor/middleware/logger_spec.rb
 - spec/processor/middleware/parse_html_spec.rb