pupa 0.0.8 → 0.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d4ec7210671485a2de58673a70088e415a9767b7
-  data.tar.gz: 8b2a77e3fe3c5775509fef59fb847ea2057f63cb
+  metadata.gz: 8124ac65b9df49b337205ce22fe6491ead5a03ec
+  data.tar.gz: e28195cea41f576dea3fa58def6ccfa403d2cf8c
 SHA512:
-  metadata.gz: b3cdcf2da535ebd8d840fe2a1f6e6dd0db68de4d92b728f6f33bf7ba80f7499331f6cc9ace81da93b7e1062e7f3b8389fbab1891d81fbc44d1c48cbbc3be8eea
-  data.tar.gz: 641956572610184f0437f0869e2ee9da3c624e493284ac3d4e0e88157395c4b8d41840ac4f1d351b63032aff1fde091862a2b83401c5ee55c1d83ac52a856a70
+  metadata.gz: acce9361e6ec70f4daf26bbc205724b25dddb7da5437f4fcbca6b7c5fb2df2a56f22e2f6c1cdcd63fac49c5f61c9ed052336223021c40dab522de0eecdaae562
+  data.tar.gz: 606a7695c7a0d722c43e6a574397dfd1870452f972d2d2a7147e67424e750645fdf2c9b5c18edb67dce05d03df1cfa3bc5535c8cda07ae9063a8795ddf4708cb

data/README.md

@@ -45,36 +45,120 @@ The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.htm
 
 1.  You may want more control over the method used to perform a scraping task. For example, a legislature may publish legislators before 1997 in one format and legislators after 1997 in another format. In this case, you may want to select the method used to scrape legislators according to the year. See [legislator.rb](http://opennorth.github.io/pupa-ruby/docs/legislator.html).
 
+### Automatic response parsing
+
+JSON parsing is enabled by default. To enable automatic parsing of HTML and XML, require the `nokogiri` and `multi_xml` gems.
+
 ## Performance
 
 Pupa.rb offers several ways to significantly improve performance.
 
-In an example case, reducing file I/O and skipping validation as described below reduced the time to scrape 10,000 documents from 100 cached HTTP responses from 100 seconds down to 5 seconds. Like fast tests, fast scrapers make development smoother.
+In an example case, reducing disk I/O and skipping validation as described below reduced the time to scrape 10,000 documents from 100 cached HTTP responses from 100 seconds down to 5 seconds. Like fast tests, fast scrapers make development smoother.
 
-The `import` action's performance (when using a dependency graph) is currently limited by MongoDB.
+The `import` action's performance is currently limited by MongoDB when a dependency graph is used to determine the evaluation order. If a dependency graph cannot be used because you don't know a related object's ID, [several optimizations](https://github.com/opennorth/pupa-ruby/issues/12) can be implemented to improve performance.
 
-### Caching HTTP requests
+### Reducing HTTP requests
 
 HTTP requests consume the most time. To avoid repeat HTTP requests while developing a scraper, cache all HTTP responses. Pupa.rb will by default use a `web_cache` directory in the same directory as your script. You can change the directory by setting the `--cache_dir` switch on the command line, for example:
 
-    ruby cat.rb --cache_dir my_cache_dir
+    ruby cat.rb --cache_dir /tmp/my_cache_dir
+
+### Parallelizing HTTP requests
+
+To enable parallel requests, use the `typhoeus` gem. Unless you are using an old version of Typhoeus (< 0.5), both Faraday and Typhoeus define a Faraday adapter, but you must use the one defined by Typhoeus, like so:
+
+```ruby
+require 'pupa'
+require 'typhoeus'
+require 'typhoeus/adapters/faraday'
+```
+
+Then, in your scraping methods, write code like:
+
+```ruby
+responses = []
+
+# Change the maximum number of concurrent requests (default 200). You usually
+# need to tweak this number by trial and error.
+# @see https://github.com/lostisland/faraday/wiki/Parallel-requests#advanced-use
+manager = Typhoeus::Hydra.new(max_concurrency: 20)
+
+begin
+  # Send HTTP requests in parallel.
+  client.in_parallel(manager) do
+    responses << client.get('http://example.com/foo')
+    responses << client.get('http://example.com/bar')
+    # More requests...
+  end
+rescue Faraday::Error::ClientError => e
+  # Log an error message if, for example, you exceed a server's maximum number
+  # of concurrent connections or if you exceed an API's rate limit.
+  error(e.response.inspect)
+end
+
+# Responses are now available for use.
+responses.each do |response|
+  # Only process the finished responses.
+  if response.success?
+    # If success...
+  elsif response.finished?
+    # If error...
+  end
+end
+```
+
+### Reducing disk I/O
+
+After HTTP requests, disk I/O is the slowest operation. Two types of files are written to disk: HTTP responses are written to the cache directory, and JSON documents are written to the output directory. Writing to memory is much faster than writing to disk.
+
+#### RAM file systems
+
+A simple solution is to create a file system in RAM, like `tmpfs` on Linux for example, and to use it as your `output_dir` and  `cache_dir`. On OS X, you must create a RAM disk. To create a 128MB RAM disk, for example, run:
 
-### Reducing file I/O
+    ramdisk=$(hdiutil attach -nomount ram://$((128 * 2048)) | tr -d ' \t')
+    diskutil erasevolume HFS+ 'ramdisk' $ramdisk
 
-After HTTP requests, file I/O is the slowest operation. Two types of files are written to disk: HTTP responses are written to the cache directory, and JSON documents are written to the output directory. Writing to memory is much faster than writing to disk. You may store HTTP responses in [Memcached](http://memcached.org/) like so:
+You can then set the `output_dir` and `cache_dir` on OS X as:
+
+    ruby cat.rb --output_dir /Volumes/ramdisk/scraped_data --cache_dir /Volumes/ramdisk/web_cache
+
+Once you are done with the RAM disk, release the memory:
+
+    diskutil unmount $ramdisk
+    hdiutil detach $ramdisk
+
+Using a RAM disk will significantly improve performance; however, the data will be lost between reboots unless you move the data to a hard disk. Using Memcached (for caching) and Redis (for storage) is moderately faster than using a RAM disk, and Redis will not lose your output data between reboots.
+
+#### Memcached
+
+You may cache HTTP responses in [Memcached](http://memcached.org/). First, require the `dalli` gem. Then:
 
     ruby cat.rb --cache_dir memcached://localhost:11211
 
-And you may store JSON documents in [Redis](http://redis.io/) like so:
+The data in Memcached will be lost between reboots.
+
+#### Redis
+
+You may dump JSON documents in [Redis](http://redis.io/). First, require the `redis-store` gem. Then:
 
     ruby cat.rb --output_dir redis://localhost:6379/0
 
-Note that Pupa.rb flushes the JSON documents before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa.rb and other applications. You can select a different database than the default `0` for use with Pupa.rb by passing an argument like `redis://localhost:6379/1`, where `1` is the Redis database number.
+To dump JSON documents in Redis moderately faster, use [pipelining](http://redis.io/topics/pipelining):
+
+    ruby cat.rb --output_dir redis://localhost:6379/0 --pipelined
+
+Requiring the `hiredis` gem will slightly improve performance.
+
+Note that Pupa.rb flushes the Redis database before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa.rb and other applications. You can select a different database than the default `0` for use with Pupa.rb by passing an argument like `redis://localhost:6379/15`, where `15` is the database number.
 
 ### Skipping validation
 
 The `json-schema` gem is slow compared to, for example, [JSV](https://github.com/garycourt/JSV). Setting the `--no-validate` switch and running JSON Schema validations separately can further reduce a scraper's running time.
 
+### Parsing JSON
+
+If the rest of your scraper is fast, you may see an improvement by using the `oj` gem. Just `require 'oj'` and Pupa.rb will automatically pick it up, since it uses [MultiJson](https://github.com/intridea/multi_json).
+
 ### Profiling
 
 You can profile your code using [perftools.rb](https://github.com/tmm1/perftools.rb). First, install the gem:
@@ -85,7 +169,7 @@ Then, run your script with the profiler (changing `/tmp/PROFILE_NAME` and `scrip
 
     CPUPROFILE=/tmp/PROFILE_NAME RUBYOPT="-r`gem which perftools | tail -1`" ruby script.rb
 
-You may want to set the `CPUPROFILE_REALTIME=1` flag; however, it seems to change the behavior of the `json-schema` gem, for whatever reason.
+You may want to set the `CPUPROFILE_REALTIME=1` flag; however, it seems to interfere with HTTP requests, for whatever reason.
 
 [perftools.rb](https://github.com/tmm1/perftools.rb) has several output formats. If your code is straight-forward, you can draw a graph (changing `/tmp/PROFILE_NAME` and `/tmp/PROFILE_NAME.pdf` as appropriate):
 

data/lib/pupa/models/concerns/contactable.rb

@@ -6,6 +6,7 @@ module Pupa
 
       included do
         attr_reader :contact_details
+        dump :contact_details
       end
 
       # Sets the contact details.

data/lib/pupa/models/concerns/identifiable.rb

@@ -6,6 +6,7 @@ module Pupa
 
       included do
         attr_reader :identifiers
+        dump :identifiers
       end
 
       # Sets the identifiers.

data/lib/pupa/models/concerns/linkable.rb

@@ -6,6 +6,7 @@ module Pupa
 
       included do
         attr_accessor :links
+        dump :links
       end
 
       # Adds a URL.

data/lib/pupa/models/concerns/nameable.rb

@@ -6,6 +6,7 @@ module Pupa
 
       included do
         attr_accessor :other_names
+        dump :other_names
       end
 
       # Adds an alternate or former name.

data/lib/pupa/models/concerns/sourceable.rb

@@ -6,6 +6,7 @@ module Pupa
 
       included do
         attr_accessor :sources
+        dump :sources
       end
 
       # Adds a source to the object.

data/lib/pupa/models/concerns/timestamps.rb

@@ -8,6 +8,7 @@ module Pupa
 
       included do
         attr_accessor :created_at, :updated_at
+        dump :created_at, :updated_at
 
         set_callback(:create, :before) do |object|
           object.created_at = Time.now.utc

data/lib/pupa/models/membership.rb

@@ -1,6 +1,8 @@
 module Pupa
   # A relationship between a person and an organization.
-  class Membership < Base
+  class Membership
+    include Model
+
     self.schema = 'popolo/membership'
 
     include Concerns::Timestamps
@@ -10,6 +12,8 @@ module Pupa
 
     attr_accessor :label, :role, :person_id, :organization_id, :post_id,
       :start_date, :end_date
+    dump :label, :role, :person_id, :organization_id, :post_id,
+      :start_date, :end_date
 
     foreign_key :person_id, :organization_id, :post_id
 

data/lib/pupa/models/{base.rb → model.rb}

@@ -3,9 +3,6 @@ require 'securerandom'
 require 'set'
 
 require 'active_support/callbacks'
-require 'active_support/core_ext/hash/except'
-require 'active_support/core_ext/hash/keys'
-require 'active_support/core_ext/hash/slice'
 require 'active_support/core_ext/object/try'
 require 'json-schema'
 
@@ -14,43 +11,36 @@ require 'pupa/refinements/json-schema'
 JSON::Validator.cache_schemas = true
 
 module Pupa
-  # The base class from which other primary Popolo classes inherit.
-  class Base
-    include ActiveSupport::Callbacks
-    define_callbacks :create, :save
-
-    class_attribute :json_schema
-    class_attribute :properties
-    class_attribute :foreign_keys
-    class_attribute :foreign_objects
-
-    self.properties = Set.new
-    self.foreign_keys = Set.new
-    self.foreign_objects = Set.new
-
-    class << self
-      # Declare the class' properties.
-      #
-      # When converting an object to a hash using the `to_h` method, only the
-      # properties declared with `attr_accessor` or `attr_reader` will be
-      # included in the hash.
-      #
-      # @param [Array<Symbol>] the class' properties
-      def attr_accessor(*attributes)
-        self.properties += attributes # use assignment to not overwrite the parent's attribute
-        super
-      end
+  # Adds methods expected by Pupa processors.
+  module Model
+    extend ActiveSupport::Concern
 
-      # Declare the class' properties.
-      #
-      # When converting an object to a hash using the `to_h` method, only the
-      # properties declared with `attr_accessor` or `attr_reader` will be
-      # included in the hash.
+    included do
+      include ActiveSupport::Callbacks
+      define_callbacks :create, :save
+
+      class_attribute :json_schema
+      class_attribute :properties
+      class_attribute :foreign_keys
+      class_attribute :foreign_objects
+
+      self.properties = Set.new
+      self.foreign_keys = Set.new
+      self.foreign_objects = Set.new
+
+      attr_reader :_id
+      attr_accessor :_type, :extras
+
+      dump :_id, :_type, :extras
+    end
+
+    module ClassMethods
+      # Declare which properties should be dumped to JSON after a scraping task
+      # is complete. A subset of these properties will be imported to MongoDB.
       #
-      # @param [Array<Symbol>] the class' properties
-      def attr_reader(*attributes)
+      # @param [Array<Symbol>] the properties to dump to JSON
+      def dump(*attributes)
         self.properties += attributes # use assignment to not overwrite the parent's attribute
-        super
       end
 
       # Declare the class' foreign keys.
@@ -91,8 +81,6 @@ module Pupa
       end
     end
 
-    attr_accessor :_id, :_type, :extras
-
     # @param [Hash] properties the object's properties
     def initialize(properties = {})
       @_type = self.class.to_s.underscore
@@ -149,14 +137,14 @@ module Pupa
     #
     # @return [Hash] a subset of the object's properties
     def fingerprint
-      to_h.except(:_id)
+      to_h(persist: true).except(:_id)
     end
 
     # Returns the object's foreign keys and foreign objects.
     #
     # @return [Hash] the object's foreign keys and foreign objects
     def foreign_properties
-      to_h(include_foreign_objects: true).slice(*foreign_keys + foreign_objects)
+      to_h.slice(*foreign_keys + foreign_objects)
     end
 
     # Validates the object against the schema.
@@ -165,17 +153,19 @@ module Pupa
     def validate!
       if self.class.json_schema
         # JSON::Validator#initialize_schema runs fastest if given a hash.
-        JSON::Validator.validate!(self.class.json_schema, stringify_keys(to_h))
+        JSON::Validator.validate!(self.class.json_schema, stringify_keys(to_h(persist: true)))
       end
     end
 
     # Returns the object as a hash.
     #
-    # @param [Boolean] include_foreign_objects whether to include foreign objects
+    # @param [Boolean] persist whether the object is being persisted, validated
+    #   or used as a MongoDB selecto, in which case foreign objects (i.e. hints)
+    #   are excluded
     # @return [Hash] the object as a hash
-    def to_h(include_foreign_objects: false)
+    def to_h(persist: false)
       {}.tap do |hash|
-        (include_foreign_objects ? properties : properties - foreign_objects).each do |property|
+        (persist ? properties - foreign_objects : properties).each do |property|
           value = self[property]
           if value == false || value.present?
             hash[property] = value

data/lib/pupa/models/organization.rb

@@ -1,7 +1,9 @@
 module Pupa
   # A group with a common purpose or reason for existence that goes beyond the set
   # of people belonging to it.
-  class Organization < Base
+  class Organization
+    include Model
+
     self.schema = 'popolo/organization'
 
     include Concerns::Timestamps
@@ -13,6 +15,8 @@ module Pupa
 
     attr_accessor :name, :classification, :parent_id, :parent, :founding_date,
       :dissolution_date, :image
+    dump :name, :classification, :parent_id, :parent, :founding_date,
+      :dissolution_date, :image
 
     foreign_key :parent_id
 

data/lib/pupa/models/person.rb

@@ -1,6 +1,8 @@
 module Pupa
   # A real person, alive or dead.
-  class Person < Base
+  class Person
+    include Model
+
     self.schema = 'popolo/person'
 
     include Concerns::Timestamps
@@ -13,6 +15,9 @@ module Pupa
     attr_accessor :name, :family_name, :given_name, :additional_name,
       :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name,
       :email, :gender, :birth_date, :death_date, :image, :summary, :biography
+    dump :name, :family_name, :given_name, :additional_name,
+      :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name,
+      :email, :gender, :birth_date, :death_date, :image, :summary, :biography
 
     # Returns the person's name.
     #

data/lib/pupa/models/post.rb

@@ -1,6 +1,8 @@
 module Pupa
   # A position that exists independent of the person holding it.
-  class Post < Base
+  class Post
+    include Model
+
     self.schema = 'popolo/post'
 
     include Concerns::Timestamps
@@ -9,6 +11,7 @@ module Pupa
     include Concerns::Linkable
 
     attr_accessor :label, :role, :organization_id, :start_date, :end_date
+    dump :label, :role, :organization_id, :start_date, :end_date
 
     foreign_key :organization_id
 

data/lib/pupa/processor/client.rb

@@ -4,6 +4,8 @@ require 'faraday_middleware/response_middleware'
 
 require 'pupa/processor/middleware/logger'
 require 'pupa/processor/middleware/parse_html'
+require 'pupa/processor/middleware/parse_json'
+require 'pupa/processor/middleware/raise_error'
 require 'pupa/refinements/faraday_middleware'
 
 begin
@@ -18,7 +20,9 @@ module Pupa
     class Client
       # Returns a configured Faraday HTTP client.
       #
-      # In order to automatically parse XML responses, you must `require 'multi_xml'`.
+      # To automatically parse XML responses, you must `require 'multi_xml'`.
+      #
+      # Memcached support depends on the `dalli` gem.
       #
       # @param [String] cache_dir a directory or a Memcached address
       #   (e.g. `memcached://localhost:11211`) in which to cache requests
@@ -29,16 +33,19 @@ module Pupa
         Faraday.new do |connection|
           connection.request :url_encoded
           connection.use Middleware::Logger, Logger.new('faraday', level: level)
+          connection.use Middleware::RaiseError # useful for breaking concurrent requests
+
+          # @see http://tools.ietf.org/html/rfc4627
+          connection.use Middleware::ParseJson, content_type: /\bjson$/
 
           # @see http://tools.ietf.org/html/rfc2854
           # @see http://tools.ietf.org/html/rfc3236
-          connection.use Middleware::ParseHtml, content_type: %w(text/html application/xhtml+xml)
-
-          # @see http://tools.ietf.org/html/rfc4627
-          connection.use FaradayMiddleware::ParseJson, content_type: /\bjson$/
+          if defined?(Nokogiri)
+            connection.use Middleware::ParseHtml, content_type: %w(text/html application/xhtml+xml)
+          end
 
+          # @see http://tools.ietf.org/html/rfc3023
           if defined?(MultiXml)
-            # @see http://tools.ietf.org/html/rfc3023
             connection.use FaradayMiddleware::ParseXml, content_type: /\bxml$/
           end
 
@@ -53,7 +60,11 @@ module Pupa
             end
           end
 
-          connection.adapter Faraday.default_adapter # must be last
+          if defined?(Typhoeus)
+            connection.adapter :typhoeus
+          else
+            connection.adapter Faraday.default_adapter # must be last
+          end
         end
       end
     end

data/lib/pupa/processor/document_store/file_store.rb

@@ -34,7 +34,7 @@ module Pupa
         # @return [Hash] the value of the given key
         def read(name)
           File.open(namespaced_key(name)) do |f|
-            JSON.load(f)
+            MultiJson.load(f)
           end
         end
 
@@ -54,7 +54,28 @@ module Pupa
         # @param [Hash] value a value
         def write(name, value)
           File.open(namespaced_key(name), 'w') do |f|
-            JSON.dump(value, f)
+            f.write(MultiJson.dump(value))
+          end
+        end
+
+        # Writes, as JSON, the value to a file with the given name, unless such
+        # a file exists.
+        #
+        # @param [String] name a key
+        # @param [Hash] value a value
+        # @return [Boolean] whether the key was set
+        def write_unless_exists(name, value)
+          !exist?(name).tap do |exists|
+            write(name, value) unless exists
+          end
+        end
+
+        # Writes, as JSON, the values to files with the given names.
+        #
+        # @param [Hash] pairs key-value pairs
+        def write_multi(pairs)
+          pairs.each do |name,value|
+            write(name, value)
           end
         end
 
@@ -72,6 +93,11 @@ module Pupa
           end
         end
 
+        # Collects commands to run all at once.
+        def pipelined
+          yield
+        end
+
       private
 
         def namespaced_key(name)

data/lib/pupa/processor/document_store/redis_store.rb

@@ -8,16 +8,17 @@ module Pupa
       # can select a different database than the default `0` for use with Pupa
       # by passing an argument like `redis://localhost:6379/0`.
       #
-      # @note Redis support depends on the `redis` gem. For better performance,
-      #   use the `hiredis` gem as well.
+      # @note Redis support depends on the `redis-store` gem. You may optionally
+      #   use the `hiredis` gem to further improve performance.
       class RedisStore
         # @param [String] address the address (e.g. `redis://localhost:6379/0`)
         #   in which to dump JSON documents
-        def initialize(address)
-          options = {}
-          if defined?(Hiredis)
-            options.update(driver: :hiredis)
-          end
+        # @param [Boolean] pipelined whether to enable
+        #   [pipelining](http://redis.io/topics/pipelining)
+        def initialize(address, pipelined: false)
+          @pipelined = pipelined
+          options = {marshalling: false}
+          options.update(driver: :hiredis) if defined?(Hiredis)
           @redis = Redis::Store::Factory.create(address, options)
         end
 
@@ -41,7 +42,7 @@ module Pupa
         # @param [String] name a key
         # @return [Hash] the value of the given key
         def read(name)
-          JSON.load(@redis.get(name))
+          MultiJson.load(@redis.get(name))
         end
 
         # Returns, as JSON, the values of the given keys.
@@ -49,7 +50,7 @@ module Pupa
         # @param [String] names keys
         # @return [Array<Hash>] the values of the given keys
         def read_multi(names)
-          @redis.mget(*names).map{|value| JSON.load(value)}
+          @redis.mget(*names).map{|value| MultiJson.load(value)}
         end
 
         # Writes, as JSON, the value to a key.
@@ -57,7 +58,28 @@ module Pupa
         # @param [String] name a key
         # @param [Hash] value a value
         def write(name, value)
-          @redis.set(name, JSON.dump(value))
+          @redis.set(name, MultiJson.dump(value))
+        end
+
+        # Writes, as JSON, the value to a key, unless the key exists.
+        #
+        # @param [String] name a key
+        # @param [Hash] value a value
+        # @return [Boolean] whether the key was set
+        def write_unless_exists(name, value)
+          @redis.setnx(name, MultiJson.dump(value))
+        end
+
+        # Writes, as JSON, the values to keys.
+        #
+        # @param [Hash] pairs key-value pairs
+        def write_multi(pairs)
+          args = []
+          pairs.each do |key,value|
+            args << key
+            args << MultiJson.dump(value)
+          end
+          @redis.mset(*args)
         end
 
         # Delete a key.
@@ -71,6 +93,17 @@ module Pupa
         def clear
           @redis.flushdb
         end
+
+        # Collects commands to run all at once.
+        def pipelined
+          if @pipelined
+            @redis.pipelined do
+              yield
+            end
+          else
+            yield
+          end
+        end
       end
     end
   end

data/lib/pupa/processor/document_store.rb

@@ -6,12 +6,15 @@ module Pupa
     class DocumentStore
       # Returns a configured JSON document store.
       #
+      # See each document store for more information.
+      #
       # @param [String] argument the filesystem directory or Redis address
       #   (e.g. `redis://localhost:6379/0`) in which to dump JSON documents
+      # @param [Hash] options optional arguments
       # @return a configured JSON document store
-      def self.new(argument)
+      def self.new(argument, **options)
         if argument[%r{\Aredis://}]
-          RedisStore.new(argument)
+          RedisStore.new(argument, options)
         else
           FileStore.new(argument)
         end

data/lib/pupa/processor/middleware/parse_html.rb

@@ -7,9 +7,9 @@ module Pupa
       class ParseHtml < FaradayMiddleware::ResponseMiddleware
         dependency 'nokogiri'
 
-        define_parser { |body|
+        define_parser do |body|
           Nokogiri::HTML(body) unless body.empty?
-        }
+        end
       end
     end
   end

data/lib/pupa/processor/middleware/parse_json.rb

@@ -0,0 +1,16 @@
+module Pupa
+  class Processor
+    module Middleware
+      # A Faraday response middleware for parsing JSON.
+      #
+      # @see https://github.com/lostisland/faraday_middleware/issues/30#issuecomment-4706892
+      class ParseJson < FaradayMiddleware::ResponseMiddleware
+        dependency 'multi_json'
+
+        define_parser do |body|
+          MultiJson.load(body) unless body.strip.empty?
+        end
+      end
+    end
+  end
+end

data/lib/pupa/processor/middleware/raise_error.rb

@@ -0,0 +1,33 @@
+module Pupa
+  class Processor
+    module Middleware
+      # A Faraday response middleware for raising an error if unsuccessful.
+      #
+      # @see Faraday::Response::RaiseError
+      class RaiseError < Faraday::Response::Middleware
+        def on_complete(env)
+          case env[:status]
+          when 404
+            raise Faraday::Error::ResourceNotFound, response_values(env)
+          when 407
+            # mimic the behavior that we get with proxy requests with HTTPS
+            raise Faraday::Error::ConnectionFailed, %{407 "Proxy Authentication Required "}
+          when 400...600
+            raise Faraday::Error::ClientError, response_values(env)
+          end
+        end
+
+        def response_values(env) # XXX add more keys
+          {
+            method: env[:method],
+            url: env[:url].to_s,
+            request_headers: env[:request_headers],
+            status: env[:status],
+            response_headers: env[:response_headers],
+            body: env[:body].to_s,
+          }
+        end
+      end
+    end
+  end
+end

data/lib/pupa/processor/persistence.rb

@@ -24,7 +24,7 @@ module Pupa
         when 1
           query.first
         else
-          raise Errors::TooManyMatches, "selector matches multiple documents during find: #{collection_name} #{JSON.dump(selector)}"
+          raise Errors::TooManyMatches, "selector matches multiple documents during find: #{collection_name} #{MultiJson.dump(selector)}"
         end
       end
 
@@ -41,14 +41,14 @@ module Pupa
           case query.count
           when 0
             @object.run_callbacks(:create) do
-              collection.insert(@object.to_h)
+              collection.insert(@object.to_h(persist: true))
               @object._id.to_s
             end
           when 1
-            query.update(@object.to_h)
+            query.update(@object.to_h(persist: true))
             query.first['_id'].to_s
           else
-            raise Errors::TooManyMatches, "selector matches multiple documents during save: #{collection_name} #{JSON.dump(selector)}"
+            raise Errors::TooManyMatches, "selector matches multiple documents during save: #{collection_name} #{MultiJson.dump(selector)}"
           end
         end
       end

data/lib/pupa/processor.rb

@@ -1,7 +1,3 @@
-require 'json'
-
-require 'nokogiri'
-
 require 'pupa/processor/client'
 require 'pupa/processor/dependency_graph'
 require 'pupa/processor/helper'
@@ -30,12 +26,13 @@ module Pupa
     # @param [String] cache_dir the directory or Memcached address
     #   (e.g. `memcached://localhost:11211`) in which to cache HTTP responses
     # @param [Integer] expires_in the cache's expiration time in seconds
+    # @param [Boolean] pipelined whether to dump JSON documents all at once
     # @param [Boolean] validate whether to validate JSON documents
     # @param [String] level the log level
     # @param [String,IO] logdev the log device
     # @param [Hash] options criteria for selecting the methods to run
-    def initialize(output_dir, cache_dir: nil, expires_in: 86400, validate: true, level: 'INFO', logdev: STDOUT, options: {})
-      @store    = DocumentStore.new(output_dir)
+    def initialize(output_dir, cache_dir: nil, expires_in: 86400, pipelined: false, validate: true, level: 'INFO', logdev: STDOUT, options: {})
+      @store    = DocumentStore.new(output_dir, pipelined: pipelined)
       @client   = Client.new(cache_dir: cache_dir, expires_in: expires_in, level: level)
       @logger   = Logger.new('pupa', level: level, logdev: logdev)
       @validate = validate
@@ -73,6 +70,15 @@ module Pupa
       client.post(url, params).body
     end
 
+    # Yields the object to the transformation task for processing, e.g. saving
+    # to disk, printing to CSV, etc.
+    #
+    # @param [Object] an object
+    # @note All the good terms are taken by Ruby: `return`, `send` and `yield`.
+    def dispatch(object)
+      Fiber.yield(object)
+    end
+
     # Adds a scraping task to Pupa.rb.
     #
     # Defines a method whose name is identical to `task_name`. This method
@@ -113,9 +119,11 @@ module Pupa
     # @return [Integer] the number of scraped objects
     def dump_scraped_objects(task_name)
       count = 0
-      send(task_name).each do |object|
-        count += 1 # we don't know the size of the enumeration
-        dump_scraped_object(object)
+      @store.pipelined do
+        send(task_name).each do |object|
+          count += 1 # we don't know the size of the enumeration
+          dump_scraped_object(object)
+        end
       end
       count
     end
@@ -182,7 +190,7 @@ module Pupa
         end
 
         unless objects.empty?
-          raise Errors::UnprocessableEntity, "couldn't resolve #{objects.size}/#{size} objects:\n  #{objects.values.map{|object| JSON.dump(object.foreign_properties)}.join("\n  ")}"
+          raise Errors::UnprocessableEntity, "couldn't resolve #{objects.size}/#{size} objects:\n  #{objects.values.map{|object| MultiJson.dump(object.foreign_properties)}.join("\n  ")}"
         end
       end
 
@@ -222,14 +230,12 @@ module Pupa
       type = object.class.to_s.demodulize.underscore
       name = "#{type}_#{object._id.gsub(File::SEPARATOR, '_')}.json"
 
-      if @store.exist?(name)
+      if @store.write_unless_exists(name, object.to_h)
+        info {"save #{type} #{object.to_s} as #{name}"}
+      else
         raise Errors::DuplicateObjectIdError, "duplicate object ID: #{object._id} (was the same objected yielded twice?)"
       end
 
-      info {"save #{type} #{object.to_s} as #{name}"}
-
-      @store.write(name, object.to_h(include_foreign_objects: true))
-
       if @validate
         begin
           object.validate!

data/lib/pupa/runner.rb

@@ -18,6 +18,7 @@ module Pupa
         output_dir:     File.expand_path('scraped_data', Dir.pwd),
         cache_dir:      File.expand_path('web_cache', Dir.pwd),
         expires_in:     86400, # 1 day
+        pipelined:      false,
         validate:       true,
         host_with_port: 'localhost:27017',
         database:       'pupa',
@@ -81,6 +82,9 @@ module Pupa
         opts.on('-e', '--expires_in SECONDS', "The cache's expiration time in seconds") do |v|
           options.expires_in = v
         end
+        opts.on('--pipelined', 'Dump JSON documents all at once') do |v|
+          options.pipelined = v
+        end
         opts.on('--[no-]validate', 'Validate JSON documents') do |v|
           options.validate = v
         end
@@ -143,6 +147,7 @@ module Pupa
       processor = @processor_class.new(options.output_dir,
         cache_dir: options.cache_dir,
         expires_in: options.expires_in,
+        pipelined: options.pipelined,
         validate: options.validate,
         level: options.level,
         options: Hash[*rest])
@@ -173,7 +178,7 @@ module Pupa
       report = {
         plan: {
           processor: @processor_class,
-          arguments: options.to_h,
+          arguments: options.dup.to_h,
           options: rest,
         },
         start: Time.now.utc,
@@ -198,7 +203,7 @@ module Pupa
 
       report[:end] = Time.now.utc
       report[:time] = report[:end] - report[:start]
-      puts JSON.dump(report)
+      puts MultiJson.dump(report)
     end
   end
 end

data/lib/pupa/version.rb

@@ -1,3 +1,3 @@
 module Pupa
-  VERSION = "0.0.8"
+  VERSION = "0.0.9"
 end

data/lib/pupa.rb

@@ -3,6 +3,8 @@ require 'forwardable'
 
 require 'active_support/concern'
 require 'active_support/core_ext/class/attribute'
+require 'active_support/core_ext/hash/except'
+require 'active_support/core_ext/hash/slice'
 require 'active_support/core_ext/object/blank'
 require 'active_support/inflector'
 
@@ -18,7 +20,7 @@ require 'pupa/models/concerns/nameable'
 require 'pupa/models/concerns/sourceable'
 require 'pupa/models/concerns/timestamps'
 
-require 'pupa/models/base'
+require 'pupa/models/model'
 require 'pupa/models/contact_detail_list'
 require 'pupa/models/identifier_list'
 require 'pupa/models/membership'

data/pupa.gemspec

@@ -22,15 +22,16 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency('json-schema', '~> 2.1.3')
   s.add_runtime_dependency('mail')
   s.add_runtime_dependency('moped', '~> 1.5.1')
-  s.add_runtime_dependency('nokogiri', '~> 1.6.0')
 
   s.add_development_dependency('coveralls')
   s.add_development_dependency('dalli')
   s.add_development_dependency('json', '~> 1.7.7') # to silence coveralls warning
   s.add_development_dependency('multi_xml')
+  s.add_development_dependency('nokogiri', '~> 1.6.0')
   s.add_development_dependency('octokit') # to update Popolo schema
   s.add_development_dependency('rake')
   s.add_development_dependency('redis-store')
   s.add_development_dependency('rspec', '~> 2.10')
+  s.add_development_dependency('typhoeus')
   s.add_development_dependency('vcr', '~> 2.5.0')
 end

data/spec/models/base_spec.rb

@@ -1,8 +1,10 @@
 require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
 
-describe Pupa::Base do
+describe Pupa::Model do
   module Music
-    class Band < Pupa::Base
+    class Band
+      include Pupa::Model
+
       self.schema = {
         '$schema' => 'http://json-schema.org/draft-03/schema#',
         'properties' => {
@@ -19,14 +21,10 @@ describe Pupa::Base do
         },
       }
 
-      attr_accessor :label, :founding_date, :inactive, :label_id, :manager_id, :links
-      attr_reader :name
+      attr_accessor :name, :label, :founding_date, :inactive, :label_id, :manager_id, :links
+      dump :name, :label, :founding_date, :inactive, :label_id, :manager_id, :links
       foreign_key :label_id, :manager_id
       foreign_object :label
-
-      def name=(name)
-        @name = name
-      end
     end
   end
 
@@ -38,20 +36,14 @@ describe Pupa::Base do
     Music::Band.new(properties)
   end
 
-  describe '.attr_accessor' do
+  describe '.dump' do
     it 'should add properties' do
-      [:_id, :_type, :extras, :label, :founding_date, :inactive, :label_id, :manager_id, :links].each do |property|
+      [:_id, :_type, :extras, :name, :label, :founding_date, :inactive, :label_id, :manager_id, :links].each do |property|
         Music::Band.properties.to_a.should include(property)
       end
     end
   end
 
-  describe '.attr_reader' do
-    it 'should add properties' do
-      Music::Band.properties.to_a.should include(:name)
-    end
-  end
-
   describe '.foreign_key' do
     it 'should add foreign keys' do
       Music::Band.foreign_keys.to_a.should == [:label_id, :manager_id]
@@ -66,13 +58,15 @@ describe Pupa::Base do
 
   describe '.schema=' do
     let :klass_with_absolute_path do
-      Class.new(Pupa::Base) do
+      Class.new do
+        include Pupa::Model
         self.schema = '/path/to/schema.json'
       end
     end
 
     let :klass_with_relative_path do
-      Class.new(Pupa::Base) do
+      Class.new do
+        include Pupa::Model
         self.schema = 'schema'
       end
     end
@@ -178,7 +172,9 @@ describe Pupa::Base do
 
   describe '#validate!' do
     let :klass_without_schema do
-      Class.new(Pupa::Base)
+      Class.new do
+        include Pupa::Model
+      end
     end
 
     it 'should do nothing if the schema is not set' do
@@ -196,12 +192,12 @@ describe Pupa::Base do
   end
 
   describe '#to_h' do
-    it 'should not include foreign objects by default' do
-      object.to_h.should == {_id: object._id, _type: 'music/band', name: 'Moderat', inactive: false, manager_id: '1', links: [{url: 'http://moderat.fm/'}]}
+    it 'should include all properties by default' do
+      object.to_h.should == {_id: object._id, _type: 'music/band', name: 'Moderat', label: {name: 'Mute'}, inactive: false, manager_id: '1', links: [{url: 'http://moderat.fm/'}]}
     end
 
-    it 'should include foreign objects if desired' do
-      object.to_h(include_foreign_objects: true).should == {_id: object._id, _type: 'music/band', name: 'Moderat', label: {name: 'Mute'}, inactive: false, manager_id: '1', links: [{url: 'http://moderat.fm/'}]}
+    it 'should exclude foreign objects if persisting' do
+      object.to_h(persist: true).should == {_id: object._id, _type: 'music/band', name: 'Moderat', inactive: false, manager_id: '1', links: [{url: 'http://moderat.fm/'}]}
     end
 
     it 'should not include blank properties' do

data/spec/models/concerns/contactable_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Contactable do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Contactable
     end
   end

data/spec/models/concerns/identifiable_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Identifiable do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Identifiable
     end
   end

data/spec/models/concerns/linkable_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Linkable do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Linkable
     end
   end

data/spec/models/concerns/nameable_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Nameable do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Nameable
     end
   end

data/spec/models/concerns/sourceable_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Sourceable do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Sourceable
     end
   end

data/spec/models/concerns/timestamps_spec.rb

@@ -2,7 +2,8 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Concerns::Timestamps do
   let :klass do
-    Class.new(Pupa::Base) do
+    Class.new do
+      include Pupa::Model
       include Pupa::Concerns::Timestamps
 
       def save

data/spec/processor/document_store/file_store_spec.rb

@@ -42,6 +42,38 @@ describe Pupa::Processor::DocumentStore::FileStore do
     end
   end
 
+  describe '#write_unless_exists' do
+    it 'should write an entry with the given value for the given key' do
+      store.exist?('new.json').should == false
+      store.write_unless_exists('new.json', {'name' => 'new'}).should == true
+      store.read('new.json').should == {'name' => 'new'}
+      store.delete('new.json') # cleanup
+    end
+
+    it 'should not write an entry with the given value for the given key if the key exists' do
+      store.write_unless_exists('foo.json', {'name' => 'new'}).should == false
+      store.read('foo.json').should == {'name' => 'foo'}
+    end
+  end
+
+  describe '#write_multi' do
+    it 'should write entries with the given values for the given keys' do
+      pairs = {}
+      %w(new1 new2).each do |name|
+        pairs["#{name}.json"] = {'name' => name}
+      end
+
+      pairs.keys.each do |name|
+        store.exist?(name).should == false
+      end
+      store.write_multi(pairs)
+      store.read_multi(pairs.keys).should == [{'name' => 'new1'}, {'name' => 'new2'}]
+      pairs.keys.each do |name| # cleanup
+        store.delete(name)
+      end
+    end
+  end
+
   describe '#delete' do
     it 'should delete an entry with the given key from the store' do
       store.write('new.json', {'name' => 'new'})

data/spec/processor/document_store/redis_store_spec.rb

@@ -6,6 +6,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
   end
 
   before :all do
+    store.clear
     %w(foo bar baz).each do |name|
       store.write("#{name}.json", {'name' => name})
     end
@@ -48,6 +49,38 @@ describe Pupa::Processor::DocumentStore::RedisStore do
     end
   end
 
+  describe '#write_unless_exists' do
+    it 'should write an entry with the given value for the given key' do
+      store.exist?('new.json').should == false
+      store.write_unless_exists('new.json', {'name' => 'new'}).should == true
+      store.read('new.json').should == {'name' => 'new'}
+      store.delete('new.json') # cleanup
+    end
+
+    it 'should not write an entry with the given value for the given key if the key exists' do
+      store.write_unless_exists('foo.json', {'name' => 'new'}).should == false
+      store.read('foo.json').should == {'name' => 'foo'}
+    end
+  end
+
+  describe '#write_multi' do
+    it 'should write entries with the given values for the given keys' do
+      pairs = {}
+      %w(new1 new2).each do |name|
+        pairs["#{name}.json"] = {'name' => name}
+      end
+
+      pairs.keys.each do |name|
+        store.exist?(name).should == false
+      end
+      store.write_multi(pairs)
+      store.read_multi(pairs.keys).should == [{'name' => 'new1'}, {'name' => 'new2'}]
+      pairs.keys.each do |name| # cleanup
+        store.delete(name)
+      end
+    end
+  end
+
   describe '#delete' do
     it 'should delete an entry with the given key from the store' do
       store.write('new.json', {'name' => 'new'})

data/spec/processor/document_store_spec.rb

@@ -8,7 +8,7 @@ describe Pupa::Processor::DocumentStore do
     end
 
     it 'should use Redis' do
-      Pupa::Processor::DocumentStore::RedisStore.should_receive(:new).with('redis://localhost').and_call_original
+      Pupa::Processor::DocumentStore::RedisStore.should_receive(:new).with('redis://localhost', {}).and_call_original
       Pupa::Processor::DocumentStore.new('redis://localhost')
     end
   end

data/spec/processor_spec.rb

@@ -15,7 +15,7 @@ describe Pupa::Processor do
     end
 
     def scrape_people
-      Fiber.yield(person)
+      dispatch(person)
     end
   end
 

data/spec/spec_helper.rb

@@ -4,6 +4,7 @@ require 'coveralls'
 Coveralls.wear!
 
 require 'multi_xml'
+require 'nokogiri'
 require 'redis-store'
 require 'rspec'
 require 'vcr'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupa
 version: !ruby/object:Gem::Version
-  version: 0.0.8
+  version: 0.0.9
 platform: ruby
 authors:
 - Open North
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-09-27 00:00:00.000000000 Z
+date: 2013-09-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -94,20 +94,6 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 1.5.1
-- !ruby/object:Gem::Dependency
-  name: nokogiri
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 1.6.0
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 1.6.0
 - !ruby/object:Gem::Dependency
   name: coveralls
   requirement: !ruby/object:Gem::Requirement
@@ -164,6 +150,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.0
 - !ruby/object:Gem::Dependency
   name: octokit
   requirement: !ruby/object:Gem::Requirement
@@ -220,6 +220,20 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '2.10'
+- !ruby/object:Gem::Dependency
+  name: typhoeus
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: vcr
   requirement: !ruby/object:Gem::Requirement
@@ -252,7 +266,6 @@ files:
 - lib/pupa.rb
 - lib/pupa/errors.rb
 - lib/pupa/logger.rb
-- lib/pupa/models/base.rb
 - lib/pupa/models/concerns/contactable.rb
 - lib/pupa/models/concerns/identifiable.rb
 - lib/pupa/models/concerns/linkable.rb
@@ -262,6 +275,7 @@ files:
 - lib/pupa/models/contact_detail_list.rb
 - lib/pupa/models/identifier_list.rb
 - lib/pupa/models/membership.rb
+- lib/pupa/models/model.rb
 - lib/pupa/models/organization.rb
 - lib/pupa/models/person.rb
 - lib/pupa/models/post.rb
@@ -274,6 +288,8 @@ files:
 - lib/pupa/processor/helper.rb
 - lib/pupa/processor/middleware/logger.rb
 - lib/pupa/processor/middleware/parse_html.rb
+- lib/pupa/processor/middleware/parse_json.rb
+- lib/pupa/processor/middleware/raise_error.rb
 - lib/pupa/processor/persistence.rb
 - lib/pupa/processor/yielder.rb
 - lib/pupa/refinements/faraday_middleware.rb