pupa 0.0.11 → 0.0.12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 88acbf7d013691871dbdaf6390f551625f6d5c4d
-  data.tar.gz: 5e52a9789851411377f3403196560648767fddcb
+  metadata.gz: c894758e0375a999ec78a825feb05ff7ec9da66a
+  data.tar.gz: 4f2633cc09888ef1f236cb2c35afdff9681e7924
 SHA512:
-  metadata.gz: b3f55a952c0cba3c80ed2dda6492144faf0b67b92583a847f08d301d8dba83a0acd22240ff856ef2a78d76f29984743482acfef5c49c4d3384afacf17d498dcf
-  data.tar.gz: afc4bdfad6dcd3702d79f5cdbebfecde163305ef8dd3cea6ac02332d8284584accf26fc5de5918e0a095de4438c554615619a87af590eac40fc5712d64d511f9
+  metadata.gz: 7e83f57b09cf99dab424032634193aa17055c22721b5ee558362f5fc07face6d6129b71d431e01f0b2b663aa771f752014ce7129e963d44d47beaa154f770c44
+  data.tar.gz: a51e3cee53727013f60623576af75b7c0ce7c047085be577a5f745fed04babc62ffd64a3c59abf234b909ffca47bc73250645f39eaae0364a89ad131f2593426

data/.travis.yml

@@ -1,6 +1,10 @@
 language: ruby
 rvm:
   - 2.0.0
+env:
+  - MODE=default
+  - MODE=compat
 services:
+  - memcached
   - mongodb
   - redis

data/README.md

@@ -7,6 +7,24 @@
 
 Pupa.rb is a Ruby 2.0 fork of Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa). It implements an Extract, Transform and Load (ETL) process to scrape data from online sources, transform it, and write it to a database.
 
+## What it tries to solve
+
+Pupa.rb's goal is to make scraping less painful by solving common problems:
+
+* If you are updating a database by scraping a website, you can either delete and recreate records, or you can merge the scraped records with the saved records. Pupa.rb offers a simple way to merge records, by using an object's stable properties for identification.
+* If you are scraping a source that references other sources – for example, a committee that references its members – you may want to link the source to its references with foreign keys. Pupa.rb will use whatever identifying information you scrape – for example, the members' names – to fill in the foreign keys for you.
+* Data sources may use different formats in different contexts. Pupa.rb makes it easy to [select scraping methods](https://github.com/opennorth/pupa-ruby#scraping-method-selection) according to criteria, like the year of publication for example.
+* By splitting the scrape (extract) and import (load) steps, it's easier for you and volunteers to start a scraper without any interaction with a database.
+
+In short, Pupa.rb lets you spend more time on the tasks that are unique to your use case, and less time on common tasks like caching, merging and storing data. It also provides helpful features like:
+
+* Logging, to make debugging and monitoring a scraper easier
+* [Automatic response parsing](https://github.com/opennorth/pupa-ruby#automatic-response-parsing) of JSON, XML and HTML
+* Option parsing, to control your scraper from the command-line
+* Object validation, using [JSON Schema](http://json-schema.org/)
+
+Pupa.rb is extensible, so that you can add your own models, parsers, helpers, actions, etc. It also offers several ways to [improve your scraper's performance](https://github.com/opennorth/pupa-ruby#performance).
+
 ## Usage
 
 You can use Pupa.rb to author scrapers that create people, organizations, memberships and posts according to the [Popolo](http://popoloproject.com/) open government data specification. If you need to scrape other types of data, you can also use your own models with Pupa.rb.
@@ -49,6 +67,18 @@ The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.htm
 
 JSON parsing is enabled by default. To enable automatic parsing of HTML and XML, require the `nokogiri` and `multi_xml` gems.
 
+### [OpenCivicData](http://opencivicdata.org/) compatibility
+
+Both Pupa.rb and Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa) implement models for people, organizations and memberships from the [Popolo](http://popoloproject.com/) open government data specification. Pupa.rb lets you use your own classes, but Pupa only supports a fixed set of classes. A consequence of Pupa.rb's flexibility is that the value of the `_type` property for `Person`, `Organization` and `Membership` objects differs between Pupa.rb and Pupa. Pupa.rb has namespaced types like `pupa/person` – to allow Ruby to load the `Person` class in the `Pupa` module – whereas Pupa has unnamespaced types like `person`.
+
+To save objects to MongoDB with unnamespaced types like Sunlight Labs' Pupa – in order to benefit from other tools in the [OpenCivicData](http://opencivicdata.org/) stack – add this line to the top of your script:
+
+```ruby
+require 'pupa/refinements/opencivicdata'
+```
+
+It is not currently possible to run the `scrape` action with one of Pupa.rb and Pupa, and to then run the `import` action with the other. Both actions must be run by the same library.
+
 ## Performance
 
 Pupa.rb offers several ways to significantly improve performance.
@@ -157,10 +187,6 @@ The `json-schema` gem is slow compared to, for example, [JSV](https://github.com
 
 The [pupa-validate](https://npmjs.org/package/pupa-validate) npm package can be used to validate JSON documents using the faster JSV. In an example case, using JSV instead of the `json-schema` gem reduced by half the time to validate 10,000 documents.
 
-### Parsing JSON
-
-If the rest of your scraper is fast, you may see an improvement by using the `oj` gem. Just `require 'oj'` and Pupa.rb will automatically pick it up, since it uses [MultiJson](https://github.com/intridea/multi_json).
-
 ### Profiling
 
 You can profile your code using [perftools.rb](https://github.com/tmm1/perftools.rb). First, install the gem:

data/lib/pupa.rb

@@ -1,5 +1,6 @@
 require 'fileutils'
 require 'forwardable'
+require 'json'
 
 require 'active_support/concern'
 require 'active_support/core_ext/class/attribute'
@@ -13,6 +14,7 @@ require 'pupa/logger'
 require 'pupa/processor'
 require 'pupa/runner'
 
+require 'pupa/models/concerns/indifferent_access'
 require 'pupa/models/concerns/contactable'
 require 'pupa/models/concerns/identifiable'
 require 'pupa/models/concerns/linkable'
@@ -20,6 +22,7 @@ require 'pupa/models/concerns/nameable'
 require 'pupa/models/concerns/sourceable'
 require 'pupa/models/concerns/timestamps'
 
+require 'pupa/models/foreign_object'
 require 'pupa/models/model'
 require 'pupa/models/contact_detail_list'
 require 'pupa/models/identifier_list'
@@ -33,3 +36,32 @@ module Pupa
     attr_accessor :session
   end
 end
+
+# ActiveSupport's String methods become bottlenecks once:
+#
+# - HTTP responses are cached in Memcached
+# - JSON documents are dumped to Redis
+# - Redis is pipelined
+# - Validation is skipped
+# - The runner is quiet
+#
+# With these optimizations, in sample scripts, garbage collection and gem
+# requiring take up two-thirds of the running time.
+class String
+  # Alternatively, check if `inflections.acronym_regex` is equal to `/(?=a)b/`.
+  # If so, to skip the substitution, which is guaranteed to fail.
+  #
+  # @see http://api.rubyonrails.org/classes/ActiveSupport/Inflector.html#method-i-underscore
+  def underscore
+    word = gsub('::', '/')
+    # word.gsub!(/(?:([A-Za-z\d])|^)(#{inflections.acronym_regex})(?=\b|[^a-z])/) { "#{$1}#{$1 && '_'}#{$2.downcase}" }
+    word.gsub!(/([A-Z\d]+)([A-Z][a-z])/,'\1_\2')
+    word.gsub!(/([a-z\d])([A-Z])/,'\1_\2')
+    word.tr!("-", "_")
+    word.downcase!
+    word
+  end
+
+  # @see http://api.rubyonrails.org/classes/String.html#method-i-blank-3F
+  alias_method :blank?, :empty?
+end

data/lib/pupa/errors.rb

@@ -3,6 +3,10 @@ module Pupa
     # An abstract class from which all Pupa errors inherit.
     class Error < StandardError; end
 
+    # This error is raised when loading a scraped object from disk if a type is
+    # not set.
+    class MissingObjectTypeError < Error; end
+
     # This error is raised when saving an object to a database if a foreign key
     # cannot be resolved.
     class MissingDatabaseIdError < Error; end

data/lib/pupa/models/concerns/indifferent_access.rb

@@ -0,0 +1,35 @@
+module Pupa
+  module Concerns
+    # Adds private methods for changing hash keys to strings or symbols.
+    module IndifferentAccess
+      extend ActiveSupport::Concern
+
+      private
+
+      def transform_keys(object, meth)
+        case object
+        when Hash
+          {}.tap do |hash|
+            object.each do |key,value|
+              hash[key.send(meth)] = transform_keys(value, meth)
+            end
+          end
+        when Array
+          object.map do |value|
+            transform_keys(value, meth)
+          end
+        else
+          object
+        end
+      end
+
+      def symbolize_keys(object)
+        transform_keys(object, :to_sym)
+      end
+
+      def stringify_keys(object)
+        transform_keys(object, :to_s)
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/timestamps.rb

@@ -16,6 +16,7 @@ module Pupa
 
         set_callback(:save, :before) do |object|
           # The object may not set created_at.
+          # @see https://github.com/opennorth/pupa-ruby/issues/17
           object.created_at = object.document['created_at'] if object.document
           object.updated_at = Time.now.utc
         end

data/lib/pupa/models/foreign_object.rb

@@ -0,0 +1,28 @@
+module Pupa
+  # A minimal model for a foreign object.
+  class ForeignObject
+    extend Forwardable
+    include Concerns::IndifferentAccess
+
+    attr_reader :attributes, :foreign_keys
+
+    def_delegators :@attributes, :[], :[]=
+
+    def initialize(properties = {})
+      hash = symbolize_keys(properties)
+      value = hash.delete(:foreign_keys) || {}
+      @attributes = hash.merge(value)
+      @foreign_keys = value.keys
+    end
+
+    def to_h
+      {}.tap do |hash|
+        attributes.each do |property,value|
+          if value == false || value.present?
+            hash[property] = value
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/model.rb

@@ -17,6 +17,8 @@ module Pupa
 
     included do
       include ActiveSupport::Callbacks
+      include Concerns::IndifferentAccess
+
       define_callbacks :create, :save
 
       class_attribute :json_schema
@@ -77,13 +79,14 @@ module Pupa
       # Sets the class' schema.
       #
       # @param [Hash,String] value a hash or a relative or absolute path
+      # @note `JSON::Validator#initialize_schema` runs fastest if given a hash.
       def schema=(value)
         self.json_schema = if Hash === value
           value
         elsif Pathname.new(value).absolute?
-          File.read(value)
+          JSON.load(File.read(value))
         else
-          File.read(File.expand_path(File.join('..', '..', '..', 'schemas', "#{value}.json"), __dir__))
+          JSON.load(File.read(File.expand_path(File.join('..', '..', '..', 'schemas', "#{value}.json"), __dir__)))
         end
       end
     end
@@ -166,7 +169,6 @@ module Pupa
     # @raises [JSON::Schema::ValidationError] if the object is invalid
     def validate!
       if self.class.json_schema
-        # JSON::Validator#initialize_schema runs fastest if given a hash.
         JSON::Validator.validate!(self.class.json_schema, stringify_keys(to_h(persist: true)))
       end
     end
@@ -200,32 +202,5 @@ module Pupa
       b.delete(:_id)
       a == b
     end
-
-  private
-
-    def transform_keys(object, meth)
-      case object
-      when Hash
-        {}.tap do |hash|
-          object.each do |key,value|
-            hash[key.send(meth)] = transform_keys(value, meth)
-          end
-        end
-      when Array
-        object.map do |value|
-          transform_keys(value, meth)
-        end
-      else
-        object
-      end
-    end
-
-    def symbolize_keys(object)
-      transform_keys(object, :to_sym)
-    end
-
-    def stringify_keys(object)
-      transform_keys(object, :to_s)
-    end
   end
 end

data/lib/pupa/processor.rb

@@ -151,7 +151,10 @@ module Pupa
           object = objects[id]
           resolve_foreign_keys(object, object_id_to_database_id)
           # The dependency graph strategy only works if there are no foreign objects.
-          object_id_to_database_id[id] = import_object(object)
+
+          database_id = import_object(object)
+          object_id_to_database_id[id] = database_id
+          object_id_to_database_id[database_id] = database_id
         end
       else
         size = objects.size
@@ -167,23 +170,16 @@ module Pupa
           progress_made = false
 
           objects.delete_if do |id,object|
-            resolvable = true
-
-            resolvable &= object.foreign_keys.all? do |property|
-              value = object[property]
-              value.nil? || object_id_to_database_id.key?(value)
-            end
-
-            resolvable &= object.foreign_objects.all? do |property|
-              selector = object[property]
-              selector.blank? || Persistence.find(selector)
-            end
-
-            if resolvable
-              progress_made = true
+            begin
               resolve_foreign_keys(object, object_id_to_database_id)
-              resolve_foreign_objects(object)
-              object_id_to_database_id[id] = import_object(object)
+              resolve_foreign_objects(object, object_id_to_database_id)
+              progress_made = true
+
+              database_id = import_object(object)
+              object_id_to_database_id[id] = database_id
+              object_id_to_database_id[database_id] = database_id
+            rescue Pupa::Errors::MissingDatabaseIdError
+              false
             end
           end
 
@@ -191,14 +187,16 @@ module Pupa
         end
 
         unless objects.empty?
-          raise Errors::UnprocessableEntity, "couldn't resolve #{objects.size}/#{size} objects:\n  #{objects.values.map{|object| MultiJson.dump(object.foreign_properties)}.join("\n  ")}"
+          raise Errors::UnprocessableEntity, "couldn't resolve #{objects.size}/#{size} objects:\n  #{objects.values.map{|object| JSON.dump(object.foreign_properties)}.join("\n  ")}"
         end
       end
 
       # Ensure that fingerprints uniquely identified objects.
       counts = {}
       object_id_to_database_id.each do |object_id,database_id|
-        (counts[database_id] ||= []) << object_id
+        unless object_id == database_id
+          (counts[database_id] ||= []) << object_id
+        end
       end
       duplicates = counts.select do |_,object_ids|
         object_ids.size > 1
@@ -251,13 +249,28 @@ module Pupa
     # @return [Hash] a hash of scraped objects keyed by ID
     def load_scraped_objects
       {}.tap do |objects|
-        @store.read_multi(@store.entries).each do |data|
-          object = data['_type'].camelize.constantize.new(data)
+        @store.read_multi(@store.entries).each do |properties|
+          object = load_scraped_object(properties)
           objects[object._id] = object
         end
       end
     end
 
+    # Loads a scraped object from its properties.
+    #
+    # @param [Hash] properties the object's properties
+    # @return [Object] a scraped object
+    # @raises [Pupa::Errors::MissingObjectTypeError] if the scraped object is
+    #   missing a `_type` property.
+    def load_scraped_object(properties)
+      type = properties['_type'] || properties[:_type]
+      if type
+        type.camelize.constantize.new(properties)
+      else
+        raise Errors::MissingObjectTypeError, "missing _type: #{JSON.dump(properties)}"
+      end
+    end
+
     # Removes all duplicate objects and re-assigns any foreign keys.
     #
     # @param [Hash] objects a hash of scraped objects keyed by ID
@@ -341,17 +354,17 @@ module Pupa
     #
     # @param [Object] an object
     # @param [Hash] a map from object ID to database ID
-    # @raises [Pupa::Errors::MissingDatabaseIdError]
+    # @raises [Pupa::Errors::MissingDatabaseIdError] if a foreign key cannot be
+    #   resolved
     def resolve_foreign_keys(object, map)
       object.foreign_keys.each do |property|
         value = object[property]
         if value
-          # If using a dependency graph, any foreign key that cannot be resolved
-          # will cause a key error while building the dependency graph.
-          #
-          # If not using a dependency graph, this method will not be called
-          # unless the foreign key is resolvable.
-          object[property] = map[value]
+          if map.key?(value)
+            object[property] = map[value]
+          else
+            raise Errors::MissingDatabaseIdError, "couldn't resolve foreign key: #{property} #{value}"
+          end
         end
       end
     end
@@ -359,13 +372,22 @@ module Pupa
     # Resolves an object's foreign objects to database IDs.
     #
     # @param [Object] object an object
-    # @raises [Pupa::Errors::MissingDatabaseIdError]
-    def resolve_foreign_objects(object)
+    # @param [Hash] a map from object ID to database ID
+    # @raises [Pupa::Errors::MissingDatabaseIdError] if a foreign object cannot
+    #   be resolved
+    def resolve_foreign_objects(object, map)
       object.foreign_objects.each do |property|
-        selector = object[property]
-        if selector.present?
-          # This method will not be called unless the foreign key is resolvable.
-          object["#{property}_id"] = Persistence.find(selector)['_id']
+        value = object[property]
+        if value.present?
+          foreign_object = ForeignObject.new(value)
+          resolve_foreign_keys(foreign_object, map)
+          document = Persistence.find(foreign_object.to_h)
+
+          if document
+            object["#{property}_id"] = document['_id']
+          else
+            raise Errors::MissingDatabaseIdError, "couldn't resolve foreign object: #{property} #{value}"
+          end
         end
       end
     end

data/lib/pupa/processor/document_store/file_store.rb

@@ -16,7 +16,7 @@ module Pupa
         # @param [String] name a key
         # @return [Boolean] whether the store contains an entry for the given key
         def exist?(name)
-          File.exist?(namespaced_key(name))
+          File.exist?(path(name))
         end
 
         # Returns all file names in the storage directory.
@@ -33,8 +33,8 @@ module Pupa
         # @param [String] name a key
         # @return [Hash] the value of the given key
         def read(name)
-          File.open(namespaced_key(name)) do |f|
-            MultiJson.load(f)
+          File.open(path(name)) do |f|
+            Oj.load(f)
           end
         end
 
@@ -53,8 +53,8 @@ module Pupa
         # @param [String] name a key
         # @param [Hash] value a value
         def write(name, value)
-          File.open(namespaced_key(name), 'w') do |f|
-            f.write(MultiJson.dump(value))
+          File.open(path(name), 'w') do |f|
+            f.write(Oj.dump(value, mode: :compat, time_format: :ruby))
           end
         end
 
@@ -83,7 +83,7 @@ module Pupa
         #
         # @param [String] name a key
         def delete(name)
-          File.delete(namespaced_key(name))
+          File.delete(path(name))
         end
 
         # Deletes all files in the storage directory.
@@ -98,9 +98,11 @@ module Pupa
           yield
         end
 
-      private
-
-        def namespaced_key(name)
+        # Returns the path to the file with the given name.
+        #
+        # @param [String] name a key
+        # @param [String] a path
+        def path(name)
           File.join(@output_dir, name)
         end
       end

data/lib/pupa/processor/document_store/redis_store.rb

@@ -42,7 +42,7 @@ module Pupa
         # @param [String] name a key
         # @return [Hash] the value of the given key
         def read(name)
-          MultiJson.load(@redis.get(name))
+          Oj.load(@redis.get(name))
         end
 
         # Returns, as JSON, the values of the given keys.
@@ -50,7 +50,7 @@ module Pupa
         # @param [String] names keys
         # @return [Array<Hash>] the values of the given keys
         def read_multi(names)
-          @redis.mget(*names).map{|value| MultiJson.load(value)}
+          @redis.mget(*names).map{|value| Oj.load(value)}
         end
 
         # Writes, as JSON, the value to a key.
@@ -58,7 +58,7 @@ module Pupa
         # @param [String] name a key
         # @param [Hash] value a value
         def write(name, value)
-          @redis.set(name, MultiJson.dump(value))
+          @redis.set(name, Oj.dump(value, mode: :compat, time_format: :ruby))
         end
 
         # Writes, as JSON, the value to a key, unless the key exists.
@@ -67,7 +67,7 @@ module Pupa
         # @param [Hash] value a value
         # @return [Boolean] whether the key was set
         def write_unless_exists(name, value)
-          @redis.setnx(name, MultiJson.dump(value))
+          @redis.setnx(name, Oj.dump(value, mode: :compat, time_format: :ruby))
         end
 
         # Writes, as JSON, the values to keys.
@@ -77,7 +77,7 @@ module Pupa
           args = []
           pairs.each do |key,value|
             args << key
-            args << MultiJson.dump(value)
+            args << Oj.dump(value, mode: :compat, time_format: :ruby)
           end
           @redis.mset(*args)
         end

data/lib/pupa/processor/middleware/parse_json.rb

@@ -5,10 +5,10 @@ module Pupa
       #
       # @see https://github.com/lostisland/faraday_middleware/issues/30#issuecomment-4706892
       class ParseJson < FaradayMiddleware::ResponseMiddleware
-        dependency 'multi_json'
+        dependency 'oj'
 
         define_parser do |body|
-          MultiJson.load(body) unless body.strip.empty?
+          Oj.load(body) unless body.strip.empty?
         end
       end
     end

data/lib/pupa/processor/persistence.rb

@@ -24,7 +24,7 @@ module Pupa
         when 1
           query.first
         else
-          raise Errors::TooManyMatches, "selector matches multiple documents during find: #{collection_name} #{MultiJson.dump(selector)}"
+          raise Errors::TooManyMatches, "selector matches multiple documents during find: #{collection_name} #{JSON.dump(selector)}"
         end
       end
 
@@ -47,13 +47,14 @@ module Pupa
           end
         when 1
           # Make the document available to the callbacks.
+          # @see https://github.com/opennorth/pupa-ruby/issues/17
           @object.document = query.first
           @object.run_callbacks(:save) do
             query.update(@object.to_h(persist: true).except(:_id))
             [false, @object.document['_id'].to_s]
           end
         else
-          raise Errors::TooManyMatches, "selector matches multiple documents during save: #{collection_name} #{MultiJson.dump(selector)}"
+          raise Errors::TooManyMatches, "selector matches multiple documents during save: #{collection_name} #{JSON.dump(selector)} for #{@object._id}"
         end
       end
 

data/lib/pupa/refinements/faraday_middleware.rb

@@ -20,6 +20,6 @@ class FaradayMiddleware::Caching
       url.query = build_query params
     end
     url.normalize!
-    url.request_uri + env[:body].to_s # Add for POST requests.
+    url.request_uri + env[:body].to_s # XXX add for POST requests
   end
 end

data/lib/pupa/refinements/json-schema.rb

@@ -2,9 +2,8 @@ require 'mail'
 
 module Pupa
   module Refinements
-    # A refinement for JSON Schema to validate "email" and "uri" formats. Using
-    # Ruby's refinements doesn't seem to work, possibly because `refine` can't
-    # be used with `prepend`.
+    # Validates "email" and "uri" formats. Using Ruby's refinements doesn't seem
+    # to work, possibly because `refine` can't be used with `prepend`.
     module FormatAttribute
       # @see http://my.rails-royce.org/2010/07/21/email-validation-in-ruby-on-rails-without-regexp/
       def validate(current_schema, data, fragments, processor, validator, options = {})

data/lib/pupa/refinements/opencivicdata.rb

@@ -0,0 +1,42 @@
+# @see https://github.com/opennorth/pupa-ruby#opencivicdata-compatibility
+
+module Pupa::Model
+  # This unfortunately won't cause the behavior of any model that has already
+  # included `Pupa::Model` to change.
+  class << self
+    def append_features(base)
+      if base.instance_variable_defined?("@_dependencies")
+        base.instance_variable_get("@_dependencies") << self
+        return false
+      else
+        return false if base < self
+        @_dependencies.each { |dep| base.send(:include, dep) }
+        super
+        base.extend const_get("ClassMethods") if const_defined?("ClassMethods")
+        base.class_eval(&@_included_block) if instance_variable_defined?("@_included_block")
+        base.class_eval do # XXX
+          set_callback(:save, :before) do |object|
+            object._type = object._type.camelize.demodulize.underscore
+          end
+        end
+      end
+    end
+  end
+end
+
+# `set_callback` is called by `class_eval` in `ActiveSupport::Concern`. Without
+# monkey-patching `ActiveSupport::Concern`, we can either iterate `ObjectSpace`,
+# implement something like ActiveSupport's `DescendantsTracker` for inclusion
+# instead of inheritance, or go back to `Pupa::Model` being a superclass instead
+# of a mixin to take advantage of `DescendantsTracker` itself.
+#
+# Instead of adding a callback, we can override `to_h` when `persist` is `true`.
+ObjectSpace.each_object(Class) do |base|
+  if base.include?(Pupa::Model)
+    base.class_eval do
+      set_callback(:save, :before) do |object|
+        object._type = object._type.camelize.demodulize.underscore
+      end
+    end
+  end
+end

data/lib/pupa/runner.rb

@@ -204,7 +204,7 @@ module Pupa
       if %w(DEBUG INFO).include?(options.level)
         report[:end] = Time.now.utc
         report[:time] = report[:end] - report[:start]
-        puts MultiJson.dump(report)
+        puts JSON.dump(report)
       end
     end
   end

data/lib/pupa/version.rb

@@ -1,3 +1,3 @@
 module Pupa
-  VERSION = "0.0.11"
+  VERSION = "0.0.12"
 end

data/pupa.gemspec

@@ -22,6 +22,7 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency('json-schema', '~> 2.1.3')
   s.add_runtime_dependency('mail')
   s.add_runtime_dependency('moped', '~> 1.5.1')
+  s.add_runtime_dependency('oj', '~> 2.1')
 
   s.add_development_dependency('coveralls')
   s.add_development_dependency('dalli')

data/spec/models/model_spec.rb

@@ -91,12 +91,12 @@ describe Pupa::Model do
 
     it 'should accept an absolute path' do
       File.should_receive(:read).and_return('{}')
-      klass_with_absolute_path.json_schema.should == '{}'
+      klass_with_absolute_path.json_schema.should == {}
     end
 
     it 'should accept a relative path' do
       File.should_receive(:read).and_return('{}')
-      klass_with_relative_path.json_schema.should == '{}'
+      klass_with_relative_path.json_schema.should == {}
     end
   end
 

data/spec/processor/document_store/file_store_spec.rb

@@ -36,7 +36,7 @@ describe Pupa::Processor::DocumentStore::FileStore do
   describe '#write' do
     it 'should write an entry with the given value for the given key' do
       store.exist?('new.json').should == false
-      store.write('new.json', {'name' => 'new'})
+      store.write('new.json', {name: 'new'})
       store.read('new.json').should == {'name' => 'new'}
       store.delete('new.json') # cleanup
     end
@@ -45,13 +45,13 @@ describe Pupa::Processor::DocumentStore::FileStore do
   describe '#write_unless_exists' do
     it 'should write an entry with the given value for the given key' do
       store.exist?('new.json').should == false
-      store.write_unless_exists('new.json', {'name' => 'new'}).should == true
+      store.write_unless_exists('new.json', {name: 'new'}).should == true
       store.read('new.json').should == {'name' => 'new'}
       store.delete('new.json') # cleanup
     end
 
     it 'should not write an entry with the given value for the given key if the key exists' do
-      store.write_unless_exists('foo.json', {'name' => 'new'}).should == false
+      store.write_unless_exists('foo.json', {name: 'new'}).should == false
       store.read('foo.json').should == {'name' => 'foo'}
     end
   end
@@ -60,7 +60,7 @@ describe Pupa::Processor::DocumentStore::FileStore do
     it 'should write entries with the given values for the given keys' do
       pairs = {}
       %w(new1 new2).each do |name|
-        pairs["#{name}.json"] = {'name' => name}
+        pairs["#{name}.json"] = {name: name}
       end
 
       pairs.keys.each do |name|
@@ -76,7 +76,7 @@ describe Pupa::Processor::DocumentStore::FileStore do
 
   describe '#delete' do
     it 'should delete an entry with the given key from the store' do
-      store.write('new.json', {'name' => 'new'})
+      store.write('new.json', {name: 'new'})
       store.exist?('new.json').should == true
       store.delete('new.json')
       store.exist?('new.json').should == false
@@ -90,8 +90,14 @@ describe Pupa::Processor::DocumentStore::FileStore do
       store.entries.should == []
 
       %w(bar baz foo).each do |name| # cleanup
-        store.write("#{name}.json", {'name' => name})
+        store.write("#{name}.json", {name: name})
       end
     end
   end
+
+  describe '#path' do
+    it 'should return the file path to the entry' do
+      store.path('foo').should == File.expand_path(File.join('..', '..', 'fixtures', 'foo'), __dir__)
+    end
+  end
 end

data/spec/processor/document_store/redis_store_spec.rb

@@ -8,7 +8,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
   before :all do
     store.clear
     %w(foo bar baz).each do |name|
-      store.write("#{name}.json", {'name' => name})
+      store.write("#{name}.json", {name: name})
     end
   end
 
@@ -43,7 +43,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
   describe '#write' do
     it 'should write an entry with the given value for the given key' do
       store.exist?('new.json').should == false
-      store.write('new.json', {'name' => 'new'})
+      store.write('new.json', {name: 'new'})
       store.read('new.json').should == {'name' => 'new'}
       store.delete('new.json') # cleanup
     end
@@ -52,13 +52,13 @@ describe Pupa::Processor::DocumentStore::RedisStore do
   describe '#write_unless_exists' do
     it 'should write an entry with the given value for the given key' do
       store.exist?('new.json').should == false
-      store.write_unless_exists('new.json', {'name' => 'new'}).should == true
+      store.write_unless_exists('new.json', {name: 'new'}).should == true
       store.read('new.json').should == {'name' => 'new'}
       store.delete('new.json') # cleanup
     end
 
     it 'should not write an entry with the given value for the given key if the key exists' do
-      store.write_unless_exists('foo.json', {'name' => 'new'}).should == false
+      store.write_unless_exists('foo.json', {name: 'new'}).should == false
       store.read('foo.json').should == {'name' => 'foo'}
     end
   end
@@ -67,7 +67,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
     it 'should write entries with the given values for the given keys' do
       pairs = {}
       %w(new1 new2).each do |name|
-        pairs["#{name}.json"] = {'name' => name}
+        pairs["#{name}.json"] = {name: name}
       end
 
       pairs.keys.each do |name|
@@ -83,7 +83,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
 
   describe '#delete' do
     it 'should delete an entry with the given key from the store' do
-      store.write('new.json', {'name' => 'new'})
+      store.write('new.json', {name: 'new'})
       store.exist?('new.json').should == true
       store.delete('new.json')
       store.exist?('new.json').should == false
@@ -97,7 +97,7 @@ describe Pupa::Processor::DocumentStore::RedisStore do
       store.entries.should == []
 
       %w(bar baz foo).each do |name| # cleanup
-        store.write("#{name}.json", {'name' => name})
+        store.write("#{name}.json", {name: name})
       end
     end
   end

data/spec/processor/persistence_spec.rb

@@ -1,6 +1,14 @@
 require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
 
 describe Pupa::Processor::Persistence do
+  def _type
+    if testing_python_compatibility?
+      'person'
+    else
+      'pupa/person'
+    end
+  end
+
   before :all do
     Pupa.session = Moped::Session.new(['localhost:27017'], database: 'pupa_test')
     Pupa.session.collections.each(&:drop)
@@ -13,11 +21,11 @@ describe Pupa::Processor::Persistence do
 
   describe '.find' do
     it 'should return nil if no matches' do
-      Pupa::Processor::Persistence.find(_type: 'pupa/person', name: 'nonexistent').should == nil
+      Pupa::Processor::Persistence.find(_type: _type, name: 'nonexistent').should == nil
     end
 
     it 'should return a document if one match' do
-      Pupa::Processor::Persistence.find(_type: 'pupa/person', name: 'existing').should be_a(Hash)
+      Pupa::Processor::Persistence.find(_type: _type, name: 'existing').should be_a(Hash)
     end
 
     it 'should raise an error if many matches' do
@@ -28,12 +36,12 @@ describe Pupa::Processor::Persistence do
   describe '#save' do
     it 'should insert a document if no matches' do
       Pupa::Processor::Persistence.new(Pupa::Person.new(_id: 'new', name: 'new', email: 'new@example.com')).save.should == [true, 'new']
-      Pupa::Processor::Persistence.find(_type: 'pupa/person', name: 'new')['email'].should == 'new@example.com'
+      Pupa::Processor::Persistence.find(_type: _type, name: 'new')['email'].should == 'new@example.com'
     end
 
     it 'should update a document if one match' do
       Pupa::Processor::Persistence.new(Pupa::Person.new(_id: 'changed', name: 'existing', email: 'changed@example.com')).save.should == [false, 'existing']
-      Pupa::Processor::Persistence.find(_type: 'pupa/person', name: 'existing')['email'].should == 'changed@example.com'
+      Pupa::Processor::Persistence.find(_type: _type, name: 'existing')['email'].should == 'changed@example.com'
     end
 
     it 'should raise an error if many matches' do

data/spec/processor_spec.rb

@@ -102,6 +102,14 @@ describe Pupa::Processor do
       Pupa.session.collections.each(&:drop)
     end
 
+    let :_type do
+      if testing_python_compatibility?
+        'organization'
+      else
+        'pupa/organization'
+      end
+    end
+
     let :graphable do
       {
         '1' => Pupa::Organization.new({
@@ -125,7 +133,7 @@ describe Pupa::Processor do
         '4' => Pupa::Organization.new({
           _id: '4',
           name: 'Child',
-          parent: {_type: 'pupa/organization', name: 'Parent'},
+          parent: {_type: _type, name: 'Parent'},
         }),
         '5' => Pupa::Organization.new({
           _id: '5',
@@ -138,6 +146,25 @@ describe Pupa::Processor do
       }
     end
 
+    let :foreign_keys_on_foreign_objects do
+      {
+        '7' => Pupa::Organization.new({
+          _id: '7',
+          name: 'Child',
+          parent: {_type: _type, name: 'Parent'},
+        }),
+        '8' => Pupa::Organization.new({
+          _id: '8',
+          name: 'Grandchild',
+          parent: {_type: _type, foreign_keys: {parent_id: '9'}}
+        }),
+        '9' => Pupa::Organization.new({
+          _id: '9',
+          name: 'Parent',
+        }),
+      }
+    end
+
     it 'should use a dependency graph if possible' do
       processor.should_receive(:load_scraped_objects).and_return(graphable)
 
@@ -145,6 +172,16 @@ describe Pupa::Processor do
       processor.import
     end
 
+    it 'should remove duplicate objects and re-assign foreign keys' do
+      processor.should_receive(:load_scraped_objects).and_return(graphable)
+
+      processor.import
+      documents = Pupa.session[:organizations].find.entries
+      documents.size.should == 2
+      documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '2', '_type' => _type, 'name' => 'Parent'}
+      documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '1', '_type' => _type, 'name' => 'Child', 'parent_id' => '2'}
+    end
+
     it 'should not use a dependency graph if not possible' do
       processor.should_receive(:load_scraped_objects).and_return(ungraphable)
 
@@ -152,24 +189,25 @@ describe Pupa::Processor do
       processor.import
     end
 
-    it 'should remove duplicate objects and re-assign foreign keys' do
-      processor.should_receive(:load_scraped_objects).and_return(graphable)
+    it 'should remove duplicate objects and resolve foreign objects' do
+      processor.should_receive(:load_scraped_objects).and_return(ungraphable)
 
       processor.import
       documents = Pupa.session[:organizations].find.entries
       documents.size.should == 2
-      documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '2', '_type' => 'pupa/organization', 'name' => 'Parent'}
-      documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '1', '_type' => 'pupa/organization', 'name' => 'Child', 'parent_id' => '2'}
+      documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '5', '_type' => _type, 'name' => 'Parent'}
+      documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '4', '_type' => _type, 'name' => 'Child', 'parent_id' => '5'}
     end
 
-    it 'should resolve foreign objects' do
-      processor.should_receive(:load_scraped_objects).and_return(ungraphable)
+    it 'should resolve foreign keys on foreign objects' do
+      processor.should_receive(:load_scraped_objects).and_return(foreign_keys_on_foreign_objects)
 
       processor.import
       documents = Pupa.session[:organizations].find.entries
-      documents.size.should == 2
-      documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '5', '_type' => 'pupa/organization', 'name' => 'Parent'}
-      documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '4', '_type' => 'pupa/organization', 'name' => 'Child', 'parent_id' => '5'}
+      documents.size.should == 3
+      documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '9', '_type' => _type, 'name' => 'Parent'}
+      documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '7', '_type' => _type, 'name' => 'Child', 'parent_id' => '9'}
+      documents[2].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '8', '_type' => _type, 'name' => 'Grandchild', 'parent_id' => '7'}
     end
 
     context 'with existing documents' do
@@ -196,12 +234,13 @@ describe Pupa::Processor do
         }
       end
 
+      # Use a foreign object to not use a dependency graph.
       let :unresolvable_foreign_key do
         {
           'a' => Pupa::Organization.new({
             _id: 'a',
             name: 'Child',
-            parent: {_type: 'pupa/organization', name: 'Parent'},
+            parent: {_type: _type, name: 'Parent'},
           }),
           'b' => Pupa::Organization.new({
             _id: 'b',
@@ -220,7 +259,7 @@ describe Pupa::Processor do
           'a' => Pupa::Organization.new({
             _id: 'a',
             name: 'Child',
-            parent: {_type: 'pupa/organization', name: 'Nonexistent'},
+            parent: {_type: _type, name: 'Nonexistent'},
           }),
           'b' => Pupa::Organization.new({
             _id: 'b',
@@ -239,7 +278,7 @@ describe Pupa::Processor do
           'a' => Pupa::Organization.new({
             _id: 'a',
             name: 'Child',
-            parent: {_type: 'pupa/organization', name: 'Parent'},
+            parent: {_type: _type, name: 'Parent'},
           }),
           'b' => Pupa::Organization.new({
             _id: 'b',
@@ -253,14 +292,33 @@ describe Pupa::Processor do
         }
       end
 
+      let :resolvable_foreign_keys_on_foreign_objects do
+        {
+          'a' => Pupa::Organization.new({
+            _id: 'a',
+            name: 'Child',
+            parent: {_type: _type, name: 'Parent'},
+          }),
+          'b' => Pupa::Organization.new({
+            _id: 'b',
+            name: 'Grandchild',
+            parent: {_type: _type, foreign_keys: {parent_id: 'c'}}
+          }),
+          'c' => Pupa::Organization.new({
+            _id: 'c',
+            name: 'Parent',
+          }),
+        }
+      end
+
       it 'should resolve foreign keys' do
         processor.should_receive(:load_scraped_objects).and_return(resolvable_foreign_key)
 
         processor.import
         documents = Pupa.session[:organizations].find.entries
         documents.size.should == 2
-        documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '2', '_type' => 'pupa/organization', 'name' => 'Parent'}
-        documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '1', '_type' => 'pupa/organization', 'name' => 'Child', 'parent_id' => '2'}
+        documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '2', '_type' => _type, 'name' => 'Parent'}
+        documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '1', '_type' => _type, 'name' => 'Child', 'parent_id' => '2'}
       end
 
       it 'should raise an error if a foreign key cannot be resolved' do
@@ -277,6 +335,17 @@ describe Pupa::Processor do
         processor.should_receive(:load_scraped_objects).and_return(duplicate_documents)
         expect{processor.import}.to raise_error(Pupa::Errors::DuplicateDocumentError)
       end
+
+      it 'should resolve foreign keys on foreign objects' do
+        processor.should_receive(:load_scraped_objects).and_return(resolvable_foreign_keys_on_foreign_objects)
+
+        processor.import
+        documents = Pupa.session[:organizations].find.entries
+        documents.size.should == 3
+        documents[0].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '2', '_type' => _type, 'name' => 'Parent'}
+        documents[1].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => '1', '_type' => _type, 'name' => 'Child', 'parent_id' => '2'}
+        documents[2].slice('_id', '_type', 'name', 'parent_id').should == {'_id' => 'b', '_type' => _type, 'name' => 'Grandchild', 'parent_id' => '1'}
+      end
     end
   end
 end

data/spec/refinements/opencivicdata_spec.rb

@@ -0,0 +1,35 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::Refinements, testing_python_compatibility: true do
+  module Music
+    class Band
+      include Pupa::Model
+
+      def save
+        run_callbacks(:save) do
+        end
+      end
+    end
+  end
+
+  module Pupa
+    class Committee < Organization
+      def save
+        run_callbacks(:save) do
+        end
+      end
+    end
+  end
+
+  it 'should demodulize the type of new models' do
+    object = Music::Band.new
+    object.save
+    object._type.should == 'band'
+  end
+
+  it 'should demodulize the type of existing models' do
+    object = Pupa::Committee.new
+    object.save
+    object._type.should == 'committee'
+  end
+end

data/spec/spec_helper.rb

@@ -18,3 +18,15 @@ VCR.configure do |c|
     VCR.use_cassette(Digest::SHA1.hexdigest(request.uri + request.body + request.headers.to_s), &request)
   end
 end
+
+def testing_python_compatibility?
+  ENV['MODE'] == 'compat'
+end
+
+if testing_python_compatibility?
+  require File.dirname(__FILE__) + '/../lib/pupa/refinements/opencivicdata'
+end
+
+RSpec.configure do |c|
+  c.filter_run_excluding :testing_python_compatibility => true unless testing_python_compatibility?
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupa
 version: !ruby/object:Gem::Version
-  version: 0.0.11
+  version: 0.0.12
 platform: ruby
 authors:
 - Open North
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-07 00:00:00.000000000 Z
+date: 2013-12-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -94,6 +94,20 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 1.5.1
+- !ruby/object:Gem::Dependency
+  name: oj
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.1'
 - !ruby/object:Gem::Dependency
   name: coveralls
   requirement: !ruby/object:Gem::Requirement
@@ -268,11 +282,13 @@ files:
 - lib/pupa/logger.rb
 - lib/pupa/models/concerns/contactable.rb
 - lib/pupa/models/concerns/identifiable.rb
+- lib/pupa/models/concerns/indifferent_access.rb
 - lib/pupa/models/concerns/linkable.rb
 - lib/pupa/models/concerns/nameable.rb
 - lib/pupa/models/concerns/sourceable.rb
 - lib/pupa/models/concerns/timestamps.rb
 - lib/pupa/models/contact_detail_list.rb
+- lib/pupa/models/foreign_object.rb
 - lib/pupa/models/identifier_list.rb
 - lib/pupa/models/membership.rb
 - lib/pupa/models/model.rb
@@ -294,6 +310,7 @@ files:
 - lib/pupa/processor/yielder.rb
 - lib/pupa/refinements/faraday_middleware.rb
 - lib/pupa/refinements/json-schema.rb
+- lib/pupa/refinements/opencivicdata.rb
 - lib/pupa/runner.rb
 - lib/pupa/version.rb
 - pupa.gemspec
@@ -340,6 +357,7 @@ files:
 - spec/processor/persistence_spec.rb
 - spec/processor/yielder_spec.rb
 - spec/processor_spec.rb
+- spec/refinements/opencivicdata_spec.rb
 - spec/runner_spec.rb
 - spec/spec_helper.rb
 homepage: http://github.com/opennorth/pupa-ruby
@@ -402,5 +420,6 @@ test_files:
 - spec/processor/persistence_spec.rb
 - spec/processor/yielder_spec.rb
 - spec/processor_spec.rb
+- spec/refinements/opencivicdata_spec.rb
 - spec/runner_spec.rb
 - spec/spec_helper.rb