pupa 0.1.7 → 0.1.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 76ad0723fbc3d2027fbb020536b0afe3a0bc148e
-  data.tar.gz: f7b89d4988aa3b244dffc777860028d12a427aa5
+  metadata.gz: f3539ca0b77fc4b2174c42857990638713dac180
+  data.tar.gz: c5a18a0390ac9a3a48e0ba868ac1dd627af0db08
 SHA512:
-  metadata.gz: f9d15fc0fb4ce9f25217bef3df9843375d353e16b95fa2f051a61026fd2a70fc659bf59fec5bf3d2b851c0eb7e3e476c28a271213b4f0e4cbfe321ada1632d1a
-  data.tar.gz: d2ad9277a81eaaae9c64f847608116d3164c499cbf8a06db8acb0fd0400743522ca1061e32deda9b6ac02e6888e515aac6b643d28e4efcb2bb40ea9e2e2f47d4
+  metadata.gz: c93da2b8ff26107c28e02e51b8ea2ec999b5e086cc8564e0af6062a58af7fe3b71429e20d26eefc57471313da578699218a4098cd35ca94eed831340f63beef1
+  data.tar.gz: 2444a4491a3c0b499f4dd40c4b188a72644fc4fddec941471816747bb2e694bd5b96c7d550fe4dd98ed6c477b358ade25b25cf17c3a329ca8733929f6041252c

data/Gemfile

@@ -2,3 +2,5 @@ source "http://rubygems.org"
 
 # Specify your gem's dependencies in the gemspec
 gemspec
+
+gem 'faraday_middleware', git: 'https://github.com/lostisland/faraday_middleware.git'

data/PERFORMANCE.md

@@ -1,6 +1,4 @@
-# Pupa.rb: A Data Scraping Framework
-
-## Performance
+# Pupa.rb: Performance
 
 Pupa.rb offers several ways to significantly improve performance.
 
@@ -8,13 +6,13 @@ In an example case, reducing disk I/O and skipping validation as described below
 
 The `import` action's performance is currently limited by the database when a dependency graph is used to determine the evaluation order. If a dependency graph cannot be used because you don't know a related object's ID, [several optimizations](https://github.com/opennorth/pupa-ruby/issues/12) can be implemented to improve performance.
 
-### Reducing HTTP requests
+## Reducing HTTP requests
 
 HTTP requests consume the most time. To avoid repeat HTTP requests while developing a scraper, cache all HTTP responses. Pupa.rb will by default use a `web_cache` directory in the same directory as your script. You can change the directory by setting the `--cache_dir` switch on the command line, for example:
 
     ruby cat.rb --cache_dir /tmp/my_cache_dir
 
-### Parallelizing HTTP requests
+## Parallelizing HTTP requests
 
 To enable parallel requests, use the `typhoeus` gem. Unless you are using an old version of Typhoeus (< 0.5), both Faraday and Typhoeus define a Faraday adapter, but you must use the one defined by Typhoeus, like so:
 
@@ -58,11 +56,11 @@ responses.each do |response|
 end
 ```
 
-### Reducing disk I/O
+## Reducing disk I/O
 
 After HTTP requests, disk I/O is the slowest operation. Two types of files are written to disk: HTTP responses are written to the cache directory, and JSON documents are written to the output directory. Writing to memory is much faster than writing to disk.
 
-#### RAM file systems
+### RAM file systems
 
 A simple solution is to create a file system in RAM, like `tmpfs` on Linux for example, and to use it as your `output_dir` and  `cache_dir`. On OS X, you must create a RAM disk. To create a 128MB RAM disk, for example, run:
 
@@ -80,7 +78,7 @@ Once you are done with the RAM disk, release the memory:
 
 Using a RAM disk will significantly improve performance; however, the data will be lost between reboots unless you move the data to a hard disk. Using Memcached (for caching) and Redis (for storage) is moderately faster than using a RAM disk, and Redis will not lose your output data between reboots.
 
-#### Memcached
+### Memcached
 
 You may cache HTTP responses in [Memcached](http://memcached.org/). First, require the `dalli` gem. Then:
 
@@ -88,7 +86,7 @@ You may cache HTTP responses in [Memcached](http://memcached.org/). First, requi
 
 The data in Memcached will be lost between reboots.
 
-#### Redis
+### Redis
 
 You may dump JSON documents in [Redis](http://redis.io/). First, require the `redis-store` gem. Then:
 
@@ -102,17 +100,17 @@ Requiring the `hiredis` gem will slightly improve performance.
 
 Note that Pupa.rb flushes the Redis database before scraping. If you use Redis, **DO NOT** share a Redis database with Pupa.rb and other applications. You can select a different database than the default `0` for use with Pupa.rb by passing an argument like `redis://localhost:6379/15`, where `15` is the database number.
 
-### Skipping validation
+## Skipping validation
 
 The `json-schema` gem is slow compared to, for example, [JSV](https://github.com/garycourt/JSV). Setting the `--no-validate` switch and running JSON Schema validations separately can further reduce a scraper's running time.
 
 The [pupa-validate](https://npmjs.org/package/pupa-validate) npm package can be used to validate JSON documents using the faster JSV. In an example case, using JSV instead of the `json-schema` gem reduced by half the time to validate 10,000 documents.
 
-### Ruby version
+## Ruby version
 
 Pupa.rb requires Ruby 2.x. If you have already made all the above optimizations, you may notice a significant improvement by using Ruby 2.1, which has better garbage collection than Ruby 2.0.
 
-### Profiling
+## Profiling
 
 You can profile your code using [perftools.rb](https://github.com/tmm1/perftools.rb). First, install the gem:
 

data/README.md

@@ -1,5 +1,6 @@
 # Pupa.rb: A Data Scraping Framework
 
+[![Gem Version](https://badge.fury.io/rb/pupa.svg)](http://badge.fury.io/rb/pupa)
 [![Build Status](https://secure.travis-ci.org/opennorth/pupa-ruby.png)](http://travis-ci.org/opennorth/pupa-ruby)
 [![Dependency Status](https://gemnasium.com/opennorth/pupa-ruby.png)](https://gemnasium.com/opennorth/pupa-ruby)
 [![Coverage Status](https://coveralls.io/repos/opennorth/pupa-ruby/badge.png?branch=master)](https://coveralls.io/r/opennorth/pupa-ruby)
@@ -9,27 +10,9 @@ Pupa.rb is a Ruby 2.x fork of Sunlight Labs' [Pupa](https://github.com/opencivic
 
     gem install pupa
 
-## What it tries to solve
-
-Pupa.rb's goal is to make scraping less painful by solving common problems:
-
-* If you are updating a database by scraping a website, you can either delete and recreate records, or you can merge the scraped records with the saved records. Pupa.rb offers a simple way to merge records, by using an object's stable properties for identification.
-* If you are scraping a source that references other sources – for example, a committee that references its members – you may want to link the source to its references with foreign keys. Pupa.rb will use whatever identifying information you scrape – for example, the members' names – to fill in the foreign keys for you.
-* Data sources may use different formats in different contexts. Pupa.rb makes it easy to [select scraping methods](https://github.com/opennorth/pupa-ruby#scraping-method-selection) according to criteria, like the year of publication for example.
-* By splitting the scrape (extract) and import (load) steps, it's easier for you and volunteers to start a scraper without any interaction with a database.
-
-In short, Pupa.rb lets you spend more time on the tasks that are unique to your use case, and less time on common tasks like caching, merging and storing data. It also provides helpful features like:
-
-* Logging, to make debugging and monitoring a scraper easier
-* [Automatic response parsing](https://github.com/opennorth/pupa-ruby#automatic-response-parsing) of JSON, XML and HTML
-* Option parsing, to control your scraper from the command-line
-* Object validation, using [JSON Schema](http://json-schema.org/)
-
-Pupa.rb is extensible, so that you can add your own models, parsers, helpers, actions, etc. It also offers several ways to [improve your scraper's performance](https://github.com/opennorth/pupa-ruby#performance).
-
 ## Usage
 
-You can use Pupa.rb to author scrapers that create people, organizations, memberships and posts according to the [Popolo](http://popoloproject.com/) open government data specification. If you need to scrape other types of data, you can also use your own models with Pupa.rb.
+You can scrape any sort of data with Pupa.rb using your own models. You can also use Pupa.rb to scrape people, organizations, memberships and posts according to the [Popolo](http://popoloproject.com/) open government data specification. This gem is up-to-date with Popolo's 2014-05-09 version, but omits the Motion, VoteEvent, Count, Vote and Area classes.
 
 The [cat.rb](http://opennorth.github.io/pupa-ruby/docs/cat.html) example shows you how to:
 
@@ -69,21 +52,21 @@ The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.htm
 
 JSON parsing is enabled by default. To enable automatic parsing of HTML and XML, require the `nokogiri` and `multi_xml` gems.
 
-## [OpenCivicData](http://opencivicdata.org/) compatibility
-
-Both Pupa.rb and Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa) implement models for people, organizations and memberships from the [Popolo](http://popoloproject.com/) open government data specification. Pupa.rb lets you use your own classes, but Pupa only supports a fixed set of classes. A consequence of Pupa.rb's flexibility is that the value of the `_type` property for `Person`, `Organization` and `Membership` objects differs between Pupa.rb and Pupa. Pupa.rb has namespaced types like `pupa/person` – to allow Ruby to load the `Person` class in the `Pupa` module – whereas Pupa has unnamespaced types like `person`.
+### Automatic response decompression
 
-To save objects to MongoDB with unnamespaced types like Sunlight Labs' Pupa – in order to benefit from other tools in the [OpenCivicData](http://opencivicdata.org/) stack – add this line to the top of your script:
+Until [Faraday Middleware](https://github.com/lostisland/faraday_middleware) releases its next version (> 0.9.1), you must use the gem from its git repository to automatically decompress responses:
 
 ```ruby
-require 'pupa/refinements/opencivicdata'
+gem 'faraday_middleware', git: 'https://github.com/lostisland/faraday_middleware.git'
 ```
 
-It is not currently possible to run the `scrape` action with one of Pupa.rb and Pupa, and to then run the `import` action with the other. Both actions must be run by the same library.
+## Performance
+
+Pupa.rb offers several ways to significantly improve performance. [Read the documentation.](https://github.com/opennorth/pupa-ruby/blob/master/PERFORMANCE.md#readme)
 
 ## Integration with ODMs
 
-`Pupa::Model` is incompatible with `Mongoid::Document`. Don't do this:
+`Pupa::Model` is incompatible with `Mongoid::Document`. **Don't do this**:
 
 ```ruby
 class Cat
@@ -92,11 +75,37 @@ class Cat
 end
 ```
 
-Instead, have a scraping model that includes `Pupa::Model` and an app model that includes `Mongoid::Document`.
+Instead, have a simple scraping model that includes `Pupa::Model` and an app model that includes `Mongoid::Document` with your app's business logic.
 
-## Performance
+## What it tries to solve
 
-Pupa.rb offers several ways to significantly improve performance. [Read the documentation.](https://github.com/opennorth/pupa-ruby/blob/master/PERFORMANCE.md#readme)
+Pupa.rb's goal is to make scraping less painful by solving common problems:
+
+* If you are updating a database by scraping a website, you can either delete and recreate records, or you can merge the scraped records with the saved records. Pupa.rb offers a simple way to merge records, by [using an object's stable properties for identification](http://opennorth.github.io/pupa-ruby/docs/cat.html#section-7).
+* If you are scraping a source that references other sources – for example, a committee that references its members – you may want to link the source to its references with foreign keys. Pupa.rb will use whatever identifying information you scrape – for example, the members' names – to [fill in the foreign keys for you](http://opennorth.github.io/pupa-ruby/docs/bill.html#section-4).
+* Data sources may use different formats in different contexts. Pupa.rb makes it easy to [select scraping methods](#scraping-method-selection) according to criteria, like the year of publication [for example](http://opennorth.github.io/pupa-ruby/docs/legislator.html#section-3).
+* By splitting the scrape (extract) and import (load) steps, it's easier for you and volunteers to start a scraper without any interaction with a database.
+
+In short, Pupa.rb lets you spend more time on the tasks that are unique to your use case, and less time on common tasks like caching, merging and storing data. It also provides helpful features like:
+
+* Logging, to make debugging and monitoring a scraper easier
+* [Automatic response parsing](#automatic-response-parsing) of JSON, XML and HTML
+* [Option parsing](http://opennorth.github.io/pupa-ruby/docs/legislator.html#section-9), to control your scraper from the command-line
+* [Object validation](http://opennorth.github.io/pupa-ruby/docs/cat.html#section-4), using [JSON Schema](http://json-schema.org/)
+
+Pupa.rb is extensible, so that you can add your own models, parsers, helpers, actions, etc. It also offers several ways to [improve your scraper's performance](#performance).
+
+## [OpenCivicData](http://opencivicdata.org/) compatibility
+
+Both Pupa.rb and Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa) implement models for people, organizations and memberships from the [Popolo](http://popoloproject.com/) open government data specification. Pupa.rb lets you use your own classes, but Pupa only supports a fixed set of classes. A consequence of Pupa.rb's flexibility is that the value of the `_type` property for `Person`, `Organization` and `Membership` objects differs between Pupa.rb and Pupa. Pupa.rb has namespaced types like `pupa/person` – to allow Ruby to load the `Person` class in the `Pupa` module – whereas Pupa has unnamespaced types like `person`.
+
+To save objects to MongoDB with unnamespaced types like Sunlight Labs' Pupa – in order to benefit from other tools in the [OpenCivicData](http://opencivicdata.org/) stack – add this line to the top of your script:
+
+```ruby
+require 'pupa/refinements/opencivicdata'
+```
+
+It is not currently possible to run the `scrape` action with one of Pupa.rb and Pupa, and to then run the `import` action with the other. Both actions must be run by the same library.
 
 ## Testing
 

data/lib/pupa/models/area.rb

@@ -0,0 +1,23 @@
+module Pupa
+  # A geographic area whose geometry may change over time.
+  class Area
+    include Model
+
+    self.schema = 'popolo/area'
+
+    include Concerns::Timestamps
+    include Concerns::Sourceable
+
+    attr_accessor :name, :identifier, :classification, :parent_id, :geometry
+    dump          :name, :identifier, :classification, :parent_id, :geometry
+
+    foreign_key :parent_id
+
+    # Returns the area's name.
+    #
+    # @return [String] the area's name
+    def to_s
+      name
+    end
+  end
+end

data/lib/pupa/models/concerns/contactable.rb

@@ -26,11 +26,23 @@ module Pupa
       # @param [String] type a type of medium, e.g. "fax" or "email"
       # @param [String] value a value, e.g. a phone number or email address
       # @param [String] note a note, e.g. for grouping contact details by physical location
-      def add_contact_detail(type, value, note: nil)
+      # @param [String] label a human-readable label for the contact detail
+      # @param [String,Date,Time] valid_from the date from which the contact detail is valid
+      # @param [String,Date,Time] valid_until the date from which the contact detail is no longer valid
+      def add_contact_detail(type, value, note: nil, label: nil, valid_from: nil, valid_until: nil)
         data = {type: type, value: value}
         if note
           data[:note] = note
         end
+        if label
+          data[:label] = label
+        end
+        if valid_from
+          data[:valid_from] = valid_from
+        end
+        if valid_until
+          data[:valid_until] = valid_until
+        end
         if type.present? && value.present?
           @contact_details << data
         end

data/lib/pupa/models/concerns/nameable.rb

@@ -24,10 +24,16 @@ module Pupa
       # Adds an alternate or former name.
       #
       # @param [String] name an alternate or former name
-      # @param [Date,Time] start_date the date on which the name was adopted
-      # @param [Date,Time] end_date the date on which the name was abandoned
+      # @param [String,Date,Time] start_date the date on which the name was adopted
+      # @param [String,Date,Time] end_date the date on which the name was abandoned
       # @param [String] note a note, e.g. "Birth name"
-      def add_name(name, start_date: nil, end_date: nil, note: nil)
+      # @param [String] family_name one or more family names
+      # @param [String] given_name one or more primary given names
+      # @param [String] additional_name one or more secondary given names
+      # @param [String] honorific_prefix one or more honorifics preceding a person's name
+      # @param [String] honorific_suffix one or more honorifics following a person's name
+      # @param [String] patronymic_name one or more patronymic names
+      def add_name(name, start_date: nil, end_date: nil, note: nil, family_name: nil, given_name: nil, additional_name: nil, honorific_prefix: nil, honorific_suffix: nil, patronymic_name: nil)
         data = {name: name}
         if start_date
           data[:start_date] = start_date
@@ -38,6 +44,24 @@ module Pupa
         if note
           data[:note] = note
         end
+        if family_name
+          data[:family_name] = family_name
+        end
+        if given_name
+          data[:given_name] = given_name
+        end
+        if additional_name
+          data[:additional_name] = additional_name
+        end
+        if honorific_prefix
+          data[:honorific_prefix] = honorific_prefix
+        end
+        if honorific_suffix
+          data[:honorific_suffix] = honorific_suffix
+        end
+        if patronymic_name
+          data[:patronymic_name] = patronymic_name
+        end
         if name.present?
           @other_names << data
         end

data/lib/pupa/models/membership.rb

@@ -10,12 +10,10 @@ module Pupa
     include Concerns::Contactable
     include Concerns::Linkable
 
-    attr_accessor :label, :role, :person_id, :organization_id, :post_id,
-      :start_date, :end_date
-    dump :label, :role, :person_id, :organization_id, :post_id,
-      :start_date, :end_date
+    attr_accessor :label, :role, :member, :person_id, :organization_id, :post_id, :on_behalf_of_id, :area_id, :start_date, :end_date
+    dump          :label, :role, :member, :person_id, :organization_id, :post_id, :on_behalf_of_id, :area_id, :start_date, :end_date
 
-    foreign_key :person_id, :organization_id, :post_id
+    foreign_key :person_id, :organization_id, :post_id, :on_behalf_of_id, :area_id
 
     # Returns the IDs of the parties to the relationship.
     #

data/lib/pupa/models/motion.rb

@@ -0,0 +1,23 @@
+module Pupa
+  # A formal step to introduce a matter for consideration by an organization.
+  class Motion
+    include Model
+
+    self.schema = 'popolo/motion'
+
+    include Concerns::Timestamps
+    include Concerns::Sourceable
+
+    attr_accessor :organization_id, :legislative_session_id, :creator_id, :text, :classification, :date, :requirement, :result
+    dump          :organization_id, :legislative_session_id, :creator_id, :text, :classification, :date, :requirement, :result
+
+    foreign_key :organization_id, :legislative_session_id, :creator_id
+
+    # Returns the motion's text and organization ID.
+    #
+    # @return [String] the motion's text and organization ID
+    def to_s
+      "#{text} in #{organization_id}"
+    end
+  end
+end

data/lib/pupa/models/organization.rb

@@ -13,14 +13,12 @@ module Pupa
     include Concerns::Contactable
     include Concerns::Linkable
 
-    attr_accessor :name, :classification, :parent_id, :parent, :founding_date,
-      :dissolution_date, :image
-    dump :name, :classification, :parent_id, :parent, :founding_date,
-      :dissolution_date, :image
+    attr_accessor :name, :classification, :parent_id, :area_id, :founding_date, :dissolution_date, :image, :parent
+    dump          :name, :classification, :parent_id, :area_id, :founding_date, :dissolution_date, :image, :parent
 
-    foreign_key :parent_id
+    foreign_key :parent_id, :area_id
 
-    foreign_object :parent
+    foreign_object :parent # for testing
 
     # Returns the name of the organization.
     #

data/lib/pupa/models/person.rb

@@ -12,12 +12,10 @@ module Pupa
     include Concerns::Contactable
     include Concerns::Linkable
 
-    attr_accessor :name, :family_name, :given_name, :additional_name,
-      :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name,
-      :email, :gender, :birth_date, :death_date, :image, :summary, :biography
-    dump :name, :family_name, :given_name, :additional_name,
-      :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name,
-      :email, :gender, :birth_date, :death_date, :image, :summary, :biography
+    attr_accessor :name, :email, :gender, :birth_date, :death_date, :image, :summary, :biography, :national_identity,
+      :family_name, :given_name, :additional_name, :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name
+    dump          :name, :email, :gender, :birth_date, :death_date, :image, :summary, :biography, :national_identity,
+      :family_name, :given_name, :additional_name, :honorific_prefix, :honorific_suffix, :patronymic_name, :sort_name
 
     # Returns the person's name.
     #

data/lib/pupa/models/post.rb

@@ -10,16 +10,16 @@ module Pupa
     include Concerns::Contactable
     include Concerns::Linkable
 
-    attr_accessor :label, :role, :organization_id, :start_date, :end_date
-    dump :label, :role, :organization_id, :start_date, :end_date
+    attr_accessor :label, :other_label, :role, :organization_id, :area_id, :start_date, :end_date
+    dump          :label, :other_label, :role, :organization_id, :area_id, :start_date, :end_date
 
-    foreign_key :organization_id
+    foreign_key :organization_id, :area_id
 
-    # Returns the post's label and organization ID.
+    # Returns the post's label or role and organization ID.
     #
-    # @return [String] the post's label and organization ID
+    # @return [String] the post's label or role and organization ID
     def to_s
-      "#{label} in #{organization_id}"
+      "#{label || role} in #{organization_id}"
     end
 
     # A post should have a unique label within an organization, through it may

data/lib/pupa/models/vote.rb

@@ -0,0 +1,23 @@
+module Pupa
+  # A voter's vote in a vote event.
+  class Vote
+    include Model
+
+    self.schema = 'popolo/vote'
+
+    include Concerns::Timestamps
+    include Concerns::Sourceable
+
+    attr_accessor :vote_event_id, :voter_id, :option, :group_id, :role, :weight, :pair_id
+    dump          :vote_event_id, :voter_id, :option, :group_id, :role, :weight, :pair_id
+
+    foreign_key :vote_event_id, :voter_id, :group_id, :pair_id
+
+    # Returns the vote's option, voter ID and vote event ID.
+    #
+    # @return [String] the vote's option, voter ID and vote event ID
+    def to_s
+      "#{option} by #{voter_id} in #{vote_event_id}"
+    end
+  end
+end

data/lib/pupa/models/vote_event.rb

@@ -0,0 +1,72 @@
+module Pupa
+  # An event at which people's votes are recorded.
+  class VoteEvent
+    include Model
+
+    self.schema = 'popolo/vote_event'
+
+    include Concerns::Timestamps
+    include Concerns::Sourceable
+
+    attr_accessor :identifier, :motion_id, :organization_id, :legislative_session_id, :start_date, :end_date, :result, :group_results, :counts
+    dump          :identifier, :motion_id, :organization_id, :legislative_session_id, :start_date, :end_date, :result, :group_results, :counts
+
+    foreign_key :motion_id, :organization_id, :legislative_session_id
+
+    def initialize(*args)
+      @group_results = []
+      @counts = []
+      super
+    end
+
+    # Returns the vote event's identifier and organization ID.
+    #
+    # @return [String] the vote event's identifier and organization ID
+    def to_s
+      "#{identifier} in #{organization_id}"
+    end
+
+    # Sets the group results.
+    #
+    # @param [Array] group_results a list of group results
+    def group_results=(group_results)
+      @group_results = symbolize_keys(group_results)
+    end
+
+    # Sets the counts.
+    #
+    # @param [Array] counts a list of counts
+    def counts=(counts)
+      @counts = symbolize_keys(counts)
+    end
+
+    # Adds a group result.
+    #
+    # @param [String] result the result of the vote event within a group of voters
+    # @param [String] group a group of voters
+    def add_group_result(result, group: nil)
+      data = {result: result}
+      if group
+        data[:group] = group
+      end
+      if result.present?
+        @group_results << data
+      end
+    end
+
+    # Adds a count.
+    #
+    # @param [String] option an option in a vote event
+    # @param [String] value the number of votes for an option
+    # @param [String] group a group of voters
+    def add_count(option, value, group: nil)
+      data = {option: option, value: value}
+      if group
+        data[:group] = group
+      end
+      if option.present? && value.present?
+        @counts << data
+      end
+    end
+  end
+end

data/lib/pupa/processor/client.rb

@@ -2,11 +2,9 @@ require 'active_support/cache'
 require 'faraday_middleware'
 require 'faraday_middleware/response_middleware'
 
-require 'pupa/processor/middleware/gzip'
 require 'pupa/processor/middleware/logger'
 require 'pupa/processor/middleware/parse_html'
 require 'pupa/processor/middleware/parse_json'
-require 'pupa/processor/middleware/raise_error'
 require 'pupa/refinements/faraday'
 require 'pupa/refinements/faraday_middleware'
 
@@ -39,7 +37,7 @@ module Pupa
         Faraday.new do |connection|
           connection.request :url_encoded
           connection.use Middleware::Logger, Logger.new('faraday', level: level)
-          connection.use Middleware::RaiseError # useful for breaking concurrent requests
+          connection.use Faraday::Response::RaiseError
 
           # @see http://tools.ietf.org/html/rfc4627
           connection.use Middleware::ParseJson, preserve_raw: true, content_type: /\bjson$/
@@ -56,7 +54,9 @@ module Pupa
           end
 
           # Must come after the parser middlewares.
-          connection.use Middleware::Gzip
+          if FaradayMiddleware.const_defined?(:Gzip)
+            connection.use FaradayMiddleware::Gzip
+          end
 
           if cache_dir
             connection.response :caching do

data/lib/pupa/version.rb

@@ -1,3 +1,3 @@
 module Pupa
-  VERSION = "0.1.7"
+  VERSION = "0.1.8"
 end

data/lib/pupa.rb

@@ -26,10 +26,14 @@ require 'pupa/models/foreign_object'
 require 'pupa/models/model'
 require 'pupa/models/contact_detail_list'
 require 'pupa/models/identifier_list'
+require 'pupa/models/area'
 require 'pupa/models/membership'
+require 'pupa/models/motion'
 require 'pupa/models/organization'
 require 'pupa/models/person'
 require 'pupa/models/post'
+require 'pupa/models/vote'
+require 'pupa/models/vote_event'
 
 module Pupa
 end

data/schemas/popolo/area.json

@@ -0,0 +1,54 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/area.json#",
+  "title": "Area",
+  "description": "A geographic area whose geometry may change over time",
+  "type": "object",
+  "properties": {
+    "id": {
+      "description": "The area's unique identifier",
+      "type": ["string", "null"]
+    },
+    "name": {
+      "description": "A primary name",
+      "type": ["string", "null"]
+    },
+    "identifier": {
+      "description": "An issued identifier",
+      "type": ["string", "null"]
+    },
+    "classification": {
+      "description": "An area category, e.g. city",
+      "type": ["string", "null"]
+    },
+    "parent_id": {
+      "description": "The ID of the area that contains this area",
+      "type": ["string", "null"]
+    },
+    "parent": {
+      "description": "The area that contains this area",
+      "$ref": "http://popoloproject.com/schemas/area.json#"
+    },
+    "geometry": {
+      "description": "A geometry",
+      "type": ["object", "null"]
+    },
+    "created_at": {
+      "description": "The time at which the resource was created",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "updated_at": {
+      "description": "The time at which the resource was last modified",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "sources": {
+      "description": "URLs to documents from which the area is derived",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/link.json#"
+      }
+    }
+  }
+}

data/schemas/popolo/contact_detail.json

@@ -28,7 +28,7 @@
       "type": ["string", "null"],
       "pattern": "^[0-9]{4}(-[0-9]{2}){0,2}$"
     },
-    "valid_through": {
+    "valid_until": {
       "description": "The date from which the contact detail is no longer valid",
       "type": ["string", "null"],
       "pattern": "^[0-9]{4}(-[0-9]{2}){0,2}$"

data/schemas/popolo/count.json

@@ -0,0 +1,27 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/count.json#",
+  "title": "Count",
+  "description": "The number of votes for an option in a vote event",
+  "type": "object",
+  "properties": {
+    "option": {
+      "description": "An option in a vote event",
+      "type": "string",
+      "required": true
+    },
+    "value": {
+      "description": "The number of votes for an option",
+      "type": "integer",
+      "required": true
+    },
+    "group_id": {
+      "description": "The ID of a group of voters",
+      "type": ["string", "null"]
+    },
+    "group": {
+      "description": "A group of voters",
+      "type": ["object", "null"]
+    }
+  }
+}

data/schemas/popolo/group_result.json

@@ -0,0 +1,22 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/group_result.json#",
+  "title": "Group result",
+  "description": "A result of a vote event within a group of voters",
+  "type": "object",
+  "properties": {
+    "group_id": {
+      "description": "The ID of a group of voters",
+      "type": ["string", "null"]
+    },
+    "group": {
+      "description": "A group of voters",
+      "type": ["object", "null"]
+    },
+    "result": {
+      "description": "The result of the vote event within a group of voters",
+      "type": "string",
+      "required": true
+    }
+  }
+}

data/schemas/popolo/membership.json

@@ -19,7 +19,7 @@
     },
     "member": {
       "description": "The person or organization that is a member of the organization",
-      "type": ["object"]
+      "type": ["object", "null"]
     },
     "person_id": {
       "description": "The ID of the person who is a member of the organization",

data/schemas/popolo/motion.json

@@ -0,0 +1,82 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/motion.json#",
+  "title": "Motion",
+  "description": "A formal step to introduce a matter for consideration by an organization",
+  "type": "object",
+  "properties": {
+    "id": {
+      "description": "The motion's unique identifier",
+      "type": ["string", "null"]
+    },
+    "organization_id": {
+      "description": "The ID of the organization in which the motion is proposed",
+      "type": ["string", "null"]
+    },
+    "organization": {
+      "description": "The organization in which the motion is proposed",
+      "$ref": "http://popoloproject.com/schemas/organization.json#"
+    },
+    "legislative_session_id": {
+      "description": "The ID of the legislative session in which the motion is proposed",
+      "type": ["string", "null"]
+    },
+    "legislative_session": {
+      "description": "The legislative session in which the motion is proposed",
+      "type": ["object", "null"]
+    },
+    "creator_id": {
+      "description": "The ID of the person who proposed the motion",
+      "type": ["string", "null"]
+    },
+    "creator": {
+      "description": "The person who proposed the motion",
+      "$ref": "http://popoloproject.com/schemas/person.json#"
+    },
+    "text": {
+      "description": "The transcript or text of the motion",
+      "type": ["string", "null"]
+    },
+    "classification": {
+      "description": "A motion category, e.g. adjournment",
+      "type": ["string", "null"]
+    },
+    "date": {
+      "description": "The date on which the motion was proposed",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "requirement": {
+      "description": "The requirement for the motion to be adopted",
+      "type": ["string", "null"]
+    },
+    "result": {
+      "description": "The result of the motion",
+      "type": ["string", "null"]
+    },
+    "vote_events": {
+      "description": "Events at which people vote on the motion",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/vote_event.json#"
+      }
+    },
+    "created_at": {
+      "description": "The time at which the resource was created",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "updated_at": {
+      "description": "The time at which the resource was last modified",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "sources": {
+      "description": "URLs to documents from which the motion is derived",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/link.json#"
+      }
+    }
+  }
+}

data/schemas/popolo/organization.json

@@ -77,7 +77,7 @@
       }
     },
     "memberships": {
-      "description": "The relationships to which the organization is a party",
+      "description": "Memberships",
       "type": "array",
       "items": {
         "$ref": "http://popoloproject.com/schemas/membership.json#"

data/schemas/popolo/person.json

@@ -106,7 +106,7 @@
       }
     },
     "memberships": {
-      "description": "The relationships to which the person is a party",
+      "description": "Memberships",
       "type": "array",
       "items": {
         "$ref": "http://popoloproject.com/schemas/membership.json#"

data/schemas/popolo/vote.json

@@ -0,0 +1,57 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/vote.json#",
+  "title": "Vote",
+  "description": "A voter's vote in a vote event",
+  "type": "object",
+  "properties": {
+    "id": {
+      "description": "The vote's unique identifier",
+      "type": ["string", "null"]
+    },
+    "vote_event_id": {
+      "description": "The ID of a vote event",
+      "type": ["string", "null"]
+    },
+    "vote_event": {
+      "description": "A vote event",
+      "$ref": "http://popoloproject.com/schemas/vote_event.json#"
+    },
+    "voter_id": {
+      "description": "The ID of the person or organization that is voting",
+      "type": ["string", "null"]
+    },
+    "voter": {
+      "description": "The person or organization that is voting",
+      "type": ["object", "null"]
+    },
+    "option": {
+      "description": "The option chosen by the voter, whether actively or passively",
+      "type": ["string", "null"]
+    },
+    "group_id": {
+      "description": "The ID of the voter's primary political group",
+      "type": ["string", "null"]
+    },
+    "group": {
+      "description": "The voter's primary political group",
+      "$ref": "http://popoloproject.com/schemas/organization.json#"
+    },
+    "role": {
+      "description": "The voter's role in the event",
+      "type": ["string", "null"]
+    },
+    "weight": {
+      "description": "The weight of the voter's vote",
+      "type": ["number", "null"]
+    },
+    "pair_id": {
+      "description": "The ID of the person with whom the voter is paired",
+      "type": ["string", "null"]
+    },
+    "pair": {
+      "description": "The person with whom the voter is paired",
+      "$ref": "http://popoloproject.com/schemas/person.json#"
+    }
+  }
+}

data/schemas/popolo/vote_event.json

@@ -0,0 +1,93 @@
+{
+  "$schema": "http://json-schema.org/draft-03/schema#",
+  "id": "http://popoloproject.com/schemas/vote_event.json#",
+  "title": "Vote event",
+  "description": "An event at which people's votes are recorded",
+  "type": "object",
+  "properties": {
+    "id": {
+      "description": "The vote event's unique identifier",
+      "type": ["string", "null"]
+    },
+    "identifier": {
+      "description": "An issued identifier",
+      "type": ["string", "null"]
+    },
+    "motion_id": {
+      "description": "The ID of the motion being decided",
+      "type": ["string", "null"]
+    },
+    "motion": {
+      "description": "The motion being decided",
+      "$ref": "http://popoloproject.com/schemas/motion.json#"
+    },
+    "organization_id": {
+      "description": "The ID of the organization whose members are voting",
+      "type": ["string", "null"]
+    },
+    "organization": {
+      "description": "The organization whose members are voting",
+      "$ref": "http://popoloproject.com/schemas/organization.json#"
+    },
+    "legislative_session_id": {
+      "description": "The ID of the legislative session in which the vote occurs",
+      "type": ["string", "null"]
+    },
+    "legislative_session": {
+      "description": "The legislative session in which the vote occurs",
+      "type": ["object", "null"]
+    },
+    "start_date": {
+      "description": "The time at which the event begins",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "end_date": {
+      "description": "The time at which the event ends",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "result": {
+      "description": "The result of the vote event",
+      "type": ["string", "null"]
+    },
+    "group_results": {
+      "description": "The result of the vote event within groups of voters",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/group_result.json#"
+      }
+    },
+    "counts": {
+      "description": "The number of votes for options",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/count.json#"
+      }
+    },
+    "votes": {
+      "description": "Voters' votes",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/vote.json#"
+      }
+    },
+    "created_at": {
+      "description": "The time at which the resource was created",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "updated_at": {
+      "description": "The time at which the resource was last modified",
+      "type": ["string", "null"],
+      "format": "date-time"
+    },
+    "sources": {
+      "description": "URLs to documents from which the vote event is derived",
+      "type": "array",
+      "items": {
+        "$ref": "http://popoloproject.com/schemas/link.json#"
+      }
+    }
+  }
+}

data/spec/models/area_spec.rb

@@ -0,0 +1,13 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::Area do
+  let :object do
+    Pupa::Area.new(name: 'Boston Ward 1')
+  end
+
+  describe '#to_s' do
+    it 'should return a human-readable string' do
+      object.to_s.should == 'Boston Ward 1'
+    end
+  end
+end

data/spec/models/concerns/contactable_spec.rb

@@ -31,15 +31,15 @@ describe Pupa::Concerns::Contactable do
     end
 
     it 'should symbolize keys' do
-      object.contact_details = [{'type' => 'email', 'value' => 'ceo@example.com', 'note' => 'work'}]
-      object.contact_details.should == [{type: 'email', value: 'ceo@example.com', note: 'work'}]
+      object.contact_details = [{'type' => 'email', 'value' => 'ceo@example.com'}]
+      object.contact_details.should == [{type: 'email', value: 'ceo@example.com'}]
     end
   end
 
   describe '#add_contact_detail' do
     it 'should add a contact detail' do
-      object.add_contact_detail('email', 'ceo@example.com', note: 'work')
-      object.contact_details.should == [{type: 'email', value: 'ceo@example.com', note: 'work'}]
+      object.add_contact_detail('email', 'ceo@example.com', label: 'Email', note: 'work', valid_from: '2010-01-01', valid_until: '2010-12-31')
+      object.contact_details.should == [{type: 'email', value: 'ceo@example.com', label: 'Email', note: 'work', valid_from: '2010-01-01', valid_until: '2010-12-31'}]
     end
 
     it 'should not add a contact detail without a type' do

data/spec/models/concerns/nameable_spec.rb

@@ -14,15 +14,15 @@ describe Pupa::Concerns::Nameable do
 
   describe '#other_names' do
     it 'should symbolize keys' do
-      object.other_names = [{'name' => 'Mr. Ziggy Q. Public, Esq.', 'start_date' => '1920-01', 'end_date' => '1949-12-31', 'note' => 'Birth name'}]
-      object.other_names.should == [{name: 'Mr. Ziggy Q. Public, Esq.', start_date: '1920-01', end_date: '1949-12-31', note: 'Birth name'}]
+      object.other_names = [{'name' => 'Mr. Ziggy Q. Public, Esq.', 'note' => 'Birth name'}]
+      object.other_names.should == [{name: 'Mr. Ziggy Q. Public, Esq.', note: 'Birth name'}]
     end
   end
 
   describe '#add_name' do
     it 'should add a name' do
-      object.add_name('Mr. Ziggy Q. Public, Esq.', start_date: '1920-01', end_date: '1949-12-31', note: 'Birth name')
-      object.other_names.should == [{name: 'Mr. Ziggy Q. Public, Esq.', start_date: '1920-01', end_date: '1949-12-31', note: 'Birth name'}]
+      object.add_name('Mr. Ziggy Q. Public, Esq.', start_date: '1920-01', end_date: '1949-12-31', note: 'Birth name', family_name: 'Public', given_name: 'John', additional_name: 'Quinlan', honorific_prefix: 'Mr.', honorific_suffix: 'Esq.')
+      object.other_names.should == [{name: 'Mr. Ziggy Q. Public, Esq.', start_date: '1920-01', end_date: '1949-12-31', note: 'Birth name', family_name: 'Public', given_name: 'John', additional_name: 'Quinlan', honorific_prefix: 'Mr.', honorific_suffix: 'Esq.'}]
     end
 
     it 'should not add a name without a name' do

data/spec/models/motion_spec.rb

@@ -0,0 +1,13 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::Motion do
+  let :object do
+    Pupa::Motion.new(text: 'That the Bill is to be read a second time.', organization_id: 'house-of-commons')
+  end
+
+  describe '#to_s' do
+    it 'should return a human-readable string' do
+      object.to_s.should == 'That the Bill is to be read a second time. in house-of-commons'
+    end
+  end
+end

data/spec/models/vote_event_spec.rb

@@ -0,0 +1,59 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::VoteEvent do
+  let :object do
+    Pupa::VoteEvent.new(identifier: '1', organization_id: 'legislative-council-of-hong-kong')
+  end
+
+  describe '#to_s' do
+    it 'should return a human-readable string' do
+      object.to_s.should == '1 in legislative-council-of-hong-kong'
+    end
+  end
+
+  describe '#group_results' do
+    it 'should symbolize keys' do
+      object.group_results = [{'result' => 'pass', 'group' => {'name' => 'Functional constituencies'}}]
+      object.group_results.should == [{result: 'pass', group: {name: 'Functional constituencies'}}]
+    end
+  end
+
+  describe '#add_group_result' do
+    it 'should add a group result' do
+      object.add_group_result('pass', group: {name: 'Functional constituencies'})
+      object.group_results.should == [{result: 'pass', group: {name: 'Functional constituencies'}}]
+    end
+
+    it 'should not add a group result without a result' do
+      object.add_group_result(nil)
+      object.add_group_result('')
+      object.group_results.blank?.should == true
+    end
+  end
+
+  describe '#counts' do
+    it 'should symbolize keys' do
+      object.counts = [{'option' => 'yes', 'value' => 9, 'group' => {'name' => 'Functional constituencies'}}]
+      object.counts.should == [{option: 'yes', value: 9, group: {name: 'Functional constituencies'}}]
+    end
+  end
+
+  describe '#add_count' do
+    it 'should add a count' do
+      object.add_count('yes', 9, group: {name: 'Functional constituencies'})
+      object.counts.should == [{option: 'yes', value: 9, group: {name: 'Functional constituencies'}}]
+    end
+
+    it 'should not add a contact detail without an option' do
+      object.add_count(nil, 9)
+      object.add_count('', 9)
+      object.counts.blank?.should == true
+    end
+
+    it 'should not add a contact detail without a value' do
+      object.add_count('yes', nil)
+      object.add_count('yes', '')
+      object.counts.blank?.should == true
+    end
+  end
+end

data/spec/models/vote_spec.rb

@@ -0,0 +1,13 @@
+require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
+
+describe Pupa::Vote do
+  let :object do
+    Pupa::Vote.new(option: 'yes', voter_id: 'john-q-public', vote_event_id: 'vote-42')
+  end
+
+  describe '#to_s' do
+    it 'should return a human-readable string' do
+      object.to_s.should == 'yes by john-q-public in vote-42'
+    end
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupa
 version: !ruby/object:Gem::Version
-  version: 0.1.7
+  version: 0.1.8
 platform: ruby
 authors:
 - Open North
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-08-14 00:00:00.000000000 Z
+date: 2014-10-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -295,6 +295,7 @@ files:
 - lib/pupa.rb
 - lib/pupa/errors.rb
 - lib/pupa/logger.rb
+- lib/pupa/models/area.rb
 - lib/pupa/models/concerns/contactable.rb
 - lib/pupa/models/concerns/identifiable.rb
 - lib/pupa/models/concerns/indifferent_access.rb
@@ -307,9 +308,12 @@ files:
 - lib/pupa/models/identifier_list.rb
 - lib/pupa/models/membership.rb
 - lib/pupa/models/model.rb
+- lib/pupa/models/motion.rb
 - lib/pupa/models/organization.rb
 - lib/pupa/models/person.rb
 - lib/pupa/models/post.rb
+- lib/pupa/models/vote.rb
+- lib/pupa/models/vote_event.rb
 - lib/pupa/processor.rb
 - lib/pupa/processor/client.rb
 - lib/pupa/processor/connection.rb
@@ -320,11 +324,9 @@ files:
 - lib/pupa/processor/document_store/file_store.rb
 - lib/pupa/processor/document_store/redis_store.rb
 - lib/pupa/processor/helper.rb
-- lib/pupa/processor/middleware/gzip.rb
 - lib/pupa/processor/middleware/logger.rb
 - lib/pupa/processor/middleware/parse_html.rb
 - lib/pupa/processor/middleware/parse_json.rb
-- lib/pupa/processor/middleware/raise_error.rb
 - lib/pupa/processor/yielder.rb
 - lib/pupa/refinements/faraday.rb
 - lib/pupa/refinements/faraday_middleware.rb
@@ -333,18 +335,25 @@ files:
 - lib/pupa/runner.rb
 - lib/pupa/version.rb
 - pupa.gemspec
+- schemas/popolo/area.json
 - schemas/popolo/contact_detail.json
+- schemas/popolo/count.json
+- schemas/popolo/group_result.json
 - schemas/popolo/identifier.json
 - schemas/popolo/link.json
 - schemas/popolo/membership.json
+- schemas/popolo/motion.json
 - schemas/popolo/organization.json
 - schemas/popolo/other_name.json
 - schemas/popolo/person.json
 - schemas/popolo/post.json
+- schemas/popolo/vote.json
+- schemas/popolo/vote_event.json
 - spec/fixtures/bar.json
 - spec/fixtures/baz.json
 - spec/fixtures/foo.json
 - spec/logger_spec.rb
+- spec/models/area_spec.rb
 - spec/models/concerns/contactable_spec.rb
 - spec/models/concerns/identifiable_spec.rb
 - spec/models/concerns/linkable_spec.rb
@@ -355,9 +364,12 @@ files:
 - spec/models/identifier_list_spec.rb
 - spec/models/membership_spec.rb
 - spec/models/model_spec.rb
+- spec/models/motion_spec.rb
 - spec/models/organization_spec.rb
 - spec/models/person_spec.rb
 - spec/models/post_spec.rb
+- spec/models/vote_event_spec.rb
+- spec/models/vote_spec.rb
 - spec/processor/client_spec.rb
 - spec/processor/connection_adapters/mongodb_adapter_spec.rb
 - spec/processor/connection_adapters/postgresql_adapter_spec.rb
@@ -404,6 +416,7 @@ test_files:
 - spec/fixtures/baz.json
 - spec/fixtures/foo.json
 - spec/logger_spec.rb
+- spec/models/area_spec.rb
 - spec/models/concerns/contactable_spec.rb
 - spec/models/concerns/identifiable_spec.rb
 - spec/models/concerns/linkable_spec.rb
@@ -414,9 +427,12 @@ test_files:
 - spec/models/identifier_list_spec.rb
 - spec/models/membership_spec.rb
 - spec/models/model_spec.rb
+- spec/models/motion_spec.rb
 - spec/models/organization_spec.rb
 - spec/models/person_spec.rb
 - spec/models/post_spec.rb
+- spec/models/vote_event_spec.rb
+- spec/models/vote_spec.rb
 - spec/processor/client_spec.rb
 - spec/processor/connection_adapters/mongodb_adapter_spec.rb
 - spec/processor/connection_adapters/postgresql_adapter_spec.rb
@@ -434,3 +450,4 @@ test_files:
 - spec/refinements/opencivicdata_spec.rb
 - spec/runner_spec.rb
 - spec/spec_helper.rb
+has_rdoc: 

data/lib/pupa/processor/middleware/gzip.rb

@@ -1,24 +0,0 @@
-module Pupa
-  class Processor
-    module Middleware
-      # A Faraday response middleware for parsing gzip responses.
-      #
-      # @see https://gist.github.com/romanbsd/3892387
-      class Gzip < Faraday::Response::Middleware
-        dependency 'zlib'
-
-        def on_complete(env)
-          encoding = env[:response_headers]['content-encoding'].to_s.downcase
-          case encoding
-          when 'gzip'
-            env[:body] = Zlib::GzipReader.new(StringIO.new(env[:body]), encoding: 'ASCII-8BIT').read
-            env[:response_headers].delete('content-encoding')
-          when 'deflate'
-            env[:body] = Zlib::Inflate.inflate(env[:body])
-            env[:response_headers].delete('content-encoding')
-          end
-        end
-      end
-    end
-  end
-end

data/lib/pupa/processor/middleware/raise_error.rb

@@ -1,34 +0,0 @@
-module Pupa
-  class Processor
-    module Middleware
-      # A Faraday response middleware for raising an error if unsuccessful.
-      #
-      # @see Faraday::Response::RaiseError
-      # @note Faraday has no tests for this middleware.
-      class RaiseError < Faraday::Response::Middleware
-        def on_complete(env)
-          case env[:status]
-          when 404
-            raise Faraday::Error::ResourceNotFound, response_values(env)
-          when 407
-            # mimic the behavior that we get with proxy requests with HTTPS
-            raise Faraday::Error::ConnectionFailed, %{407 "Proxy Authentication Required "}
-          when 400...600
-            raise Faraday::Error::ClientError, response_values(env)
-          end
-        end
-
-        def response_values(env) # XXX add more keys
-          {
-            method: env[:method],
-            url: env[:url].to_s,
-            request_headers: env[:request_headers],
-            status: env[:status],
-            response_headers: env[:response_headers],
-            body: env[:body].to_s,
-          }
-        end
-      end
-    end
-  end
-end