pupa 0.1.10 → 0.1.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5ff448a08241852d405068e030d4b689b45373e0
-  data.tar.gz: 7b1327e204e05de6659155659ac6b3b7c1f3d853
+  metadata.gz: 08051be1ed0ea215d0e1c258d9b01bd189215dc2
+  data.tar.gz: dd163a78b33959069adbb5d971781a17a582a15b
 SHA512:
-  metadata.gz: 45805c371d13c20c22a600197df6d7a57ff70d6e1323ece52c51b108c7973d379ba2523d91afa7bf39e977a0a7c91538e437ada33be24aecdc83e8535dbe865d
-  data.tar.gz: 66fa473a639735566a13c2c2ad7c782a037865c3f5f38c7da4524f8865f25b0a3fb64c8a0d0687823968a0eadab1c908e3e6eeecd336a1b9c4e5a10bd123c2e6
+  metadata.gz: 597d6a420ae3c8ba04336b9ed38f4073925e90720d51895fbd83a9c77232f8c588834f66d5106fd9906c402bba752f4805c1014919812427b119aa870ead1cda
+  data.tar.gz: 98e65c5af6ea3f1a63b54c7c0c2091d0664660705d19e77bf22b2d5e7dfa911d56155e8cb6fa69ab11c22191e74444f1e60981f3523239e6a8873bc0d3c92efd

data/.travis.yml

@@ -2,9 +2,6 @@ language: ruby
 rvm:
   - 2.0.0
   - 2.1.0
-env:
-  - MODE=default
-  - MODE=compat
 services:
   - memcached
   - mongodb

data/README.md

@@ -6,13 +6,13 @@
 [![Coverage Status](https://coveralls.io/repos/opennorth/pupa-ruby/badge.png?branch=master)](https://coveralls.io/r/opennorth/pupa-ruby)
 [![Code Climate](https://codeclimate.com/github/opennorth/pupa-ruby.png)](https://codeclimate.com/github/opennorth/pupa-ruby)
 
-Pupa.rb is a Ruby 2.x fork of Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa). It implements an Extract, Transform and Load (ETL) process to scrape data from online sources, transform it, and write it to a database.
+Pupa.rb is a Ruby 2.x fork of Python [Pupa](https://github.com/opencivicdata/pupa). It implements an Extract, Transform and Load (ETL) process to scrape data from online sources, transform it, and write it to a database.
 
     gem install pupa
 
 ## Usage
 
-You can scrape any sort of data with Pupa.rb using your own models. You can also use Pupa.rb to scrape people, organizations, memberships and posts according to the [Popolo](http://www.popoloproject.com/) open government data specification. This gem is up-to-date with Popolo's 2014-05-09 version, but omits the Motion, VoteEvent, Count, Vote and Area classes.
+You can scrape any sort of data with Pupa.rb using your own models. You can also use Pupa.rb to scrape people, organizations, memberships and posts according to the [Popolo](http://www.popoloproject.com/) open government data specification. This gem is up-to-date with Popolo's 2014-10-28 version.
 
 The [cat.rb](http://opennorth.github.io/pupa-ruby/docs/cat.html) example shows you how to:
 
@@ -52,14 +52,6 @@ The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.htm
 
 JSON parsing is enabled by default. To enable automatic parsing of HTML and XML, require the `nokogiri` and `multi_xml` gems.
 
-### Automatic response decompression
-
-Until [Faraday Middleware](https://github.com/lostisland/faraday_middleware) releases its next version (> 0.9.1), you must use the gem from its git repository to automatically decompress responses:
-
-```ruby
-gem 'faraday_middleware', git: 'https://github.com/lostisland/faraday_middleware.git'
-```
-
 ## Performance
 
 Pupa.rb offers several ways to significantly improve performance. [Read the documentation.](https://github.com/opennorth/pupa-ruby/blob/master/PERFORMANCE.md#readme)
@@ -90,22 +82,15 @@ In short, Pupa.rb lets you spend more time on the tasks that are unique to your
 
 * Logging, to make debugging and monitoring a scraper easier
 * [Automatic response parsing](#automatic-response-parsing) of JSON, XML and HTML
+* Automatic response decompression
 * [Option parsing](http://opennorth.github.io/pupa-ruby/docs/legislator.html#section-9), to control your scraper from the command-line
 * [Object validation](http://opennorth.github.io/pupa-ruby/docs/cat.html#section-4), using [JSON Schema](http://json-schema.org/)
 
 Pupa.rb is extensible, so that you can add your own models, parsers, helpers, actions, etc. It also offers several ways to [improve your scraper's performance](#performance).
 
-## [OpenCivicData](http://opencivicdata.org/) compatibility
-
-Both Pupa.rb and Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa) implement models for people, organizations and memberships from the [Popolo](http://www.popoloproject.com/) open government data specification. Pupa.rb lets you use your own classes, but Pupa only supports a fixed set of classes. A consequence of Pupa.rb's flexibility is that the value of the `_type` property for `Person`, `Organization` and `Membership` objects differs between Pupa.rb and Pupa. Pupa.rb has namespaced types like `pupa/person` – to allow Ruby to load the `Person` class in the `Pupa` module – whereas Pupa has unnamespaced types like `person`.
-
-To save objects to MongoDB with unnamespaced types like Sunlight Labs' Pupa – in order to benefit from other tools in the [OpenCivicData](http://opencivicdata.org/) stack – add this line to the top of your script:
-
-```ruby
-require 'pupa/refinements/opencivicdata'
-```
+## Python [Pupa](https://github.com/opencivicdata/pupa) differences
 
-It is not currently possible to run the `scrape` action with one of Pupa.rb and Pupa, and to then run the `import` action with the other. Both actions must be run by the same library.
+Both Pupa.rb and Python [Pupa](https://github.com/opencivicdata/pupa) implement models from the [Popolo](http://www.popoloproject.com/) open government data specifications, but Pupa.rb also lets you use your own classes. Pupa.rb stores data in either MongoDB (default) or PostgreSQL (experimental); Python Pupa stores data in PostgreSQL. The PostgreSQL schema of Pupa.rb and Python Pupa differ.
 
 ## Testing
 

data/lib/pupa/runner.rb

@@ -13,9 +13,9 @@ module Pupa
       @options = OpenStruct.new({
         actions:            [],
         tasks:              [],
-        output_dir:         File.expand_path('scraped_data', Dir.pwd),
+        output_dir:         File.expand_path('_data', Dir.pwd),
         pipelined:          false,
-        cache_dir:          File.expand_path('web_cache', Dir.pwd),
+        cache_dir:          File.expand_path('_cache', Dir.pwd),
         expires_in:         86400, # 1 day
         value_max_bytes:    1048576, # 1 MB
         memcached_username: nil,

data/lib/pupa/version.rb

@@ -1,3 +1,3 @@
 module Pupa
-  VERSION = "0.1.10"
+  VERSION = "0.1.11"
 end

data/pupa.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_runtime_dependency('activesupport', '~> 4.0.0')
+  s.add_runtime_dependency('activesupport', '~> 4.0')
   s.add_runtime_dependency('colored', '~> 1.2')
   s.add_runtime_dependency('faraday_middleware', '~> 0.9.0')
   s.add_runtime_dependency('json-schema', '~> 2.1.3')

data/spec/processor/connection_adapters/mongodb_adapter_spec.rb

@@ -2,11 +2,7 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Processor::Connection::MongoDBAdapter do
   def _type
-    if testing_python_compatibility?
-      'person'
-    else
-      'pupa/person'
-    end
+    'pupa/person'
   end
 
   def connection

data/spec/processor/connection_adapters/postgresql_adapter_spec.rb

@@ -2,11 +2,7 @@ require File.expand_path(File.dirname(__FILE__) + '/../../spec_helper')
 
 describe Pupa::Processor::Connection::PostgreSQLAdapter do
   def _type
-    if testing_python_compatibility?
-      'person'
-    else
-      'pupa/person'
-    end
+    'pupa/person'
   end
 
   def connection

data/spec/processor_spec.rb

@@ -102,11 +102,7 @@ describe Pupa::Processor do
     end
 
     let :_type do
-      if testing_python_compatibility?
-        'organization'
-      else
-        'pupa/organization'
-      end
+      'pupa/organization'
     end
 
     let :graphable do

data/spec/spec_helper.rb

@@ -8,15 +8,3 @@ require 'nokogiri'
 require 'redis-store'
 require 'rspec'
 require File.dirname(__FILE__) + '/../lib/pupa'
-
-def testing_python_compatibility?
-  ENV['MODE'] == 'compat'
-end
-
-if testing_python_compatibility?
-  require File.dirname(__FILE__) + '/../lib/pupa/refinements/opencivicdata'
-end
-
-RSpec.configure do |c|
-  c.filter_run_excluding :testing_python_compatibility => true unless testing_python_compatibility?
-end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pupa
 version: !ruby/object:Gem::Version
-  version: 0.1.10
+  version: 0.1.11
 platform: ruby
 authors:
 - Open North
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-10-18 00:00:00.000000000 Z
+date: 2015-01-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.0.0
+        version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.0.0
+        version: '4.0'
 - !ruby/object:Gem::Dependency
   name: colored
   requirement: !ruby/object:Gem::Requirement
@@ -331,7 +331,6 @@ files:
 - lib/pupa/refinements/faraday.rb
 - lib/pupa/refinements/faraday_middleware.rb
 - lib/pupa/refinements/json-schema.rb
-- lib/pupa/refinements/opencivicdata.rb
 - lib/pupa/runner.rb
 - lib/pupa/version.rb
 - pupa.gemspec
@@ -385,7 +384,6 @@ files:
 - spec/processor/yielder_spec.rb
 - spec/processor_spec.rb
 - spec/refinements/json-schema_spec.rb
-- spec/refinements/opencivicdata_spec.rb
 - spec/runner_spec.rb
 - spec/spec_helper.rb
 homepage: http://github.com/opennorth/pupa-ruby
@@ -449,7 +447,5 @@ test_files:
 - spec/processor/yielder_spec.rb
 - spec/processor_spec.rb
 - spec/refinements/json-schema_spec.rb
-- spec/refinements/opencivicdata_spec.rb
 - spec/runner_spec.rb
 - spec/spec_helper.rb
-has_rdoc: 

data/lib/pupa/refinements/opencivicdata.rb

@@ -1,42 +0,0 @@
-# @see https://github.com/opennorth/pupa-ruby#opencivicdata-compatibility
-
-module Pupa::Model
-  # This unfortunately won't cause the behavior of any model that has already
-  # included `Pupa::Model` to change.
-  class << self
-    def append_features(base)
-      if base.instance_variable_defined?("@_dependencies")
-        base.instance_variable_get("@_dependencies") << self
-        return false
-      else
-        return false if base < self
-        @_dependencies.each { |dep| base.send(:include, dep) }
-        super
-        base.extend const_get("ClassMethods") if const_defined?("ClassMethods")
-        base.class_eval(&@_included_block) if instance_variable_defined?("@_included_block")
-        base.class_eval do # XXX
-          set_callback(:save, :before) do |object|
-            object._type = object._type.camelize.demodulize.underscore
-          end
-        end
-      end
-    end
-  end
-end
-
-# `set_callback` is called by `class_eval` in `ActiveSupport::Concern`. Without
-# monkey-patching `ActiveSupport::Concern`, we can either iterate `ObjectSpace`,
-# implement something like ActiveSupport's `DescendantsTracker` for inclusion
-# instead of inheritance, or go back to `Pupa::Model` being a superclass instead
-# of a mixin to take advantage of `DescendantsTracker` itself.
-#
-# Instead of adding a callback, we can override `to_h` when `persist` is `true`.
-ObjectSpace.each_object(Class) do |base|
-  if base != Sequel::Model && base.include?(Pupa::Model) # Sequel::Model will error on #include?
-    base.class_eval do
-      set_callback(:save, :before) do |object|
-        object._type = object._type.camelize.demodulize.underscore
-      end
-    end
-  end
-end

data/spec/refinements/opencivicdata_spec.rb

@@ -1,35 +0,0 @@
-require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')
-
-describe Pupa::Refinements, testing_python_compatibility: true do
-  module Music
-    class Band
-      include Pupa::Model
-
-      def save
-        run_callbacks(:save) do
-        end
-      end
-    end
-  end
-
-  module Pupa
-    class Committee < Organization
-      def save
-        run_callbacks(:save) do
-        end
-      end
-    end
-  end
-
-  it 'should demodulize the type of new models' do
-    object = Music::Band.new
-    object.save
-    object._type.should == 'band'
-  end
-
-  it 'should demodulize the type of existing models' do
-    object = Pupa::Committee.new
-    object.save
-    object._type.should == 'committee'
-  end
-end