pupa 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 97c9d9fa0543551ea5408129269350f772f5622b
+  data.tar.gz: c5784cad0628a12a1b6bd7f1b88e023da0c93b21
+SHA512:
+  metadata.gz: 9fb0c239b6b7d8f3b3c9e264ca83051a6b7e7156578543f99d29034c8385da32055e0b03ab685d740afc46a6cea1a0c18e2ec92a65605575153c1da30a2bd5c9
+  data.tar.gz: 49dd4745dcb8ad7db1bd54ca9d5aaf3c2a640d16c62494699fbf3ab601034055ba77d8a800e1b632efe4638d74fb4d92aa99525bc8286e63591bb120a28990a3

data/.gitignore

@@ -0,0 +1,6 @@
+*.gem
+.bundle
+.yardoc
+Gemfile.lock
+doc/*
+pkg/*

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - 2.0.0
+services:
+  - mongodb

data/.yardopts

@@ -0,0 +1,4 @@
+--no-private
+--hide-void-return
+--embed-mixin ClassMethods
+--markup=markdown

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in the gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Open North Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,52 @@
+# Pupa.rb: A Data Scraping Framework
+
+[![Build Status](https://secure.travis-ci.org/opennorth/pupa-ruby.png)](http://travis-ci.org/opennorth/pupa-ruby)
+[![Dependency Status](https://gemnasium.com/opennorth/pupa-ruby.png)](https://gemnasium.com/opennorth/pupa-ruby)
+[![Coverage Status](https://coveralls.io/repos/opennorth/pupa-ruby/badge.png?branch=master)](https://coveralls.io/r/opennorth/pupa-ruby)
+[![Code Climate](https://codeclimate.com/github/opennorth/pupa-ruby.png)](https://codeclimate.com/github/opennorth/pupa-ruby)
+
+Pupa.rb is a Ruby 2.0 fork of Sunlight Labs' [Pupa](https://github.com/opencivicdata/pupa). It implements an Extract, Transform and Load (ETL) process to scrape data from online sources, transform it, and write it to a database.
+
+## Usage
+
+You can use Pupa.rb to author scrapers that create people, organizations, memberships and posts according to the [Popolo](http://popoloproject.com/) open government data specification. If you need to scrape other types of data, you can also use your own models with Pupa.rb.
+
+The [cat.rb](http://opennorth.github.io/pupa-ruby/docs/cat.html) example shows you how to:
+
+* write a simple Cat class that is compatible with Pupa.rb
+* use mixins to add Popolo properties to your class
+* write a processor to scrape Cat objects from the Internet
+* register a scraping task with Pupa.rb
+* run the processor to save the Cat objects to MongoDB
+
+The [bill.rb](http://opennorth.github.io/pupa-ruby/docs/bill.html) example shows you how to:
+
+* create relations between objects
+* relate two objects, even if you do not know the ID of one object
+* write separate scraping tasks for different types of data
+* run each scraping task separately
+
+The [legislator.rb](http://opennorth.github.io/pupa-ruby/docs/legislator.html) example shows you how to:
+
+* use a different HTTP client than the default [Faraday](https://github.com/lostisland/faraday)
+* select a scraping method according to criteria like the legislative term
+* pass selection criteria to the processor before running scraping tasks
+
+The [organization.rb](http://opennorth.github.io/pupa-ruby/docs/organization.html) example shows you how to:
+
+* register a transformation task with Pupa.rb
+* run the processor's transformation task
+
+### Scraping method selection
+
+1.  For simple processing, your processor class need only define a single `scrape_objects` method, which will perform all scraping. See [cat.rb](http://opennorth.github.io/pupa-ruby/docs/cat.html) for an example.
+
+1.  If you scrape many types of data from the same source, you may want to split the scraping into separate tasks according to the type of data being scraped. See [bill.rb](http://opennorth.github.io/pupa-ruby/docs/bill.html) for an example.
+
+1.  You may want more control over the method used to perform a scraping task. For example, a legislature may publish legislators before 1997 in one format and legislators after 1997 in another format. In this case, you may want to select the method used to scrape legislators according to the year. See [legislator.rb](http://opennorth.github.io/pupa-ruby/docs/legislator.html).
+
+## Bugs? Questions?
+
+This project's main repository is on GitHub: [http://github.com/opennorth/pupa-ruby](http://github.com/opennorth/pupa-ruby), where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.
+
+Copyright (c) 2013 Open North Inc., released under the MIT license

data/Rakefile

@@ -0,0 +1,37 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec
+
+begin
+  require 'yard'
+  YARD::Rake::YardocTask.new
+rescue LoadError
+  task :yard do
+    abort 'YARD is not available. In order to run yard, you must: gem install yard'
+  end
+end
+
+namespace :popolo do
+  desc 'Update Popolo schemas'
+  task :schemas do
+    require 'base64'
+
+    require 'octokit'
+
+    Octokit.contents('opennorth/popolo-spec', path: 'schemas', ref: 'gh-pages').each do |file|
+      response = Octokit.contents('opennorth/popolo-spec', path: file.path, ref: 'gh-pages')
+      if response.encoding == 'base64'
+        content = Base64.decode64(response.content)
+      else
+        raise "Can't handle #{response.encoding} encoding"
+      end
+      File.open(File.expand_path(File.join('schemas', 'popolo', file.name), __dir__), 'w') do |f|
+        f.write(content)
+      end
+    end
+  end
+end

data/USAGE

@@ -0,0 +1 @@
+See README.md for full usage details.

data/lib/pupa/errors.rb

@@ -0,0 +1,30 @@
+module Pupa
+  module Errors
+    # An abstract class from which all Pupa errors inherit.
+    class Error < StandardError; end
+
+    # This error is raised when saving an object to a database if a foreign key
+    # cannot be resolved.
+    class MissingDatabaseIdError < Error; end
+
+    # This error is raised when dumping scraped objects to disk if two of those
+    # objects share an ID.
+    class DuplicateObjectIdError < Error; end
+
+    # This error is raised when attempting to get or set a property that does
+    # not exist in an object.
+    class MissingAttributeError < Error; end
+
+    # This error is raised when saving an object to a database if the object
+    # matches more than one document in the database.
+    class TooManyMatches < Error; end
+
+    # This error is raised if an object's foreign keys or foreign objects cannot
+    # be resolved.
+    class UnprocessableEntity < Error; end
+
+    # This error is raised if duplicate objects were inadvertently saved to the
+    # database.
+    class DuplicateDocumentError < Error; end
+  end
+end

data/lib/pupa/logger.rb

@@ -0,0 +1,37 @@
+require 'logger'
+
+require 'colored'
+
+module Pupa
+  # A logger factory.
+  class Logger
+    # Returns a configured logger.
+    #
+    # @param [String] progname the name of the program performing the logging
+    # @param [String] level the log level, one of "DEBUG", "INFO", "WARN",
+    #   "ERROR", "FATAL" or "UNKNOWN"
+    # @param [String,IO] logdev the log device
+    # @return [Logger] a configured logger
+    def self.new(progname, level: 'INFO', logdev: STDOUT)
+      logger = ::Logger.new(logdev)
+      logger.level = ::Logger.const_get(level)
+      logger.progname = progname
+      logger.formatter = proc do |severity, datetime, progname, msg|
+        message = "#{datetime.strftime('%T')} #{severity} #{progname}: #{msg}\n"
+        case severity
+        when 'DEBUG'
+          message.magenta
+        when 'INFO'
+          message.white
+        when 'WARN'
+          message.yellow
+        when 'ERROR'
+          message.red
+        when 'FATAL'
+          message.bold.red_on_white
+        end
+      end
+      logger
+    end
+  end
+end

data/lib/pupa/models/base.rb

@@ -0,0 +1,190 @@
+require 'pathname'
+require 'securerandom'
+require 'set'
+
+require 'active_support/callbacks'
+require 'active_support/core_ext/hash/except'
+require 'active_support/core_ext/hash/slice'
+require 'active_support/core_ext/object/try'
+require 'json-schema'
+
+require 'pupa/refinements/json-schema'
+
+JSON::Validator.cache_schemas = true
+
+module Pupa
+  # The base class from which other primary Popolo classes inherit.
+  class Base
+    include ActiveSupport::Callbacks
+    define_callbacks :create, :save
+
+    class_attribute :json_schema
+    class_attribute :properties
+    class_attribute :foreign_keys
+    class_attribute :foreign_objects
+
+    self.properties = Set.new
+    self.foreign_keys = Set.new
+    self.foreign_objects = Set.new
+
+    class << self
+      # Declare the class' properties.
+      #
+      # When converting an object to a hash using the `to_h` method, only the
+      # properties declared with `attr_accessor` will be included in the hash.
+      #
+      # @param [Array<Symbol>] the class' properties
+      def attr_accessor(*attributes)
+        self.properties += attributes # use assignment to not overwrite the parent's attribute
+        super
+      end
+
+      # Declare the class' foreign keys.
+      #
+      # When importing scraped objects, the foreign keys will be used to draw a
+      # dependency graph and derive an evaluation order.
+      #
+      # @param [Array<Symbol>] the class' foreign keys
+      def foreign_key(*attributes)
+        self.foreign_keys += attributes
+      end
+
+      # Declare the class' foreign objects.
+      #
+      # If some cases, you may not know the ID of an existing foreign object,
+      # but you may have other information to identify the object. In that case,
+      # put the information you have in a property named after the foreign key
+      # without the `_id` suffix: for example, `person` for `person_id`. Before
+      # saving the object to the database, Pupa.rb will use this information to
+      # identify the foreign object.
+      #
+      # @param [Array<Symbol>] the class' foreign objects
+      def foreign_object(*attributes)
+        self.foreign_objects += attributes
+      end
+
+      # Sets the class' schema.
+      #
+      # @param [Hash,String] value a hash or a relative or absolute path
+      def schema=(value)
+        self.json_schema = if Hash === value
+          value
+        elsif Pathname.new(value).absolute?
+          value
+        else
+          File.expand_path(File.join('..', '..', '..', 'schemas', "#{value}.json"), __dir__)
+        end
+      end
+    end
+
+    attr_accessor :_id, :_type, :extras
+
+    # @param [Hash] properties the object's properties
+    def initialize(properties = {})
+      @_type = self.class.to_s.underscore
+      @_id = SecureRandom.uuid
+      @extras = {}
+
+      properties.each do |key,value|
+        self[key] = value
+      end
+    end
+
+    # Returns the value of a property.
+    #
+    # @param [Symbol] property a property name
+    # @raises [Pupa::Errors::MissingAttributeError] if class is missing the property
+    def [](property)
+      if properties.include?(property.to_sym)
+        send(property)
+      else
+        raise Errors::MissingAttributeError, "missing attribute: #{property}"
+      end
+    end
+
+    # Sets the value of a property.
+    #
+    # @param [Symbol] property a property name
+    # @param value a value
+    # @raises [Pupa::Errors::MissingAttributeError] if class is missing the property
+    def []=(property, value)
+      if properties.include?(property.to_sym)
+        send("#{property}=", value)
+      else
+        raise Errors::MissingAttributeError, "missing attribute: #{property}"
+      end
+    end
+
+    # Sets the object's ID.
+    #
+    # @param [String,Moped::BSON::ObjectId] id an ID
+    def _id=(id)
+      @_id = id.to_s # in case of Moped::BSON::ObjectId
+    end
+
+    # Adds a key-value pair to the object.
+    #
+    # @param [Symbol] key a key
+    # @param value a value
+    def add_extra(key, value)
+      @extras[key] = value
+    end
+
+    # Returns a subset of the object's properties that should uniquely identify
+    # the object.
+    #
+    # @return [Hash] a subset of the object's properties
+    def fingerprint
+      to_h.except(:_id)
+    end
+
+    # Returns the object's foreign keys and foreign objects.
+    #
+    # @return [Hash] the object's foreign keys and foreign objects
+    def foreign_properties
+      to_h(include_foreign_objects: true).slice(*foreign_keys + foreign_objects)
+    end
+
+    # Validates the object against the schema.
+    #
+    # @raises [JSON::Schema::ValidationError] if the object is invalid
+    def validate!
+      if self.class.json_schema
+        result = {}
+        to_h.each do |key,value|
+          result[key.to_s] = value
+        end
+        # JSON::Validator#initialize_data runs fastest if given a hash.
+        JSON::Validator.validate!(self.class.json_schema, result)
+      end
+    end
+
+    # Returns the object as a hash.
+    #
+    # @param [Boolean] include_foreign_objects whether to include foreign objects
+    # @return [Hash] the object as a hash
+    def to_h(include_foreign_objects: false)
+      {}.tap do |hash|
+        (include_foreign_objects ? properties : properties - foreign_objects).each do |property|
+          value = self[property]
+          if value == false || value.present?
+            hash[property] = value
+          end
+        end
+      end
+    end
+
+    # Returns whether two objects are identical, ignoring any differences in
+    # the objects' machine IDs.
+    #
+    # @param [Object] other another object
+    # @return [Boolean] whether the objects are identical
+    def ==(other)
+      a = to_h
+      b = other.to_h
+      a.delete(:_id)
+      b.delete(:_id)
+      a == b
+    end
+  end
+end

data/lib/pupa/models/concerns/contactable.rb

@@ -0,0 +1,34 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `contact_details` property to a model.
+    module Contactable
+      extend ActiveSupport::Concern
+
+      included do
+        attr_reader :contact_details
+      end
+
+      # Sets the contact details.
+      #
+      # @param [Array] contact_details a list of contact details
+      def contact_details=(contact_details)
+        @contact_details = ContactDetailList.new(contact_details)
+      end
+
+      # Adds a contact detail.
+      #
+      # @param [String] type a type of medium, e.g. "fax" or "email"
+      # @param [String] value a value, e.g. a phone number or email address
+      # @param [String] note a note, e.g. for grouping contact details by physical location
+      def add_contact_detail(type, value, note: nil)
+        data = {type: type, value: value}
+        if note
+          data[:note] = note
+        end
+        if type && value
+          (@contact_details ||= ContactDetailList.new) << data
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/identifiable.rb

@@ -0,0 +1,26 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `identifiers` property to a model.
+    module Identifiable
+      extend ActiveSupport::Concern
+
+      included do
+        attr_accessor :identifiers
+      end
+
+      # Adds an issued identifier.
+      #
+      # @param [String] identifier an issued identifier, e.g. a DUNS number
+      # @param [String] scheme an identifier scheme, e.g. DUNS
+      def add_identifier(identifier, scheme: nil)
+        data = {identifier: identifier}
+        if scheme
+          data[:scheme] = scheme
+        end
+        if identifier
+          (@identifiers ||= []) << data
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/linkable.rb

@@ -0,0 +1,26 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `links` property to a model.
+    module Linkable
+      extend ActiveSupport::Concern
+
+      included do
+        attr_accessor :links
+      end
+
+      # Adds a URL.
+      #
+      # @param [String] url a URL
+      # @param [String] note a note, e.g. "Wikipedia page"
+      def add_link(url, note: nil)
+        data = {url: url}
+        if note
+          data[:note] = note
+        end
+        if url
+          (@links ||= []) << data
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/nameable.rb

@@ -0,0 +1,34 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `other_names` property to a model.
+    module Nameable
+      extend ActiveSupport::Concern
+
+      included do
+        attr_accessor :other_names
+      end
+
+      # Adds an alternate or former name.
+      #
+      # @param [String] name an alternate or former name
+      # @param [Date,Time] start_date the date on which the name was adopted
+      # @param [Date,Time] end_date the date on which the name was abandoned
+      # @param [String] note a note, e.g. "Birth name"
+      def add_name(name, start_date: nil, end_date: nil, note: nil)
+        data = {name: name}
+        if start_date
+          data[:start_date] = start_date
+        end
+        if end_date
+          data[:end_date] = end_date
+        end
+        if note
+          data[:note] = note
+        end
+        if name
+          (@other_names ||= []) << data
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/sourceable.rb

@@ -0,0 +1,26 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `sources` property to a model.
+    module Sourceable
+      extend ActiveSupport::Concern
+
+      included do
+        attr_accessor :sources
+      end
+
+      # Adds a source to the object.
+      #
+      # @param [String] url a URL
+      # @param [String] note a note
+      def add_source(url, note: nil)
+        data = {url: url}
+        if note
+          data[:note] = note
+        end
+        if url
+          (@sources ||= []) << data
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/concerns/timestamps.rb

@@ -0,0 +1,22 @@
+module Pupa
+  module Concerns
+    # Adds the Popolo `created_at` and `updated_at` properties to a model. The
+    # `created_at` and `updated_at` properties will automatically be set when
+    # the object is inserted into or updated in the database.
+    module Timestamps
+      extend ActiveSupport::Concern
+
+      included do
+        attr_accessor :created_at, :updated_at
+
+        set_callback(:create, :before) do |object|
+          object.created_at = Time.now.utc
+        end
+
+        set_callback(:save, :before) do |object|
+          object.updated_at = Time.now.utc
+        end
+      end
+    end
+  end
+end

data/lib/pupa/models/contact_detail_list.rb

@@ -0,0 +1,28 @@
+module Pupa
+  # A list of contact details.
+  class ContactDetailList < Array
+    # Returns the first postal address within the list of contact details.
+    #
+    # @return [String,nil] a postal address
+    def address
+      find_by_type('address')
+    end
+
+    # Returns the first email address within the list of contact details.
+    #
+    # @return [String,nil] an email address
+    def email
+      find_by_type('email')
+    end
+
+  private
+
+    def find_by_type(type)
+      find{|contact_detail|
+        contact_detail[:type] == type
+      }.try{|contact_detail|
+        contact_detail[:value]
+      }
+    end
+  end
+end

data/lib/pupa/models/membership.rb

@@ -0,0 +1,37 @@
+module Pupa
+  # A relationship between a person and an organization.
+  class Membership < Base
+    self.schema = 'popolo/membership'
+
+    include Concerns::Timestamps
+    include Concerns::Sourceable
+    include Concerns::Contactable
+    include Concerns::Linkable
+
+    attr_accessor :label, :role, :person_id, :organization_id, :post_id,
+      :start_date, :end_date
+
+    foreign_key :person_id, :organization_id, :post_id
+
+    # Returns the IDs of the parties to the relationship.
+    #
+    # @return [String] the IDs of the parties to the relationship
+    def to_s
+      "#{person_id} in #{organization_id}"
+    end
+
+    # A person's relationship with an organization must have a unique label,
+    # though it may share a label with a historical relationship. Similarly, a
+    # person may hold a post in an organization multiple times historically but
+    # not simultaneously.
+    def fingerprint
+      hash = super
+      {
+        '$or' => [
+          hash.slice(:label, :person_id, :organization_id, :end_date),
+          hash.slice(:person_id, :organization_id, :post_id, :end_date),
+        ],
+      }
+    end
+  end
+end