data_miner 2.0.1 → 2.0.2

data/.gitignore

@@ -1,10 +1,8 @@
-*.sw?
 .DS_Store
-coverage
-rdoc
-pkg
-test/test.sqlite3
-data_miner.log
+/coverage
+/rdoc
+/pkg
 Gemfile.lock
 *.gem
-test.log
+/.yardoc
+/doc

data/CHANGELOG

@@ -1,3 +1,16 @@
+2.0.2 / 2012-05-04
+
+* Breaking changes
+
+  * Import descriptions are no longer optional
+  * Import options are no longer optional (but then, they never were)
+
+* Enhancements
+
+  * Real documentation!
+  * Replace class-level mutexes with simple Thread.exclusive calls
+  * Simplified DataMiner::Dictionary
+
 2.0.1 / 2012-04-18
 
 * Enhancements

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2011 Brighter Planet
+Copyright (c) 2012 Brighter Planet
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.markdown

@@ -0,0 +1,112 @@
+# data_miner
+
+Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models.
+
+Tested in MRI 1.8.7+, MRI 1.9.2+, and JRuby 1.6.7+. Thread safe.
+
+## Real-world usage
+
+<p><a href="http://brighterplanet.com"><img src="https://s3.amazonaws.com/static.brighterplanet.com/assets/logos/flush-left/inline/green/rasterized/brighter_planet-160-transparent.png" alt="Brighter Planet logo"/></a></p>
+
+We use `data_miner` for [data science at Brighter Planet](http://brighterplanet.com/research) and in production at
+
+* [Brighter Planet's reference data web service](http://data.brighterplanet.com)
+* [Brighter Planet's impact estimate web service](http://impact.brighterplanet.com)
+
+The killer combination for us is:
+
+1. [`active_record_inline_schema`](https://github.com/seamusabshere/active_record_inline_schema) - define table structure
+2. [`remote_table`](https://github.com/seamusabshere/remote_table) - download data and parse it
+3. [`errata`](https://github.com/seamusabshere/errata) - apply corrections in a transparent way
+4. [`data_miner`](https://github.com/seamusabshere/remote_table) (this library!) - import data idempotently
+
+## Documentation
+
+Check out the [extensive documentation](http://rdoc.info/github/seamusabshere/data_miner).
+
+## Quick start
+
+You define <code>data_miner</code> blocks in your ActiveRecord models. For example, in <code>app/models/country.rb</code>:
+
+    class Country < ActiveRecord::Base
+      self.primary_key = 'iso_3166_code'
+  
+      data_miner do
+        import("OpenGeoCode.org's Country Codes to Country Names list",
+               :url => 'http://opengeocode.org/download/countrynames.txt',
+               :format => :delimited,
+               :delimiter => '; ',
+               :headers => false,
+               :skip => 22) do
+          key   :iso_3166_code, :field_number => 0
+          store :iso_3166_alpha_3_code, :field_number => 1
+          store :iso_3166_numeric_code, :field_number => 2
+          store :name, :field_number => 5
+        end
+      end
+    end
+
+Now you can run:
+
+    >> Country.run_data_miner!
+    => nil
+
+## More advanced usage
+
+The [`earth` library](https://github.com/brighterplanet/earth) has dozens of real-life examples showing how to download, pull out of a ZIP/TAR/BZ2 archive, parse, correct, and import CSVs, fixed-width files, ODS, XLS, XLSX, even HTML and XML:
+
+<table>
+  <tr>
+    <th>Model</th>
+    <th>Highlights</th>
+    <th>Reference</th>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/aircraft">Aircraft</a></td>
+    <td>parsing Microsoft Frontpage HTML (!)</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/air/aircraft/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/airports">Airports</a></td>
+    <td>forcing column names and use of <code>:select</code> block (<code>Proc</code>)</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/air/airport/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/automobile_make_model_year_variants">Automobile model variants</a></td>
+    <td>super advanced usage of "custom parser" and errata</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/automobile/automobile_make_model_year_variant/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/countries">Country</a></td>
+    <td>parsing CSV and a few other tricks</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/locality/country/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/egrid_regions">EGRID regions</a></td>
+    <td>parsing XLS</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/locality/egrid_region/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/flight_segments">Flight segment (stage)</a></td>
+    <td>super advanced usage of POSTing form data</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/air/flight_segment/data_miner.rb">data_miner.rb</a></td>
+  </tr>
+  <tr>
+    <td><a href="http://data.brighterplanet.com/zip_codes">Zip codes</a></td>
+    <td>downloading a ZIP file and pulling an XLSX out of it</td>
+    <td><a href="https://github.com/brighterplanet/earth/blob/master/lib/earth/locality/zip_code.rb">data_miner.rb</a></td>
+  </tr>
+</table>
+
+And many more - look for the `data_miner.rb` file that corresponds to each model. Note that you would normally put the `data_miner` declaration right inside the ActiveRecord model file... it's kept separate in `earth` so that loading it is optional.
+
+## Authors
+
+* Seamus Abshere <seamus@abshere.net>
+* Andy Rossmeissl <andy@rossmeissl.net>
+* Derek Kastner <dkastner@gmail.com>
+* Ian Hough <ijhough@gmail.com>
+
+## Copyright
+
+Copyright (c) 2012 Brighter Planet. See LICENSE for details.

data/data_miner.gemspec

@@ -7,8 +7,8 @@ Gem::Specification.new do |s|
   s.authors     = ["Seamus Abshere", "Andy Rossmeissl", "Derek Kastner"]
   s.email       = ["seamus@abshere.net"]
   s.homepage    = "https://github.com/seamusabshere/data_miner"
-  s.summary     = %{Mine remote data into your ActiveRecord models.}
-  s.description = %q{Mine remote data into your ActiveRecord models. You can also convert units.}
+  s.summary     = %{Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models.}
+  s.description = %q{Download, pull out of a ZIP/TAR/GZ/BZ2 archive, parse, correct, and import XLS, ODS, XML, CSV, HTML, etc. into your ActiveRecord models. You can also convert units.}
 
   s.rubyforge_project = "data_miner"
 

data/lib/data_miner.rb

@@ -14,7 +14,7 @@ if RUBY_VERSION >= '1.9'
   end
 end
 
-require 'data_miner/active_record_extensions'
+require 'data_miner/active_record_class_methods'
 require 'data_miner/attribute'
 require 'data_miner/script'
 require 'data_miner/dictionary'
@@ -24,14 +24,13 @@ require 'data_miner/step/tap'
 require 'data_miner/step/process'
 require 'data_miner/run'
 
+# A singleton class that holds global configuration for data mining.
+#
+# All of its instance methods are delegated to +DataMiner.instance+, so you can call +DataMiner.model_names+, for example.
+#
+# @see DataMiner::ActiveRecordClassMethods#data_miner Overview of how to define data miner scripts inside of ActiveRecord models.
 class DataMiner
   class << self
-    delegate :perform, :to => :instance
-    delegate :run, :to => :instance
-    delegate :logger, :to => :instance
-    delegate :logger=, :to => :instance
-    delegate :model_names, :to => :instance
-
     # @private
     def downcase(str)
       defined?(::UnicodeUtils) ? ::UnicodeUtils.downcase(str) : str.downcase
@@ -48,16 +47,20 @@ class DataMiner
     end
   end
 
-  MUTEX = ::Mutex.new
   INNER_SPACE = /[ ]+/
 
   include ::Singleton
 
   attr_writer :logger
 
+  # Run data miner scripts on models identified by their names. Defaults to all models.
+  #
+  # @param [optional, Array<String>] model_names Names of models to be run.
+  #
+  # @return [Array<DataMiner::Run>]
   def perform(model_names = DataMiner.model_names)
     Script.uniq do
-      model_names.each do |model_name|
+      model_names.map do |model_name|
         model_name.constantize.run_data_miner!
       end
     end
@@ -66,8 +69,11 @@ class DataMiner
   # legacy
   alias :run :perform
 
+  # Where DataMiner logs to. Defaults to +Rails.logger+ or +ActiveRecord::Base.logger+ if either is available.
+  #
+  # @return [Logger]
   def logger
-    @logger || MUTEX.synchronize do
+    @logger || ::Thread.exclusive do
       @logger ||= if defined?(::Rails)
         ::Rails.logger
       elsif defined?(::ActiveRecord) and active_record_logger = ::ActiveRecord::Base.logger
@@ -79,12 +85,20 @@ class DataMiner
     end
   end
 
+  # Names of the models that have defined a data miner script.
+  #
+  # @note Models won't appear here until the files containing their data miner scripts have been +require+'d.
+  #
+  # @return [Set<String>]
   def model_names
-    @model_names || MUTEX.synchronize do
+    @model_names || ::Thread.exclusive do
       @model_names ||= ::Set.new
     end
   end
 
+  class << self
+    delegate(*DataMiner.instance_methods(false), :to => :instance)
+  end
 end
 
-::ActiveRecord::Base.extend ::DataMiner::ActiveRecordExtensions
+::ActiveRecord::Base.extend ::DataMiner::ActiveRecordClassMethods

data/lib/data_miner/active_record_class_methods.rb

@@ -0,0 +1,108 @@
+require 'active_record'
+require 'lock_method'
+
+class DataMiner
+  # Class methods that are mixed into models (i.e. ActiveRecord::Base)
+  module ActiveRecordClassMethods
+    # Access this model's script.
+    #
+    # @return [DataMiner::Script] This model's data miner script.
+    def data_miner_script
+      @data_miner_script || ::Thread.exclusive do
+        @data_miner_script ||= DataMiner::Script.new(self)
+      end
+    end
+    
+    # Access to recordkeeping.
+    #
+    # @return [ActiveRecord::Relation] Records of running the data miner script.
+    def data_miner_runs
+      DataMiner::Run.scoped :conditions => { :model_name => name }
+    end
+
+    # Run this model's script.
+    #
+    # @return [DataMiner::Run]
+    def run_data_miner!
+      data_miner_script.perform
+    end
+    
+    # Run the data miner scripts of parent associations. Useful for dependencies. Safe to call using +process+.
+    #
+    # @note Used extensively in https://github.com/brighterplanet/earth
+    #
+    # @example Since Provinces depend on Countries, make sure Countries are data mined first
+    #   class Country < ActiveRecord::Base
+    #     [...some data miner script...]
+    #   end
+    #   class Province < ActiveRecord::Base
+    #     belongs_to :country
+    #     data_miner do
+    #       [...]
+    #       process "make sure my dependencies have been loaded" do
+    #         run_data_miner_on_parent_associations!
+    #       end
+    #       [...]
+    #     end
+    #   end
+    #
+    # @return [Array<DataMiner::Run>]
+    def run_data_miner_on_parent_associations!
+      reflect_on_all_associations(:belongs_to).reject do |assoc|
+        assoc.options[:polymorphic]
+      end.map do |non_polymorphic_belongs_to_assoc|
+        non_polymorphic_belongs_to_assoc.klass.run_data_miner!
+      end
+    end
+    
+    # Define a data miner script.
+    #
+    # @param [optional, Hash] options
+    # @option options [TrueClass, FalseClass] :append (false) Add steps to existing data miner script instead of starting from scratch.
+    #
+    # @yield [] The block defining the steps.
+    #
+    # @see DataMiner::Script#import
+    # @see DataMiner::Script#process
+    # @see DataMiner::Script#tap
+    #
+    # @example Creating steps
+    #   class MyModel < ActiveRecord::Base
+    #     data_miner do
+    #       process [...]
+    #       import [...]
+    #       import [...yes, it's ok to have more than one import step...]
+    #       process [...]
+    #       [...etc...]
+    #     end
+    #   end
+    #
+    # @example From the README
+    #   class Country < ActiveRecord::Base
+    #     self.primary_key = 'iso_3166_code'
+    #     data_miner do
+    #       import("OpenGeoCode.org's Country Codes to Country Names list",
+    #              :url => 'http://opengeocode.org/download/countrynames.txt',
+    #              :format => :delimited,
+    #              :delimiter => '; ',
+    #              :headers => false,
+    #              :skip => 22) do
+    #         key   :iso_3166_code, :field_number => 0
+    #         store :iso_3166_alpha_3_code, :field_number => 1
+    #         store :iso_3166_numeric_code, :field_number => 2
+    #         store :name, :field_number => 5
+    #       end
+    #     end
+    #   end
+    #
+    # @return [nil]
+    def data_miner(options = {}, &blk)
+      DataMiner.model_names.add name
+      unless options[:append]
+        @data_miner_script = nil
+      end
+      data_miner_script.append_block blk
+      nil
+    end
+  end
+end

data/lib/data_miner/attribute.rb

@@ -1,8 +1,14 @@
 require 'conversions'
 
 class DataMiner
+  # A mapping between a local model column and a remote data source column.
+  #
+  # @see DataMiner::ActiveRecordClassMethods#data_miner Overview of how to define data miner scripts inside of ActiveRecord models.
+  # @see DataMiner::Step::Import#store
+  # @see DataMiner::Step::Import#key
   class Attribute
     class << self
+      # @private
       def check_options(options)
         errors = []
         if options[:dictionary].is_a?(Dictionary)
@@ -18,26 +24,26 @@ class DataMiner
       end
     end
 
-    VALID_OPTIONS = %w{
-      from_units
-      to_units
-      static
-      dictionary
-      matcher
-      field_name
-      delimiter
-      split
-      units
-      sprintf
-      nullify
-      overwrite
-      upcase
-      units_field_name
-      units_field_number
-      field_number
-      chars
-      synthesize
-    }.map(&:to_sym)
+    VALID_OPTIONS = [
+      :from_units,
+      :to_units,
+      :static,
+      :dictionary,
+      :matcher,
+      :field_name,
+      :delimiter,
+      :split,
+      :units,
+      :sprintf,
+      :nullify,
+      :overwrite,
+      :upcase,
+      :units_field_name,
+      :units_field_number,
+      :field_number,
+      :chars,
+      :synthesize,
+    ]
 
     VALID_UNIT_DEFINITION_SETS = [
       [:units],
@@ -48,30 +54,102 @@ class DataMiner
       [:units_field_number, :to_units],
     ]
 
-    DEFAULT_SPLIT = /\s+/
-    DEFAULT_KEEP = 0
+    DEFAULT_SPLIT_PATTERN = /\s+/
+    DEFAULT_SPLIT_KEEP = 0
     DEFAULT_DELIMITER = ', '
     DEFAULT_NULLIFY = false
     DEFAULT_UPCASE = false
     DEFAULT_OVERWRITE = true
 
+    # @private
     attr_reader :step
+    
+    # Local column name.
+    # @return [Symbol]
     attr_reader :name
+    
+    # Synthesize a value by passing a proc that will receive +row+ and should return a final value.
+    #
+    # +row+ will be a +Hash+ with string keys or (less often) an +Array+
+    #
+    # @return [Proc]
     attr_reader :synthesize
+    
+    # An object that will be sent +#match(row)+ and should return a final value.
+    #
+    # Can be specified as a String which will be constantized into a class and an object of that class instantized with no arguments.
+    #
+    # +row+ will be a +Hash+ with string keys or (less often) an +Array+
+    # @return [Object]
     attr_reader :matcher
+    
+    # Index of where to find the data in the row, starting from zero.
+    #
+    # If you pass a +Range+, then multiple fields will be joined together.
+    #
+    # @return [Integer, Range]
     attr_reader :field_number
+
+    # Where to find the data in the row.
+    # @return [Symbol]
     attr_reader :field_name
-    # For use when joining a range of field numbers
+
+    # A delimiter to be used when joining fields together into a single final value. Used when +:field_number+ is a +Range+. Defaults to DEFAULT_DELIMITER.
+    # @return [String]
     attr_reader :delimiter
+
+    # Which characters in a field to keep. Zero-based.
+    # @return [Range]
     attr_reader :chars
+
+    # How to split a field. You specify two options:
+    #
+    # +:pattern+: what to split on. Defaults to DEFAULT_SPLIT_PATTERN.
+    # +:keep+: which of elements resulting from the split to keep. Defaults to DEFAULT_SPLIT_KEEP.
+    #
+    # @return [Hash]
     attr_reader :split
+
+    # Final units. May invoke a conversion using https://github.com/seamusabshere/conversions
+    #
+    # If a local column named +[name]_units+ exists, it will be populated with this value.
+    #
+    # @return [Symbol]
     attr_reader :to_units
+
+    # Initial units. May invoke a conversion using https://github.com/seamusabshere/conversions
+    # @return [Symbol]
     attr_reader :from_units
+
+    # If every row specifies its own units, index of where to find the units. Zero-based.
+    # @return [Integer]
     attr_reader :units_field_number
+
+    # If every row specifies its own units, where to find the units.
+    # @return [Symbol]
     attr_reader :units_field_name
+
+    # A +sprintf+-style format to apply.
+    # @return [String]
     attr_reader :sprintf
+
+    # A static value to be used.
+    # @return [String,Numeric,TrueClass,FalseClass,Object]
     attr_reader :static
 
+    # Whether to nullify the value in a local column if it was not previously null. Defaults to DEFAULT_NULLIFY.
+    # @return [TrueClass,FalseClass]
+    attr_reader :nullify
+
+    # Whether to upcase value. Defaults to DEFAULT_UPCASE.
+    # @return [TrueClass,FalseClass]
+    attr_reader :upcase
+
+    # Whether to overwrite the value in a local column if it is not null. Defaults to DEFAULT_OVERWRITE.
+    # @return [TrueClass,FalseClass]
+    attr_reader :overwrite
+
+    # @private
     def initialize(step, name, options = {})
       options = options.symbolize_keys
       if (errors = Attribute.check_options(options)).any?
@@ -81,7 +159,7 @@ class DataMiner
       @name = name
       @synthesize = options[:synthesize]
       if @dictionary_boolean = options.has_key?(:dictionary)
-        @dictionary_options = options[:dictionary]
+        @dictionary_settings = options[:dictionary]
       end
       @matcher = options[:matcher].is_a?(::String) ? options[:matcher].constantize.new : options[:matcher]
       if @static_boolean = options.has_key?(:static)
@@ -94,52 +172,42 @@ class DataMiner
       if split = options[:split]
         @split = split.symbolize_keys
       end
-      @nullify_boolean = options.fetch :nullify, DEFAULT_NULLIFY
-      @upcase_boolean = options.fetch :upcase, DEFAULT_UPCASE
+      @nullify = options.fetch :nullify, DEFAULT_NULLIFY
+      @upcase = options.fetch :upcase, DEFAULT_UPCASE
       @from_units = options[:from_units]
       @to_units = options[:to_units] || options[:units]
       @sprintf = options[:sprintf]
-      @overwrite_boolean = options.fetch :overwrite, DEFAULT_OVERWRITE
+      @overwrite = options.fetch :overwrite, DEFAULT_OVERWRITE
       @units_field_name = options[:units_field_name]
       @units_field_number = options[:units_field_number]
       @dictionary_mutex = ::Mutex.new
     end
 
-    def model
-      step.model
-    end
-
-    def static?
-      @static_boolean
-    end
-
-    def nullify?
-      @nullify_boolean
-    end
-
-    def upcase?
-      @upcase_boolean
-    end
-
-    def dictionary?
-      @dictionary_boolean
-    end
-
-    def convert?
-      from_units.present? or units_field_name.present? or units_field_number.present?
+    # Dictionary for translating.
+    #
+    # You pass a +Hash+ of options which is used to initialize a +DataMiner::Dictionary+.
+    #
+    # @return [DataMiner::Dictionary]
+    def dictionary
+      @dictionary || @dictionary_mutex.synchronize do
+        @dictionary ||= Dictionary.new(@dictionary_settings)
+      end
     end
 
-    def units?
-      to_units.present? or units_field_name.present? or units_field_number.present?
+    # @private
+    def set_from_row(local_record, remote_row)
+      if overwrite or local_record.send(name).nil?
+        local_record.send "#{name}=", read(remote_row)
+      end
+      if units? and ((final_to_units = (to_units || read_units(remote_row))) or nullify)
+        local_record.send "#{name}_units=", final_to_units
+      end
     end
 
-    def overwrite?
-      @overwrite_boolean
-    end
-        
+    # @private
     def read(row)
-      if matcher and matched_row = matcher.match(row)
-        return matched_row
+      if matcher and matcher_output = matcher.match(row)
+        return matcher_output
       end
       if synthesize
         return synthesize.call(row)
@@ -168,15 +236,15 @@ class DataMiner
         value = value[chars]
       end
       if split
-        pattern = split.fetch :pattern, DEFAULT_SPLIT
-        keep = split.fetch :keep, DEFAULT_KEEP
+        pattern = split.fetch :pattern, DEFAULT_SPLIT_PATTERN
+        keep = split.fetch :keep, DEFAULT_SPLIT_KEEP
         value = value.to_s.split(pattern)[keep].to_s
       end
       value = DataMiner.compress_whitespace value
-      if nullify? and value.blank?
+      if nullify and value.blank?
         return
       end
-      if upcase?
+      if upcase
         value = DataMiner.upcase value
       end
       if convert?
@@ -201,27 +269,33 @@ class DataMiner
       value
     end
 
-    def set_from_row(target, row)
-      if overwrite? or target.send(name).nil?
-        target.send "#{name}=", read(row)
-      end
-      if units? and ((final_to_units = (to_units || read_units(row))) or nullify?)
-        target.send "#{name}_units=", final_to_units
-      end
-    end
-
-    def dictionary
-      @dictionary || @dictionary_mutex.synchronize do
-        @dictionary ||= Dictionary.new(@dictionary_options)
-      end
-    end
-
+    # @private
     def refresh
       @dictionary = nil
     end
         
     private
 
+    def model
+      step.model
+    end
+
+    def static?
+      @static_boolean
+    end
+
+    def dictionary?
+      @dictionary_boolean
+    end
+
+    def convert?
+      from_units.present? or units_field_name.present? or units_field_number.present?
+    end
+
+    def units?
+      to_units.present? or units_field_name.present? or units_field_number.present?
+    end
+
     def read_units(row)
       if units = row[units_field_name || units_field_number]
         DataMiner.compress_whitespace(units).underscore.to_sym