abstract_importer 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 3f5a316e4e45d1c7a09af7b4930dda62e690d5c8
+  data.tar.gz: 333effaf5481100cd62f0f7523161747e923d515
+SHA512:
+  metadata.gz: a49953a3ecee96971314712785578a4b4fb1c09a2461b02448995fe1c3589f471e158c87c6e054aa613386a5ecaa064fd49a03e009d881bc5d29a6702dd23697
+  data.tar.gz: a18398b96c81a7f8f8ecb469c76fa06b7513f534a2c7d6c9b515479c76209161f08e844bc1b7bbf768881dc8b060730aaf2e7d47b44d44a6248d1087c24887fc

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in abstract_importer.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Bob Lail
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,168 @@
+# AbstractImporter
+
+AbstractImporter provides services for importing complex data from an arbitrary data source. It:
+
+ * Preserves relationships between tables that are imported as a set
+ * Allows you to extend and modify the import process through a DSL and callbacks
+ * Supports partial and idempotent imports
+ * Sports flexible reporting and logging
+
+
+
+## Getting Started
+
+### Installation
+
+Add this line to your application's `Gemfile`:
+
+    gem 'abstract_importer'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install abstract_importer
+
+
+
+### Usage
+
+Derive your own importer from `AbstractImporter::Base` and specify the tables you intend to import:
+
+```ruby
+class MyImporter < AbstractImporter::Base
+  
+  import do |import|
+    import.students
+    import.parents
+  end
+  
+end
+```
+
+`AbstractImporter` now knows it must import two collections: `students` and `parents`, in that order. It refers to this as its "Import Plan".
+
+
+##### Parent and Data Source
+
+`MyImporter`'s initializer takes two arguments: `parent` and `data_source`:
+
+ * `parent` is any object that will respond to the names of your collections with an `ActiveRecord::Relation`.
+ * `data_source` is any object that will respond to the names of your collections by yielding a hash of attributes once for every record you should import.
+
+Here are reasonable classes for `parent` and `data_source`:
+
+```ruby
+# parent
+class Account < ActiveRecord::Base
+  has_many :students
+  has_many :parents
+end
+```
+
+```ruby
+# data source
+class Database
+  def students
+    yield id: 457, name: "Ron"
+    yield id: 458, name: "Ginny"
+    yield id: 459, name: "Fred"
+    yield id: 460, name: "George"
+  end
+
+  def parents
+    yield id: 88, name: "Arthur"
+    yield id: 89, name: "Molly"
+  end
+end
+```
+
+
+##### legacy_id
+
+For every record that AbstractImporter creates, it will assign the attribute `legacy_id`.
+
+AbstractImporter uses this value to make sure that we don't import the same record twice in case an import is interrupted and needs to be retried or a user imports their old database more than once.
+
+
+##### Performing an Import
+
+A straightforward import looks like this:
+
+```ruby
+summary = MyImport.new(parent, data_source).perform!
+```
+
+AbstractImporter optionally takes a hash of settings as a third argument:
+
+ * `:dry_run` (default: `false`) when set to `true`, goes through all the steps except creating the records
+ * `:io` (default: `$stderr`) an IO object that is passed to the reporter
+ * `:reporter` (default: `AbstractImporter::Reporter.new(io)`) performs logging in response to various events
+
+
+
+### Customizing the Import Plan
+
+You can customize the Import Plan by defining various callbacks on each collection you declare:
+
+```ruby
+class MyImporter < AbstractImporter::Base
+  
+  import do |import|
+    import.students do |options|
+      options.finder :find_student
+      options.before_build { |attrs| attrs.merge(name: attrs[:name].capitalize) }
+      options.on_complete :students_completed
+    end
+    import.parents
+  end
+  
+  def find_student
+    ...
+  end
+  
+  def students_completed
+    ...
+  end
+  
+end
+```
+
+The complete list of callbacks is below.
+
+##### finder
+
+`finder` accepts a hash of attributes for a record to be imported and returns a corresponding record (if one exists). This can be useful for finding an preexisting counterpart to an imported record. (e.g. The user has created the tag "Butterbeer" and tries to import a tag with the same name. Although the legacy "Butterbeer" tag was never imported, it should not be, and any legacy articles associated with it should be associated with the native one.)
+
+##### before_build
+
+`before_build` allows a callback to modify the hash of attributes before it is passed to `ActiveRecord::Relation#build`.
+
+##### before_create
+
+`before_create` allows a callback to modify a record before `save` is called on it.
+
+##### rescue
+
+`rescue` (like `before_create`) is called with a record just before `save` is called. Unlike `before_create`, `rescue` is only called if the record does not pass validations.
+
+##### after_create
+
+`after_create` is called with the original hash of attributes and the newly-saved record right after it is successfully saved.
+
+##### on_complete
+
+`on_complete` is called when all of the records in a collection have been processed.
+
+
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end

data/abstract_importer.gemspec

@@ -0,0 +1,33 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'abstract_importer/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "abstract_importer"
+  spec.version       = AbstractImporter::VERSION
+  spec.authors       = ["Bob Lail"]
+  spec.email         = ["bob.lail@cph.org"]
+  spec.summary       = %q{Provides services for the mass-import of complex relational data}
+  spec.homepage      = "https://github.com/concordia-publishing-house/abstract_importer"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+  
+  spec.add_dependency "activerecord"
+  
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rails"
+  spec.add_development_dependency "sqlite3"
+  spec.add_development_dependency "turn"
+  spec.add_development_dependency "pry"
+  spec.add_development_dependency "rr"
+  spec.add_development_dependency "database_cleaner"
+  spec.add_development_dependency "simplecov"
+  spec.add_development_dependency "shoulda-context"
+  
+end

data/lib/abstract_importer/base.rb

@@ -0,0 +1,170 @@
+require 'abstract_importer/import_options'
+require 'abstract_importer/import_plan'
+require 'abstract_importer/reporter'
+require 'abstract_importer/collection'
+require 'abstract_importer/collection_importer'
+require 'abstract_importer/id_map'
+
+
+module AbstractImporter
+  class Base
+    
+    class << self
+      def import
+        yield @import_plan = ImportPlan.new
+      end
+      
+      attr_reader :import_plan
+    end
+    
+    
+    
+    def initialize(parent, source, options={})
+      @source       = source
+      @parent       = parent
+      
+      io            = options.fetch(:io, $stderr)
+      @reporter     = Reporter.new(io, Rails.env.production?)
+      @dry_run      = options.fetch(:dry_run, false)
+      
+      @id_map       = IdMap.new
+      @summary      = {}
+      @import_plan  = self.class.import_plan.to_h
+      @collections  = []
+    end
+    
+    attr_reader :source, :parent, :reporter, :id_map, :summary
+    
+    def dry_run?
+      @dry_run
+    end
+    
+    
+    
+    
+    
+    def perform!
+      reporter.start_all(self)
+      
+      ms = Benchmark.ms do
+        setup
+      end
+      reporter.finish_setup(ms)
+      
+      ms = Benchmark.ms do
+        collections.each &method(:import_collection)
+      end
+      
+      teardown
+      reporter.finish_all(self, ms)
+      @summary
+    end
+    
+    def setup
+      verify_source!
+      verify_parent!
+      instantiate_collections!
+      prepopulate_id_map!
+    end
+    
+    def import_collection(collection)
+      @summary[collection.name] = CollectionImporter.new(self, collection).perform!
+    end
+    
+    def teardown
+    end
+    
+    
+    
+    
+    
+    def describe_source
+      source.to_s
+    end
+    
+    def describe_destination
+      parent.to_s
+    end
+    
+    
+    
+    
+    
+    def remap_foreign_key?(plural, foreign_key)
+      true
+    end
+    
+    def map_foreign_key(legacy_id, plural, foreign_key, depends_on)
+      id_map.apply!(legacy_id, depends_on)
+    rescue IdMap::IdNotMappedError
+      record_no_id_in_map_error(legacy_id, plural, foreign_key, depends_on)
+      nil
+    end
+    
+    
+    
+    
+    
+  private
+    
+    attr_reader :collections, :import_plan
+    
+    def verify_source!
+      import_plan.keys.each do |collection|
+        next if source.respond_to?(collection)
+        
+        raise "#{source.class} does not respond to `#{collection}`; " <<
+              "but #{self.class} plans to import records with that name" 
+      end
+    end
+    
+    def verify_parent!
+      import_plan.keys.each do |collection|
+        next if parent.respond_to?(collection)
+        
+        raise "#{parent.class} does not have a collection named `#{collection}`; " <<
+              "but #{self.class} plans to import records with that name"
+      end
+    end
+    
+    def instantiate_collections!
+      @collections = import_plan.map do |name, block|
+        reflection = parent.class.reflect_on_association(name)
+        model = reflection.klass
+        table_name = model.table_name
+        scope = parent.public_send(name)
+        
+        options = ImportOptions.new
+        instance_exec(options, &block) if block
+        
+        Collection.new(name, model, table_name, scope, options)
+      end
+    end
+    
+    def prepopulate_id_map!
+      collections.each do |collection|
+        query = collection.scope.where("#{collection.table_name}.legacy_id IS NOT NULL")
+        map = values_of(query, :id, :legacy_id) \
+          .each_with_object({}) { |(id, legacy_id), map| map[legacy_id] = id }
+        
+        id_map.init collection.table_name, map
+      end
+    end
+    
+    def values_of(query, *columns)
+      if Rails.version < "4.0.0"
+        query = query.select(columns.map { |column| "#{query.table_name}.#{column}" }.join(", "))
+        ActiveRecord::Base.connection.select_rows(query.to_sql)
+      else
+        query.pluck(*columns)
+      end
+    end
+    
+    
+    
+    def record_no_id_in_map_error(legacy_id, plural, foreign_key, depends_on)
+      reporter.count_notice "#{plural}.#{foreign_key} will be nil: a #{depends_on.to_s.singularize} with the legacy id #{legacy_id} was not mapped."
+    end
+    
+  end
+end

data/lib/abstract_importer/collection.rb

@@ -0,0 +1,4 @@
+module AbstractImporter
+  class Collection < Struct.new(:name, :model, :table_name, :scope, :options)
+  end
+end

data/lib/abstract_importer/collection_importer.rb

@@ -0,0 +1,211 @@
+module AbstractImporter
+  class CollectionImporter
+    
+    def initialize(importer, collection)
+      @importer = importer
+      @collection = collection
+    end
+    
+    attr_reader :importer, :collection, :summary
+    
+    delegate :name,
+             :table_name,
+             :model,
+             :scope,
+             :options,
+             :to => :collection
+    
+    delegate :dry_run?,
+             :parent,
+             :source,
+             :reporter,
+             :remap_foreign_key?,
+             :id_map,
+             :map_foreign_key,
+             :to => :importer
+    
+    
+    
+    def perform!
+      reporter.start_collection(self)
+      prepare!
+      
+      summary[5] = Benchmark.ms do
+        each_new_record &method(:process_record)
+      end
+      
+      invoke_callback(:on_complete)
+      reporter.finish_collection(self, summary)
+      summary
+    end
+    
+    
+    
+    def prepare!
+      #          [total, existing_records, new_records, already_imported, invalid, milliseconds]
+      @summary = [    0,                0,           0,                0,       0,            0]
+      @already_imported = load_already_imported_records!
+      @mappings = prepare_mappings!
+    end
+    
+    def load_already_imported_records!
+      # We keep a _separate_ list of legacy IDs that
+      # have already been imported. It would optimal to
+      # check @id_map to see if a record has been imported;
+      # but because of a bug with tags, that won't work:
+      #
+      # Tags import from three table: Activity, Skill,
+      # and Training. Those tables yield tags whose
+      # legacy_ids collide. As a result several tags
+      # can share the same ID; and tags that would collide
+      # are [erroneously] not imported.
+      #
+      # Fixing this problem would involve changing the
+      # legacy_id identifier for each tag which would
+      # break the connection between already-imported tags
+      # and new imports.
+      #
+      id_map[table_name]
+    end
+    
+    def prepare_mappings!
+      mappings = []
+      model.reflect_on_all_associations.each do |association|
+        
+        # We only want the associations where this record
+        # has foreign keys that refer to another
+        next unless association.macro == :belongs_to
+        
+        # We don't at this time support polymorphic associations
+        # which would require extending id_map to take the foreign
+        # type fields into account.
+        #
+        # Rails can't return `association.table_name` so easily
+        # because `table_name` comes from `klass` and `klass`
+        # isn't predetermined.
+        next if association.options[:polymorphic]
+        
+        depends_on = association.table_name.to_sym
+        foreign_key = association.foreign_key.to_sym
+        
+        # We support skipping some mappings entirely. I believe
+        # this is largely to cut down on verbosity in the log
+        # files and should be refactored to another place in time.
+        next unless remap_foreign_key?(name, foreign_key)
+        
+        mappings << Proc.new do |hash|
+          if hash.key?(foreign_key)
+            hash[foreign_key] = map_foreign_key(hash[foreign_key], name, foreign_key, depends_on)
+          else
+            reporter.count_notice "#{name}.#{foreign_key} will not be mapped because it is not used"
+          end
+        end
+      end
+      mappings
+    end
+    
+    
+    
+    
+    
+    def each_new_record
+      source.public_send(name) do |hash_or_hashes|
+        Array.wrap(hash_or_hashes).each do |hash|
+          yield hash.dup
+        end
+      end
+    end
+    
+    def process_record(hash)
+      summary[0] += 1
+      
+      if already_imported?(hash)
+        summary[3] += 1
+        return
+      end
+      
+      remap_foreign_keys!(hash)
+      
+      if redundant_record?(hash)
+        summary[1] += 1
+        return
+      end
+      
+      if create_record(hash)
+        summary[2] += 1
+      else
+        summary[4] += 1
+      end
+    end
+    
+    
+    
+    
+    
+    def already_imported?(hash)
+      @already_imported.key? hash[:id]
+    end
+    
+    def remap_foreign_keys!(hash)
+      @mappings.each do |proc|
+        proc.call(hash)
+      end
+    end
+    
+    def redundant_record?(hash)
+      existing_record = invoke_callback(:finder, hash)
+      if existing_record
+        id_map.register(record: existing_record, legacy_id: hash[:id])
+        true
+      else
+        false
+      end
+    end
+    
+    
+    
+    
+    
+    def create_record(hash)
+      record = build_record(hash)
+      
+      return true if dry_run?
+      
+      invoke_callback(:before_create, record)
+      
+      # rescue_callback has one shot to fix things
+      invoke_callback(:rescue, record) unless record.valid?
+      
+      if record.save
+        invoke_callback(:after_create, hash, record)
+        id_map << record
+        
+        reporter.record_created(record)
+        true
+      else
+        
+        reporter.record_failed(record)
+        false
+      end
+    end
+    
+    def build_record(hash)
+      hash = invoke_callback(:before_build, hash) || hash
+      
+      legacy_id = hash.delete(:id)
+      
+      scope.build hash.merge(legacy_id: legacy_id)
+    end
+    
+    
+    
+    def invoke_callback(callback, *args)
+      callback_name = :"#{callback}_callback"
+      callback = options.public_send(callback_name)
+      return unless callback
+      callback = importer.method(callback) if callback.is_a?(Symbol)
+      callback.call(*args)
+    end
+    
+  end
+end

data/lib/abstract_importer/id_map.rb

@@ -0,0 +1,47 @@
+module AbstractImporter
+  class IdMap
+    
+    class IdNotMappedError < StandardError; end
+    
+    def initialize
+      @id_map = Hash.new { |hash, key| hash[key] = {} }
+    end
+    
+    
+    
+    def init(table_name, map)
+      table_name = table_name.to_sym
+      @id_map[table_name] = map
+    end
+    
+    def get(table_name)
+      @id_map[table_name.to_sym].dup
+    end
+    alias :[] :get
+    
+    def <<(record)
+      register(record: record)
+    end
+    
+    def register(options={})
+      if options.key?(:record)
+        record = options[:record]
+        table_name, record_id, legacy_id = record.class.table_name, record.id, record.legacy_id
+      end
+      table_name = options[:table_name] if options.key?(:table_name)
+      legacy_id = options[:legacy_id] if options.key?(:legacy_id)
+      record_id = options[:record_id] if options.key?(:record_id)
+      
+      table_name = table_name.to_sym
+      @id_map[table_name][legacy_id] = record_id
+    end
+    
+    def apply!(legacy_id, depends_on)
+      return nil if legacy_id.blank?
+      id_map = @id_map[depends_on]
+      raise IdNotMappedError.new unless id_map.key?(legacy_id)
+      id_map[legacy_id]
+    end
+    
+  end
+end