mlins-active_migration 1.0.2 → 1.0.3

data/CHANGELOG

@@ -0,0 +1,16 @@
+*1.0.3 (November 22, 2008)*
+
+* Fixed gemspec problems
+
+*1.0.2 (November 22, 2008)*
+
+* Renamed gem back to active_migration
+
+*1.0.1 (November 22, 2008)*
+
+* Renamed gem to activemigration
+* Fixed dependencies
+
+*1.0.0 (November 22, 2008)*
+
+* Initial Release

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Matt Lins
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,169 @@
+= ActiveMigration
+
+Github[http://github.com/mlins/active_migration/]
+
+ActiveMigration is a library to assist with the migration of data from legacy databases.  This library was not designed
+for speed so much as it was designed to maintain data integrity.  By default ActiveMigration runs all data through your
+ActiveRecord validators and callbacks.  It can be extended to run faster with the ar-extenstions library if speed is
+more important than data integrity.
+
+ActiveMigration was written by: Matt Lins.
+
+You'll probably want to use ActiveMigration with the Godwit[http://github.com/mlins/godwit] framework for migrating
+databases.
+
+== Installation
+
+  $ gem sources -a http://gems.github.com
+
+  $ sudo gem install mlins-active_migration
+
+== Terms
+
+I use a couple of terms throughout the codebase and documentation that could be confusing.  I use the term *legacy* to
+refer to the old data that you'll be migrating from.  I use the term *active* to refer the new data that you'll be
+migrating to.  This can be confusing because ActiveMigration makes use of ActiveRecord.  You'll see *active*
+and *legacy* used to refer to:
+
+- databases
+- models
+- records
+- fields
+
+Other terms used:
+
+- *PK* - Primary Key
+- *FK* - Foreign Key
+
+== Usage
+
+Once you have written your migration, you can run it like this:
+
+  MyMigration.new.run
+
+ActiveMigration::Base is intended to be subclassed and defines a simple DSL similar to ActiveRecord.  ActiveMigration
+assumes you have an ActiveRecord class defined for both the legacy model and the active model.  Godwit namespaces
+legacy models with the Legacy module.  You can then map fields with a muli-dimensional array.  Each element of the
+array represents one field mapping.  Within the element array the first element is the legacy field and the second
+element is the active field.
+
+A simple example:
+
+  class PostMigration < ActiveMigration::Base
+
+    set_active_model 'Post'
+
+    set_legacy_model 'Legacy::Post'
+
+    map              [['old_field_name', 'new_field_name']]
+
+  end
+
+ActiveMigration::Callbacks provides callback support via the ActiveSupport library.  You can use callbacks exactly
+as you would in your ActiveRecord models.  This example below also illustrates accessing the record instance
+variables.  You can access both the legacy and active records at anytime during the migration lifcycle via:
+@active_record and @legacy_record.
+
+A callbacks example:
+
+  class PostMigration < ActiveMigration::Base
+
+    set_active_model 'Post'
+
+    set_legacy_model 'Legacy::Post'
+
+    map              [['title_tx',  'title'     ],
+                      ['writer_id', 'author_id']]
+
+    before_save     :upcase_name
+
+    def upcase_name
+      @active_record.name.upcase
+    end
+
+  end
+
+ActiveMigration::Dependencies provides a dependency tree for your migrations.  If you have dependencies set, they'll
+be ran first.
+
+A dependencies example:
+
+  class PostMigration < ActiveMigration::Base
+
+    set_active_model 'Post'
+
+    set_legacy_model 'Legacy::Post'
+
+    map              [['title_tx',  'title'     ],
+                      ['writer_id', 'author_id']]
+
+    set_dependencies [:author_migration]
+
+  end
+
+ActiveMigration::KeyMapper provides a system to persist legacy PK/FK relationships.  It's possible to migrate your
+PK's and FK's through ActiveMigration.  However, sometimes that's not possible or desirable.  The keymapper allows
+you to serialize the PK of the legacy record mapped to the new PK of the active record.  You can then recall that
+mapping (usually with a legacy FK) later in other migrations to maintain relationships.
+
+First you need to serialize the PK of a model.  Let's say your Manufacturer model has_may Products.
+
+  class AuthorMigration < ActiveMigration::Base
+
+    set_active_model 'Author'
+
+    set_legacy_model 'Legacy::Writer'
+
+    map              [['name_tx',   'name'    ],
+                      ['handle_tx', 'nickname']]
+
+    write_key_map    true
+
+  end
+
+Later, in your PostMigration you may need to recall the legacy Author PK to maintain the relationship, that can be
+done like so:
+
+  class PostMigration < ActiveMigration::Base
+
+    set_active_model 'Post'
+
+    set_legacy_model 'Legacy::Post'
+
+    map              [['title_tx',  'title'                       ],
+                      ['writer_id', 'author_id', :author_migration]]
+
+  end
+
+This will lookup the PK of the legacy author by the 'writer_id' and return the new PK assigned to the model when it
+was saved.
+
+== Requirements
+
+- ActiveSupport
+- ActiveRecord
+
+== License
+
+(The MIT License)
+
+Copyright (c) 2008 Matt Lins
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,35 @@
+require 'rake'
+require 'rake/rdoctask'
+require 'spec/rake/spectask'
+
+desc 'Run the specs'
+Spec::Rake::SpecTask.new(:spec) do |t|
+  t.spec_opts = ['--colour --format progress --loadby mtime --reverse']
+  t.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Rake::RDocTask.new do |t|
+  t.rdoc_dir = 'doc'
+  t.rdoc_files.include('README')
+  t.rdoc_files.include('lib/**/*.rb')
+  t.options << '--inline-source'
+  t.options << '--all'
+  t.options << '--line-numbers'
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "active_migration"
+    s.summary = "A library to assist with the migration of data from legacy databases."
+    s.email = "mattlins@gmail.com"
+    s.homepage = "http://github.com/mlins/active_migration"
+    s.description = "A library to assist with the migration of data from legacy databases."
+    s.authors = ["Matt Lins"]
+    s.files = FileList["[A-Z]*", "{lib,spec}/**/*"]
+    s.add_dependency 'activesupport', '>= 2.2.2'
+    s.add_dependency 'activerecord', '>= 2.2.2'
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end

data/VERSION.yml

@@ -0,0 +1,4 @@
+---
+major: 1
+patch: 3
+minor: 0

data/lib/active_migration/base.rb

@@ -0,0 +1,237 @@
+module ActiveMigration
+
+  # Generic ActiveMigration exception class.
+  class ActiveMigrationError < StandardError
+  end
+
+  # ActiveMigration::Base is subclassed by your migration.  It defines a DSL similar to ActiveRecord, in which it feel more
+  # like a configuration.
+  #
+  # == Typical Usage:
+  #
+  #   class PostMigration < ActiveMigration::Base
+  #
+  #     set_active_model 'Post'
+  #
+  #     set_legacy_model 'Legacy::Post'
+  #
+  #     map              [['name_tx',        'name'       ],
+  #                       ['description_tx', 'description'],
+  #                       ['date',           'created_at' ]]
+  #
+  #   end
+  #
+  class Base
+
+    cattr_accessor :logger
+
+    class << self
+
+      attr_accessor :legacy_model, :active_model, :mappings, :legacy_find_options, :active_record_mode
+
+      # Sets the legacy model to be migrated from.  It's wise to namespace your legacy
+      # models to prevent class duplicates.
+      #
+      # Also, *args can be passed a Hash to hold finder options for legacy record lookup.
+      #
+      # *Note:* If you set :limit, it will stagger your selects with an offset. This is intended to break up large datasets
+      # to conserve memory.  Keep in mind, for this functionality to work :offset(because it is needed internally)
+      # can never be specified, it will be deleted.
+      #
+      #   set_legacy_model Legacy::Post
+      #
+      #   set_legacy_model Legacy::Post,
+      #                    :conditions => 'some_field = value',
+      #                    :order => 'this_field ASC',
+      #                    :limit => 5
+      #
+      def set_legacy_model(legacy_model, *args)
+        @legacy_model = eval(legacy_model)
+        args[0].delete(:offset) if args[0]
+        @legacy_find_options = args[0] unless args.empty?
+        @legacy_find_options ||= {}
+      end
+      alias legacy_model= set_legacy_model
+
+      # Sets the active model to be migrated to.
+      #
+      # Also, an additional parameter for the method of instantiation.  Valid
+      # parameters are: :create or :update.  Defaults to :create.  Use this if records already
+      # exist in the active database.  Lookup with :update will be done via the PK of the legacy
+      # record.
+      #
+      #   set_active_model 'Post'
+      #
+      #   set_active_model 'Post',
+      #                    :update
+      #
+      def set_active_model(active_model, mode=:create)
+        @active_model = eval(active_model)
+        @active_record_mode = mode
+      end
+      alias active_model= set_active_model
+
+      # Sets the mappings for the migration.  Mappings are specified in a multidimensional array.  Each array
+      # elment contains another array in which the legacy field is the first element and the active field is
+      # the second elment.
+      #
+      #   map [['some_old_field', 'new_spiffy_field']]
+      #
+      def map(mappings)
+        @mappings = mappings
+      end
+      alias mappings= map
+
+    end
+
+    # Runs the migration.
+    #
+    #   MyMigration.new.run
+    #
+    def run
+      logger.info("#{self.class.to_s} is starting.")
+      count_options = self.class.legacy_find_options.dup
+      count_options.delete(:order)
+      count_options.delete(:group)
+      count_options.delete(:limit)
+      count_options.delete(:offset)
+      @num_of_records = self.class.legacy_model.count(count_options)
+      if self.class.legacy_find_options[:limit] && (@num_of_records > self.class.legacy_find_options[:limit])
+        run_in_batches @num_of_records
+      else
+        run_normal
+      end
+      logger.info("#{self.class.to_s} migrated all #{@num_of_records} records successfully.")
+    end
+
+    protected
+
+    # This is called everytime there is an error.  You should override this method
+    # and handle it in the apporpriate way.
+    #
+    def handle_error()
+    end
+
+    # This is called everytime there is a successful record migration.  You should override this
+    # method and handle it in the appropriate way.
+    #
+    def handle_success()
+    end
+
+    # This method can be called at any point in the in the migration lifecycle (usually within callbacks)
+    # to stop the current record migration and continue on to the next.
+    #
+    def skip
+      @skip = true
+    end
+
+    private
+
+    def run_in_batches(num_of_records) #:nodoc:
+      num_of_last_record = 0
+      while num_of_records > 0 do
+        self.class.legacy_find_options[:offset] = num_of_last_record
+        num_of_last_record += self.class.legacy_find_options[:limit]
+        num_of_records -= self.class.legacy_find_options[:limit]
+        run_normal
+      end
+    end
+
+    def run_normal #:nodoc:
+      legacy_records = self.class.legacy_model.find(:all, self.class.legacy_find_options)
+      legacy_records.each do |@legacy_record|
+        find_or_create_active_record
+        migrate_record
+        unless skip?
+          save
+          unless skip?
+            unless @active_record.is_a?(Array)
+              logger.debug("#{self.class.to_s} successfully migrated a record from #{self.class.legacy_model.table_name} to #{self.class.active_model.table_name}. The legacy record had an id of #{@legacy_record.id}. The active record has an id of #{@active_record.id}")
+            else
+              @active_record.each do |record|
+                logger.debug("#{self.class.to_s} successfully migrated a record from #{self.class.legacy_model.table_name} to #{self.class.active_model.table_name}. The legacy record had an id of #{@legacy_record.id}. The active record has an id of #{record.id}")
+              end
+            end
+          end
+        else
+          handle_success
+        end
+        @skip = false
+      end
+    end
+
+    def find_or_create_active_record #:nodoc:
+      @active_record = (self.class.active_record_mode == :create) ? self.class.active_model.new : self.class.active_model.find(@legacy_record.id)
+    end
+
+    def migrate_record #:nodoc:
+      self.class.mappings.each do |@mapping|
+        migrate_field
+      end unless self.class.mappings.nil?
+    end
+
+    # FIXME - #migrate_field needs to be refactored.
+    def migrate_field #:nodoc:
+      begin
+        eval("@active_record.#{@mapping[1]} = @legacy_record.#{@mapping[0]}")
+      rescue
+        logger.error("#{self.class.to_s} had an error while trying to migrate #{self.class.legacy_model.table_name}.#{@mapping[0]} to #{self.class.active_model.table_name}.#{@mapping[1]}. The legacy record had an id of #{@legacy_record.id}.")
+        handle_error
+      end
+    end
+
+    def save #:nodoc:
+      if self.class.active_record_mode == :create
+        create
+      else
+        update
+      end
+    end
+
+    def create #:nodoc:
+      process_records
+    end
+
+    def update #:nodoc:
+      process_records
+    end
+
+    def process_records #:nodoc:
+      return if skip?
+      unless @active_record.is_a?(Array)
+        save_and_resolve(@active_record)
+      else
+        @active_record.each do |record|
+          save_and_resolve(record)
+          handle_success if skip?
+          @skip = false
+        end
+      end
+      @validate_record = true
+    end
+
+    def save_and_resolve(record) #:nodoc:
+      while record.changed? && !skip?
+        if record.save(validate_record?)
+          handle_success
+        else
+          while !record.valid? && !skip? do
+            errors = record.errors.collect {|field,msg| field + " " + msg}.join(", ")
+            logger.error("#{self.class.to_s} had an error while trying to save the active_record. The associated legacy_record had an id of #{@legacy_record.id}. The active record had the following errors: #{errors}")
+            handle_error
+          end
+          handle_success
+        end
+      end
+    end
+
+    def validate_record? #:nodoc:
+      @validate_record.nil? ? true : @validate_record
+    end
+
+    def skip? #:nodoc:
+      @skip.nil? ? false : @skip
+    end
+
+  end
+end

data/lib/active_migration/callbacks.rb

@@ -0,0 +1,114 @@
+module ActiveMigration
+  # Callbacks are hooks into the ActiveMigration migration lifecycle.  This typical flow is
+  # below.  Bold items are internal calls.
+  #
+  #   - before_run
+  #   - *run*
+  #   - before_migrate_record
+  #   - *migrate_record*
+  #   - after_migrate_record
+  #   - before_migrate_field
+  #   - *migrate_field*
+  #   - after_migrate_field
+  #   - before_save (before_create, before_update)
+  #   - *save*
+  #   - after_save (after_create, after_update)
+  #   - after_run
+  #
+  module Callbacks
+
+    CALLBACKS = %w(before_run after_run
+    before_migrate_record after_migrate_record
+    before_migrate_field after_migrate_field
+    before_save after_save before_create after_create
+    before_update after_update)
+
+    def self.included(base)#:nodoc:
+      [:run, :migrate_record, :migrate_field, :save, :update, :create].each do |method|
+        base.send :alias_method_chain, method, :callbacks
+      end
+      base.send :include, ActiveSupport::Callbacks
+      base.define_callbacks *CALLBACKS
+    end
+
+    # This is called before you anything actually starts.
+    #
+    def before_run() end
+    # This is called after everything else finishes.
+    #
+    def after_run() end
+    def run_with_callbacks #:nodoc:
+      callback(:before_run)
+      run_without_callbacks
+      callback(:after_run)
+    end
+
+    # This is called before the iteration of field migrations.
+    #
+    def before_migrate_record() end
+    # This is called after the iteration of field migrations.
+    #
+    def after_migrate_record() end
+    def migrate_record_with_callbacks #:nodoc:
+      callback(:before_migrate_record)
+      migrate_record_without_callbacks
+      callback(:after_migrate_record)
+    end
+
+    # This is called before each field migration.
+    #
+    def before_migrate_field() end
+    # This is called directly after each field migration.
+    #
+    def after_migrate_field() end
+    def migrate_field_with_callbacks#:nodoc:
+      callback(:before_migrate_field)
+      migrate_field_without_callbacks
+      callback(:after_migrate_field)
+    end
+
+    # This is called directly before the active record is saved.
+    #
+    def before_save() end
+    # This is called directly after the active record is saved.
+    #
+    def after_save() end
+    def save_with_callbacks
+      callback(:before_save)
+      save_without_callbacks
+      callback(:after_save)
+    end
+
+    # This is only called before update if active_record_mode is set to :update.
+    #
+    def before_update() end
+    # This is only called after update if active_record_mode is set to :update.
+    #
+    def after_update() end
+    def update_with_callbacks
+      callback(:before_update)
+      update_without_callbacks
+      callback(:after_update)
+    end
+
+    # This is only called before create if active_record_mode is set to :create(default).
+    #
+    def before_create() end
+    # This is only called after update if active_record_mode is set to :create(default).
+    #
+    def after_create() end
+    def create_with_callbacks
+      callback(:before_create)
+      create_without_callbacks
+      callback(:after_create)
+    end
+
+    private
+
+    def callback(method) #:nodoc:
+      run_callbacks(method)
+      send(method)
+    end
+
+  end
+end

data/lib/active_migration/dependencies.rb

@@ -0,0 +1,53 @@
+module ActiveMigration
+  # Dependencies are supported by ActiveMigration in this module.  If you set some dependencies
+  # they'll be ran before Base#run is called.  Specifying dependencies is easy:
+  #
+  #   set_dependencies  [:supplier_migration, :manufacturer_migration]
+  #
+  module Dependencies
+
+    def self.included(base)#:nodoc:
+      base.class_eval do
+        alias_method_chain :run, :dependencies
+        class << self
+          attr_accessor :dependencies, :completed
+          # Sets the dependencies for the migration
+          #
+          #   set_dependencies  [:supplier_migration, :manufacturer_migration]
+          def set_dependencies(dependencies)
+            @dependencies = dependencies
+          end
+          alias dependencies= set_dependencies
+          def completed? #:nodoc:
+            @completed
+          end
+          def is_completed #:nodoc:
+            @completed = true
+          end
+          alias completed completed?
+          def dependencies #:nodoc:
+            @dependencies || []
+          end
+        end
+      end
+    end
+
+    def run_with_dependencies(skip_dependencies=false) #:nodoc:
+      if skip_dependencies
+        logger.info("#{self.class.to_s} is skipping dependencies.")
+        run_without_dependencies
+      else
+        self.class.dependencies.each do |dependency|
+          migration = dependency.to_s.camelize.constantize
+          unless migration.completed?
+            logger.info("#{self.class.to_s} is running #{migration.to_s} as a dependency.")
+            migration.new.run
+            migration.is_completed
+          end
+        end
+        run_without_dependencies unless self.class.completed?
+      end
+    end
+
+  end
+end