migratrix 0.8.5 → 0.8.7

data/README.md

@@ -4,17 +4,89 @@ Dominate your legacy Rails migrations! Migratrix is a gem to help you
 generate and control Migrations, which extract data from legacy systems
 and import them into your current one.
 
-## Warning: Experimental Developmental In-Progress Stuff
+## General Info
 
-I am currently extracting Migratrix from an ancient legacy codebase.
-(Oh the irony.) A lot of the stuff I say Migratrix supports is stuff
-that it supports over there, not in here. Be aware that most of this
-document is more of a TODO list than a statement of fact. I'll remove
-this message once Migratrix does what it says on the tin.
+Migratrix is a framework that supports various migration strategies.
+You tell it how you want to connect to a data source, define any
+transformations, and then describe the "load target": how and where
+you want the new data to come out. You can, of course, extract data
+from a legacy database and load it into your all-new database, but you
+can also read and write from logs, flat files, or even external
+services.
 
-## General Info
+If you can get at the data from Ruby, Migratrix can use it as a source
+of extractable data. If you can get at the new storage mechanism from
+Ruby, Migratrix can use it as a load target.
+
+You can extract data from an API and load it into your database. You
+can extract enumerables from a database and load it to source code
+that defines constants for the enumeration.
+
+## Weapons-Grade Migrations
+
+Migratrix tries to keep simple things simple, but really shines when
+it scales.
+
+Migratrix is intended to be as simple and easy to use as possible for
+small cases, but to have no problems scaling upwards. As a result if
+you have an ultra-simple migration that one developer on the team will
+only ever run once, Migratrix will work but may not be the best tool.
+
+If, however, you have
+
+* Complex and/or complicated migrations
+
+* Migrations that need to be run more than once, especially if they
+  need to be run frequently and/or regularly
+
+* Migrations that need to be run by multiple developers or by
+  distributed machines
+
+then your migration strategy is complicated enough that it needs to be
+part of the codebase and knowledgebase for your project, and you want
+a heavier-duty migration tool. Migratrix is the perfect tool in that
+situation.
+
+
+## Object-Oriented Data Transformation
+
+The first few times you write a special-purpose data migration tool,
+you're going to be tempted to just say "here's a hash for the source,
+here's a hash for the destination, and here's a function to transform
+from one to the other". And it will work. But then the extra
+requirements start to roll in:
+
+* We have 20 million rows, can you do the migration in batches?
+
+* We need to split the users table into users and addresses tables,
+  how hard is that?
 
-Migratrix is a legacy migration tool strategy
+* We have a TON of duplicate data, can you merge it down when you
+  migrate?
+
+* Can we just migrate one or two exemplar rows to test the migration
+  tool?
+
+And my all-time personal favorite:
+
+* We've decided we want to keep the old site live while the beta site
+  is running, can you keep the databases in sync?
+
+I started writing Migratrix when my third client asked me to keep a
+legacy site and a beta site synchronized. For me this tool reduced the
+problem from "utterly impossible" to "merely very difficult". :-)
+
+As a result, Migratrix is very much object based. You get at a data
+source by using an Extract object. If Migratrix does not support the
+kind of extraction you want to do, it provides the Extract base class
+so you can write your own extractor.
+
+Then the data is handed off to a Transform object, and finally it is
+given to a Load object.
+
+Migratrix provides a handful of Extract, Transform and Load classes to
+handle common types of migration (reading and writing models with
+ActiveRecord, writing YAML files, etc).
 
 ## Motivation
 
@@ -22,42 +94,116 @@ So... much... legacy... data....
 
 ## Rails and Ruby Requirements
 
-### Rails 3 Only
+### Rails 3
 
-Migratrix was originally developed under Rails 2, but come on. Rails 2
-apps are legacy SOURCES now, not destinations. Migratrix requires
-Rails 3. Once everything's in place I'll bump the Migratrix version to
-3.x to indicate that Migratrix is in keeping with Rails 3.
+Migratrix depends on Rails 3 for its ActiveRecord and ActiveSupport
+libraries. If your project was written in an older version of Rails,
+and you want to use ActiveRecord as your extractor, you may need to
+define new models that are compatible with Rails 3. But try just
+loading the old models first; as long as the models aren't too
+complicated or interconnected, they may work.
 
-### Ruby 1.9
+### Ruby 1.9.2 or later
 
 Because I can.
 
-## Example
+## Examples
 
-## ETL
+### ETL
 
 I use the term "ETL" here in a loosely similar mechanism as in data
 warehousing: Extract, Transform and Load. Migratrix approaches
 migrations in three phases:
 
-* **Extract** The Migration obtains the legacy data from 1 or more
-    sources
-    
-* **Transform** The Migration transforms the data into 1 or more
-    outputs
-    
+* **Extract** The Migration obtains the legacy data from 1 or more sources
+
+* **Transform** The Migration transforms the data into 1 or more outputs
+
 * **Load** The Migration saves the data into the new database or other
-    output(s)
-    
+output(s)
+
+### General Structure
+
+I like to create a folder structure in db/legacy with a /migrations
+folder and a /models folder. Then I write rake tasks to handle the
+more common migrations. Migration classes go in /migrations,
+naturally; /models is where I store ActiveRecord models that must
+access the legacy database. It is important to keep these namespaced
+so that your `User`model doesn't conflict with your `Legacy::User`
+model.
+
+### Simple Example: Straight-up ActiveRecord Copy
+
+Let's say your legacy app has a simple table that you want to keep
+unchanged. Migratrix can bring this over using straight ActiveRecord
+copies:
+
+    TODO: Write Sample App so these examples make sense
+    TODO: And then come write this example
+
+
+### Slightly More Complicated: Defining your own extensible class
+
+Let's say your legacy app has a table that stores what amount to
+constants in a table in the database. Either the data never changes
+(in fact, you don't even have an admin page to edit the data), or the
+data changes rarely enough that you're willing to restart the
+webserver if the data DOES change. It's also really simple data, and
+has no dependencies. For the sake of debmonstration, let's say that
+your app stores all 50 US States and their abbreviations. (And you put
+it in the database because ONE day you were SURE you were going to
+need to add Canadian provinces or Japanese boroughs.)
+
+Let's migrate this to a YAML file and store it in config/constants,
+and at boot time we'll write some code to load the states.yml file and
+create a frozen hash called MyApp::STATES.
+
+This should be straightforward at this point, but wait. It also turns
+out we need to do the same thing with the countries table, the
+shipping_carriers table, and about six other tables. They're all the
+same: we need to extract from the legacy database and write to a
+yaml file in the constants folder.
+
+We can do that. Here's the entire migration for States:
+
+    require 'constants_migration'
+    module Legacy
+      class StatesMigration < ConstantsMigration
+        extend_extraction source: Legacy::State
+        extend_transform transform: { id: :id, name: :name, abbreviation: :state_code }
+        extend_load filename: MyApp::CONSTANTS_PATH + "states.yml"
+      end
+    end
+
+And the reason this works is that you've written your own custom
+Migration class, which looks like this:
+
+    module Legacy
+      # Migrates a Legacy ActiveRecord table through a transform to a
+      # constants file. Child classes should extend_extraction with
+      # :source, transform with a :transform hash, and load with the
+      # filename to write to.
+      class ConstantsMigration < Migratrix::Migration
+        set_extraction :active_record, source: Model
+        set_transform :transform, {
+          transform_collection: Hash,
+          transform_class:  Hash,
+          extract_attribute:  :[],
+          apply_attribute: :[]=,
+          store_transformed_object: ->(object,collection){  collection[object[:id]] = object }
+        }
+        set_load :yaml
+      end
+    end
 
 ## Migration Dependencies
 
 Migratrix isn't quite smart enough to know that a migrator depends on
 another migrator. (Actually, that's easy. What's hard is knowing if a
-dependent migrator has run and is up-to-date.)
+dependent migrator has run and is up-to-date. This is a solvable
+problem, I just haven't needed to solve it with Migratrix yet.)
 
-## Strategies
+## Migration Strategies
 
 Migratrix supports multiple migration strategies:
 
@@ -89,9 +235,8 @@ Migratrix supports multiple migration strategies:
 ## Slices of Data
 
 Migratrix supports taking partial slices of data. You can migrate a
-single record to test a migraton, grab 100 records or 1000 to get an
-idea for how long a full migration will take, or perform the entire
-migration.
+single record to test a migraton, or grab a few thousand to get an
+idea for how long a full migration will take.
 
 ## Ongoing Migrations
 
@@ -107,7 +252,9 @@ can also record the source object, which is useful for debugging or
 handling migration cases where legacy records get changed or deleted
 after migrating.
 
-## Migration Tests
+## Known Limitations and Issues
+
+### Migration Tests
 
 Sorry, nothing to see here yet. Migratrix was originally developed in
 an environment where the migrations were so heavy-duty and hairy that
@@ -118,6 +265,12 @@ mostly a note to remind myself that that heavy-duty migrations can and
 should be developed in a TDD style, and Migratrix should make this
 easy.
 
+### Rake Tasks and/or Rails Generators
+
+Migratrix is definitely a heavyweight tool for migrating data. It
+would be nice if the gem provided rake tasks or Rails generators to
+help with the boilerplate.
+
 ## A note about the name
 
 In old Latin, -or versus -ix endings aren't just about feminine and
@@ -142,3 +295,10 @@ MIT. See the license file.
 
 * David Brady -- github@shinybit.com
 
+## Contributing
+
+YES PLEASE! For bugs and other issues, send a pull request. If you
+would like to extend or change a feature of Migratrix, discussion is
+also very welcome.
+
+

data/bin/migratrix

@@ -1,2 +1,2 @@
 #!/usr/bin/env ruby
-puts "Nothing to see here yet -- use rake tasks to call Migratrix.migrate!(:migrator, options) directly"
+puts "Nothing to see here yet -- call Migratrix.migrate!(:migrator, options) directly from your own driver code (scripts, rake tasks, etc)"

data/lib/migratrix.rb

@@ -29,6 +29,7 @@ module Migratrix
   require APP + 'transforms/map'
 
   require APP + 'loads/load'
+  require APP + 'loads/active_record'
   require APP + 'loads/no_op'
   require APP + 'loads/yaml'
 #  require APP + 'loads/csv'
@@ -84,6 +85,7 @@ module Migratrix
   register_transform :no_op, Transforms::NoOp
 
   register_load :load, Loads::Load
+  register_load :active_record, Loads::ActiveRecord
   register_load :no_op, Loads::NoOp
   register_load :yaml, Loads::Yaml
 

data/lib/migratrix/loads/active_record.rb

@@ -0,0 +1,73 @@
+require 'yaml'
+
+module Migratrix
+  module Loads
+    # An ActiveRecord-based Load that tries to update existing objects
+    # rather than always doing new saves. If :update is true, before
+    # saving we attempt to find the object by the primary_key column.
+    # If found, we call .update_attributes on that record instead of
+    # .save.
+    #
+    # TODO: Verify that update_attributes still calls callbacks, e.g.
+    # validations and before_save? If not we'll need to load the
+    # object, copy attributes manually from the transformed object,
+    # and save it.
+    #
+    # TODO: primary_key is a bit presumptive. Would be better if it
+    # were a where clause.
+    class ActiveRecord < Load
+      set_valid_options :primary_key, :legacy_key, :finder, :update, :cache_key
+
+      def seen
+        @seen ||= { }
+      end
+
+      def seen?(object)
+        if options[:cache_key]
+          seen[object[options[:cache_key]]]
+        end
+      end
+
+      def seen!(object)
+        if options[:cache_key]
+          seen[object[options[:cache_key]]] = true
+        end
+      end
+
+      def load(transformed_objects)
+        transformed_objects.each do |transformed_object|
+          next if seen?(transformed_object)
+          if options[:update]
+            object = if options[:finder]
+                       options[:finder].call(transformed_object)
+                     elsif options[:primary_key] && options[:legacy_key]
+                       transformed_object.class.where("#{options[:primary_key]}=?", transformed_object[options[:legacy_key]]).first
+                     end
+            if object
+              update_object object, transformed_object
+            else
+              save_object transformed_object
+            end
+          else
+            save_object transformed_object
+          end
+        end
+        transformed_objects
+      end
+
+      def save_object(transformed_object)
+        return if seen?(transformed_object)
+        transformed_object.save
+        seen! transformed_object
+        transformed_object
+      end
+
+      def update_object(original_object, transformed_object)
+        return if seen?(transformed_object)
+        original_object.update_attributes transformed_object.attributes
+        seen! original_object
+        original_object
+      end
+    end
+  end
+end

data/lib/migratrix/migration.rb

@@ -34,11 +34,11 @@ module Migratrix
           opts += transform.valid_options
         end
       end
-#       if loads
-#         loads.each do |name, load|
-#           opts += load.valid_options
-#         end
-#       end
+      if loads
+        loads.each do |name, load|
+          opts += load.valid_options
+        end
+      end
       opts.uniq.sort
     end
 
@@ -86,6 +86,8 @@ module Migratrix
       self.class.extractions
     end
 
+    # TODO: THIS IS HUGE DUPLICATION, REFACTOR REFACTOR REFACTOR
+
     # transform crap
     # set_transform :nickname, :registered_name, options_hash
     # set_transform :nickname, :registered_name # options = {}
@@ -128,6 +130,8 @@ module Migratrix
       self.class.transforms
     end
 
+    # TODO: THIS IS HUGE DUPLICATION, REFACTOR REFACTOR REFACTOR
+
     # load crap
     # set_load :nickname, :registered_name, options_hash
     # set_load :nickname, :registered_name # options = {}

data/lib/migratrix/transforms/transform.rb

@@ -78,18 +78,6 @@ module Migratrix
       # parts.
       #
       # ----------------------------------------------------------------------
-      # Map's strategy, as used by PetsMigration
-      #
-      # create_transformed_collection -> Hash.new
-      # create_new_object -> Hash.new
-      # transformation -> {:id => :id, :name => :name }
-      # extract_attribute -> object[attribute_or_extract]
-      # apply_attribute -> object[attribute] = attribute_or_apply
-      # finalize_object -> no-op
-      # store_transformed_object -> collection[object[:id]] = object
-      # ----------------------------------------------------------------------
-      #
-      # ----------------------------------------------------------------------
       # Default strategy:
       #
       # create_transformed_collection -> Array.new

data/spec/lib/migratrix/migration_spec.rb

@@ -170,7 +170,7 @@ describe Migratrix::Migration do
 
     describe ".valid_options" do
       it "returns valid options from itself and components" do
-        TestMigration.valid_options.should == [:console, :fetchall, :limit, :map, :offset, :order, :where]
+        TestMigration.valid_options.should == [:console, :fetchall, :filename, :limit, :map, :offset, :order, :where]
       end
     end
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: migratrix
 version: !ruby/object:Gem::Version
-  version: 0.8.5
+  version: 0.8.7
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2011-10-31 00:00:00.000000000Z
+date: 2012-04-20 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: trollop
-  requirement: &2164754380 !ruby/object:Gem::Requirement
+  requirement: &2152597120 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,7 +21,7 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *2164754380
+  version_requirements: *2152597120
 description: Migratrix, a Rails legacy database migration tool supporting multiple
   strategies, including arbitrary n-ary migrations (1->n, n->1, n->m), arbitrary inputs
   and outputs (ActiveRecord, bare SQL, CSV) and migration logging
@@ -42,6 +42,7 @@ files:
 - lib/migratrix/extractions/active_record.rb
 - lib/migratrix/extractions/extraction.rb
 - lib/migratrix/extractions/no_op.rb
+- lib/migratrix/loads/active_record.rb
 - lib/migratrix/loads/load.rb
 - lib/migratrix/loads/no_op.rb
 - lib/migratrix/loads/yaml.rb
@@ -102,7 +103,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.6
+rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: Rails 3 legacy database migratrion tool supporting multiple strategies