mlins-active_migration 1.0.2 → 1.0.3

Sign up to get free protection for your applications and to get access to all the features.
data/CHANGELOG ADDED
@@ -0,0 +1,16 @@
1
+ *1.0.3 (November 22, 2008)*
2
+
3
+ * Fixed gemspec problems
4
+
5
+ *1.0.2 (November 22, 2008)*
6
+
7
+ * Renamed gem back to active_migration
8
+
9
+ *1.0.1 (November 22, 2008)*
10
+
11
+ * Renamed gem to activemigration
12
+ * Fixed dependencies
13
+
14
+ *1.0.0 (November 22, 2008)*
15
+
16
+ * Initial Release
data/MIT-LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2008 Matt Lins
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README ADDED
@@ -0,0 +1,169 @@
1
+ = ActiveMigration
2
+
3
+ Github[http://github.com/mlins/active_migration/]
4
+
5
+ ActiveMigration is a library to assist with the migration of data from legacy databases. This library was not designed
6
+ for speed so much as it was designed to maintain data integrity. By default ActiveMigration runs all data through your
7
+ ActiveRecord validators and callbacks. It can be extended to run faster with the ar-extenstions library if speed is
8
+ more important than data integrity.
9
+
10
+ ActiveMigration was written by: Matt Lins.
11
+
12
+ You'll probably want to use ActiveMigration with the Godwit[http://github.com/mlins/godwit] framework for migrating
13
+ databases.
14
+
15
+ == Installation
16
+
17
+ $ gem sources -a http://gems.github.com
18
+
19
+ $ sudo gem install mlins-active_migration
20
+
21
+ == Terms
22
+
23
+ I use a couple of terms throughout the codebase and documentation that could be confusing. I use the term *legacy* to
24
+ refer to the old data that you'll be migrating from. I use the term *active* to refer the new data that you'll be
25
+ migrating to. This can be confusing because ActiveMigration makes use of ActiveRecord. You'll see *active*
26
+ and *legacy* used to refer to:
27
+
28
+ - databases
29
+ - models
30
+ - records
31
+ - fields
32
+
33
+ Other terms used:
34
+
35
+ - *PK* - Primary Key
36
+ - *FK* - Foreign Key
37
+
38
+ == Usage
39
+
40
+ Once you have written your migration, you can run it like this:
41
+
42
+ MyMigration.new.run
43
+
44
+ ActiveMigration::Base is intended to be subclassed and defines a simple DSL similar to ActiveRecord. ActiveMigration
45
+ assumes you have an ActiveRecord class defined for both the legacy model and the active model. Godwit namespaces
46
+ legacy models with the Legacy module. You can then map fields with a muli-dimensional array. Each element of the
47
+ array represents one field mapping. Within the element array the first element is the legacy field and the second
48
+ element is the active field.
49
+
50
+ A simple example:
51
+
52
+ class PostMigration < ActiveMigration::Base
53
+
54
+ set_active_model 'Post'
55
+
56
+ set_legacy_model 'Legacy::Post'
57
+
58
+ map [['old_field_name', 'new_field_name']]
59
+
60
+ end
61
+
62
+ ActiveMigration::Callbacks provides callback support via the ActiveSupport library. You can use callbacks exactly
63
+ as you would in your ActiveRecord models. This example below also illustrates accessing the record instance
64
+ variables. You can access both the legacy and active records at anytime during the migration lifcycle via:
65
+ @active_record and @legacy_record.
66
+
67
+ A callbacks example:
68
+
69
+ class PostMigration < ActiveMigration::Base
70
+
71
+ set_active_model 'Post'
72
+
73
+ set_legacy_model 'Legacy::Post'
74
+
75
+ map [['title_tx', 'title' ],
76
+ ['writer_id', 'author_id']]
77
+
78
+ before_save :upcase_name
79
+
80
+ def upcase_name
81
+ @active_record.name.upcase
82
+ end
83
+
84
+ end
85
+
86
+ ActiveMigration::Dependencies provides a dependency tree for your migrations. If you have dependencies set, they'll
87
+ be ran first.
88
+
89
+ A dependencies example:
90
+
91
+ class PostMigration < ActiveMigration::Base
92
+
93
+ set_active_model 'Post'
94
+
95
+ set_legacy_model 'Legacy::Post'
96
+
97
+ map [['title_tx', 'title' ],
98
+ ['writer_id', 'author_id']]
99
+
100
+ set_dependencies [:author_migration]
101
+
102
+ end
103
+
104
+ ActiveMigration::KeyMapper provides a system to persist legacy PK/FK relationships. It's possible to migrate your
105
+ PK's and FK's through ActiveMigration. However, sometimes that's not possible or desirable. The keymapper allows
106
+ you to serialize the PK of the legacy record mapped to the new PK of the active record. You can then recall that
107
+ mapping (usually with a legacy FK) later in other migrations to maintain relationships.
108
+
109
+ First you need to serialize the PK of a model. Let's say your Manufacturer model has_may Products.
110
+
111
+ class AuthorMigration < ActiveMigration::Base
112
+
113
+ set_active_model 'Author'
114
+
115
+ set_legacy_model 'Legacy::Writer'
116
+
117
+ map [['name_tx', 'name' ],
118
+ ['handle_tx', 'nickname']]
119
+
120
+ write_key_map true
121
+
122
+ end
123
+
124
+ Later, in your PostMigration you may need to recall the legacy Author PK to maintain the relationship, that can be
125
+ done like so:
126
+
127
+ class PostMigration < ActiveMigration::Base
128
+
129
+ set_active_model 'Post'
130
+
131
+ set_legacy_model 'Legacy::Post'
132
+
133
+ map [['title_tx', 'title' ],
134
+ ['writer_id', 'author_id', :author_migration]]
135
+
136
+ end
137
+
138
+ This will lookup the PK of the legacy author by the 'writer_id' and return the new PK assigned to the model when it
139
+ was saved.
140
+
141
+ == Requirements
142
+
143
+ - ActiveSupport
144
+ - ActiveRecord
145
+
146
+ == License
147
+
148
+ (The MIT License)
149
+
150
+ Copyright (c) 2008 Matt Lins
151
+
152
+ Permission is hereby granted, free of charge, to any person obtaining
153
+ a copy of this software and associated documentation files (the
154
+ 'Software'), to deal in the Software without restriction, including
155
+ without limitation the rights to use, copy, modify, merge, publish,
156
+ distribute, sublicense, and/or sell copies of the Software, and to
157
+ permit persons to whom the Software is furnished to do so, subject to
158
+ the following conditions:
159
+
160
+ The above copyright notice and this permission notice shall be
161
+ included in all copies or substantial portions of the Software.
162
+
163
+ THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
164
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
165
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
166
+ IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
167
+ CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
168
+ TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
169
+ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/Rakefile ADDED
@@ -0,0 +1,35 @@
1
+ require 'rake'
2
+ require 'rake/rdoctask'
3
+ require 'spec/rake/spectask'
4
+
5
+ desc 'Run the specs'
6
+ Spec::Rake::SpecTask.new(:spec) do |t|
7
+ t.spec_opts = ['--colour --format progress --loadby mtime --reverse']
8
+ t.spec_files = FileList['spec/**/*_spec.rb']
9
+ end
10
+
11
+ Rake::RDocTask.new do |t|
12
+ t.rdoc_dir = 'doc'
13
+ t.rdoc_files.include('README')
14
+ t.rdoc_files.include('lib/**/*.rb')
15
+ t.options << '--inline-source'
16
+ t.options << '--all'
17
+ t.options << '--line-numbers'
18
+ end
19
+
20
+ begin
21
+ require 'jeweler'
22
+ Jeweler::Tasks.new do |s|
23
+ s.name = "active_migration"
24
+ s.summary = "A library to assist with the migration of data from legacy databases."
25
+ s.email = "mattlins@gmail.com"
26
+ s.homepage = "http://github.com/mlins/active_migration"
27
+ s.description = "A library to assist with the migration of data from legacy databases."
28
+ s.authors = ["Matt Lins"]
29
+ s.files = FileList["[A-Z]*", "{lib,spec}/**/*"]
30
+ s.add_dependency 'activesupport', '>= 2.2.2'
31
+ s.add_dependency 'activerecord', '>= 2.2.2'
32
+ end
33
+ rescue LoadError
34
+ puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
35
+ end
data/VERSION.yml ADDED
@@ -0,0 +1,4 @@
1
+ ---
2
+ major: 1
3
+ patch: 3
4
+ minor: 0
@@ -0,0 +1,237 @@
1
+ module ActiveMigration
2
+
3
+ # Generic ActiveMigration exception class.
4
+ class ActiveMigrationError < StandardError
5
+ end
6
+
7
+ # ActiveMigration::Base is subclassed by your migration. It defines a DSL similar to ActiveRecord, in which it feel more
8
+ # like a configuration.
9
+ #
10
+ # == Typical Usage:
11
+ #
12
+ # class PostMigration < ActiveMigration::Base
13
+ #
14
+ # set_active_model 'Post'
15
+ #
16
+ # set_legacy_model 'Legacy::Post'
17
+ #
18
+ # map [['name_tx', 'name' ],
19
+ # ['description_tx', 'description'],
20
+ # ['date', 'created_at' ]]
21
+ #
22
+ # end
23
+ #
24
+ class Base
25
+
26
+ cattr_accessor :logger
27
+
28
+ class << self
29
+
30
+ attr_accessor :legacy_model, :active_model, :mappings, :legacy_find_options, :active_record_mode
31
+
32
+ # Sets the legacy model to be migrated from. It's wise to namespace your legacy
33
+ # models to prevent class duplicates.
34
+ #
35
+ # Also, *args can be passed a Hash to hold finder options for legacy record lookup.
36
+ #
37
+ # *Note:* If you set :limit, it will stagger your selects with an offset. This is intended to break up large datasets
38
+ # to conserve memory. Keep in mind, for this functionality to work :offset(because it is needed internally)
39
+ # can never be specified, it will be deleted.
40
+ #
41
+ # set_legacy_model Legacy::Post
42
+ #
43
+ # set_legacy_model Legacy::Post,
44
+ # :conditions => 'some_field = value',
45
+ # :order => 'this_field ASC',
46
+ # :limit => 5
47
+ #
48
+ def set_legacy_model(legacy_model, *args)
49
+ @legacy_model = eval(legacy_model)
50
+ args[0].delete(:offset) if args[0]
51
+ @legacy_find_options = args[0] unless args.empty?
52
+ @legacy_find_options ||= {}
53
+ end
54
+ alias legacy_model= set_legacy_model
55
+
56
+ # Sets the active model to be migrated to.
57
+ #
58
+ # Also, an additional parameter for the method of instantiation. Valid
59
+ # parameters are: :create or :update. Defaults to :create. Use this if records already
60
+ # exist in the active database. Lookup with :update will be done via the PK of the legacy
61
+ # record.
62
+ #
63
+ # set_active_model 'Post'
64
+ #
65
+ # set_active_model 'Post',
66
+ # :update
67
+ #
68
+ def set_active_model(active_model, mode=:create)
69
+ @active_model = eval(active_model)
70
+ @active_record_mode = mode
71
+ end
72
+ alias active_model= set_active_model
73
+
74
+ # Sets the mappings for the migration. Mappings are specified in a multidimensional array. Each array
75
+ # elment contains another array in which the legacy field is the first element and the active field is
76
+ # the second elment.
77
+ #
78
+ # map [['some_old_field', 'new_spiffy_field']]
79
+ #
80
+ def map(mappings)
81
+ @mappings = mappings
82
+ end
83
+ alias mappings= map
84
+
85
+ end
86
+
87
+ # Runs the migration.
88
+ #
89
+ # MyMigration.new.run
90
+ #
91
+ def run
92
+ logger.info("#{self.class.to_s} is starting.")
93
+ count_options = self.class.legacy_find_options.dup
94
+ count_options.delete(:order)
95
+ count_options.delete(:group)
96
+ count_options.delete(:limit)
97
+ count_options.delete(:offset)
98
+ @num_of_records = self.class.legacy_model.count(count_options)
99
+ if self.class.legacy_find_options[:limit] && (@num_of_records > self.class.legacy_find_options[:limit])
100
+ run_in_batches @num_of_records
101
+ else
102
+ run_normal
103
+ end
104
+ logger.info("#{self.class.to_s} migrated all #{@num_of_records} records successfully.")
105
+ end
106
+
107
+ protected
108
+
109
+ # This is called everytime there is an error. You should override this method
110
+ # and handle it in the apporpriate way.
111
+ #
112
+ def handle_error()
113
+ end
114
+
115
+ # This is called everytime there is a successful record migration. You should override this
116
+ # method and handle it in the appropriate way.
117
+ #
118
+ def handle_success()
119
+ end
120
+
121
+ # This method can be called at any point in the in the migration lifecycle (usually within callbacks)
122
+ # to stop the current record migration and continue on to the next.
123
+ #
124
+ def skip
125
+ @skip = true
126
+ end
127
+
128
+ private
129
+
130
+ def run_in_batches(num_of_records) #:nodoc:
131
+ num_of_last_record = 0
132
+ while num_of_records > 0 do
133
+ self.class.legacy_find_options[:offset] = num_of_last_record
134
+ num_of_last_record += self.class.legacy_find_options[:limit]
135
+ num_of_records -= self.class.legacy_find_options[:limit]
136
+ run_normal
137
+ end
138
+ end
139
+
140
+ def run_normal #:nodoc:
141
+ legacy_records = self.class.legacy_model.find(:all, self.class.legacy_find_options)
142
+ legacy_records.each do |@legacy_record|
143
+ find_or_create_active_record
144
+ migrate_record
145
+ unless skip?
146
+ save
147
+ unless skip?
148
+ unless @active_record.is_a?(Array)
149
+ logger.debug("#{self.class.to_s} successfully migrated a record from #{self.class.legacy_model.table_name} to #{self.class.active_model.table_name}. The legacy record had an id of #{@legacy_record.id}. The active record has an id of #{@active_record.id}")
150
+ else
151
+ @active_record.each do |record|
152
+ logger.debug("#{self.class.to_s} successfully migrated a record from #{self.class.legacy_model.table_name} to #{self.class.active_model.table_name}. The legacy record had an id of #{@legacy_record.id}. The active record has an id of #{record.id}")
153
+ end
154
+ end
155
+ end
156
+ else
157
+ handle_success
158
+ end
159
+ @skip = false
160
+ end
161
+ end
162
+
163
+ def find_or_create_active_record #:nodoc:
164
+ @active_record = (self.class.active_record_mode == :create) ? self.class.active_model.new : self.class.active_model.find(@legacy_record.id)
165
+ end
166
+
167
+ def migrate_record #:nodoc:
168
+ self.class.mappings.each do |@mapping|
169
+ migrate_field
170
+ end unless self.class.mappings.nil?
171
+ end
172
+
173
+ # FIXME - #migrate_field needs to be refactored.
174
+ def migrate_field #:nodoc:
175
+ begin
176
+ eval("@active_record.#{@mapping[1]} = @legacy_record.#{@mapping[0]}")
177
+ rescue
178
+ logger.error("#{self.class.to_s} had an error while trying to migrate #{self.class.legacy_model.table_name}.#{@mapping[0]} to #{self.class.active_model.table_name}.#{@mapping[1]}. The legacy record had an id of #{@legacy_record.id}.")
179
+ handle_error
180
+ end
181
+ end
182
+
183
+ def save #:nodoc:
184
+ if self.class.active_record_mode == :create
185
+ create
186
+ else
187
+ update
188
+ end
189
+ end
190
+
191
+ def create #:nodoc:
192
+ process_records
193
+ end
194
+
195
+ def update #:nodoc:
196
+ process_records
197
+ end
198
+
199
+ def process_records #:nodoc:
200
+ return if skip?
201
+ unless @active_record.is_a?(Array)
202
+ save_and_resolve(@active_record)
203
+ else
204
+ @active_record.each do |record|
205
+ save_and_resolve(record)
206
+ handle_success if skip?
207
+ @skip = false
208
+ end
209
+ end
210
+ @validate_record = true
211
+ end
212
+
213
+ def save_and_resolve(record) #:nodoc:
214
+ while record.changed? && !skip?
215
+ if record.save(validate_record?)
216
+ handle_success
217
+ else
218
+ while !record.valid? && !skip? do
219
+ errors = record.errors.collect {|field,msg| field + " " + msg}.join(", ")
220
+ logger.error("#{self.class.to_s} had an error while trying to save the active_record. The associated legacy_record had an id of #{@legacy_record.id}. The active record had the following errors: #{errors}")
221
+ handle_error
222
+ end
223
+ handle_success
224
+ end
225
+ end
226
+ end
227
+
228
+ def validate_record? #:nodoc:
229
+ @validate_record.nil? ? true : @validate_record
230
+ end
231
+
232
+ def skip? #:nodoc:
233
+ @skip.nil? ? false : @skip
234
+ end
235
+
236
+ end
237
+ end
@@ -0,0 +1,114 @@
1
+ module ActiveMigration
2
+ # Callbacks are hooks into the ActiveMigration migration lifecycle. This typical flow is
3
+ # below. Bold items are internal calls.
4
+ #
5
+ # - before_run
6
+ # - *run*
7
+ # - before_migrate_record
8
+ # - *migrate_record*
9
+ # - after_migrate_record
10
+ # - before_migrate_field
11
+ # - *migrate_field*
12
+ # - after_migrate_field
13
+ # - before_save (before_create, before_update)
14
+ # - *save*
15
+ # - after_save (after_create, after_update)
16
+ # - after_run
17
+ #
18
+ module Callbacks
19
+
20
+ CALLBACKS = %w(before_run after_run
21
+ before_migrate_record after_migrate_record
22
+ before_migrate_field after_migrate_field
23
+ before_save after_save before_create after_create
24
+ before_update after_update)
25
+
26
+ def self.included(base)#:nodoc:
27
+ [:run, :migrate_record, :migrate_field, :save, :update, :create].each do |method|
28
+ base.send :alias_method_chain, method, :callbacks
29
+ end
30
+ base.send :include, ActiveSupport::Callbacks
31
+ base.define_callbacks *CALLBACKS
32
+ end
33
+
34
+ # This is called before you anything actually starts.
35
+ #
36
+ def before_run() end
37
+ # This is called after everything else finishes.
38
+ #
39
+ def after_run() end
40
+ def run_with_callbacks #:nodoc:
41
+ callback(:before_run)
42
+ run_without_callbacks
43
+ callback(:after_run)
44
+ end
45
+
46
+ # This is called before the iteration of field migrations.
47
+ #
48
+ def before_migrate_record() end
49
+ # This is called after the iteration of field migrations.
50
+ #
51
+ def after_migrate_record() end
52
+ def migrate_record_with_callbacks #:nodoc:
53
+ callback(:before_migrate_record)
54
+ migrate_record_without_callbacks
55
+ callback(:after_migrate_record)
56
+ end
57
+
58
+ # This is called before each field migration.
59
+ #
60
+ def before_migrate_field() end
61
+ # This is called directly after each field migration.
62
+ #
63
+ def after_migrate_field() end
64
+ def migrate_field_with_callbacks#:nodoc:
65
+ callback(:before_migrate_field)
66
+ migrate_field_without_callbacks
67
+ callback(:after_migrate_field)
68
+ end
69
+
70
+ # This is called directly before the active record is saved.
71
+ #
72
+ def before_save() end
73
+ # This is called directly after the active record is saved.
74
+ #
75
+ def after_save() end
76
+ def save_with_callbacks
77
+ callback(:before_save)
78
+ save_without_callbacks
79
+ callback(:after_save)
80
+ end
81
+
82
+ # This is only called before update if active_record_mode is set to :update.
83
+ #
84
+ def before_update() end
85
+ # This is only called after update if active_record_mode is set to :update.
86
+ #
87
+ def after_update() end
88
+ def update_with_callbacks
89
+ callback(:before_update)
90
+ update_without_callbacks
91
+ callback(:after_update)
92
+ end
93
+
94
+ # This is only called before create if active_record_mode is set to :create(default).
95
+ #
96
+ def before_create() end
97
+ # This is only called after update if active_record_mode is set to :create(default).
98
+ #
99
+ def after_create() end
100
+ def create_with_callbacks
101
+ callback(:before_create)
102
+ create_without_callbacks
103
+ callback(:after_create)
104
+ end
105
+
106
+ private
107
+
108
+ def callback(method) #:nodoc:
109
+ run_callbacks(method)
110
+ send(method)
111
+ end
112
+
113
+ end
114
+ end
@@ -0,0 +1,53 @@
1
+ module ActiveMigration
2
+ # Dependencies are supported by ActiveMigration in this module. If you set some dependencies
3
+ # they'll be ran before Base#run is called. Specifying dependencies is easy:
4
+ #
5
+ # set_dependencies [:supplier_migration, :manufacturer_migration]
6
+ #
7
+ module Dependencies
8
+
9
+ def self.included(base)#:nodoc:
10
+ base.class_eval do
11
+ alias_method_chain :run, :dependencies
12
+ class << self
13
+ attr_accessor :dependencies, :completed
14
+ # Sets the dependencies for the migration
15
+ #
16
+ # set_dependencies [:supplier_migration, :manufacturer_migration]
17
+ def set_dependencies(dependencies)
18
+ @dependencies = dependencies
19
+ end
20
+ alias dependencies= set_dependencies
21
+ def completed? #:nodoc:
22
+ @completed
23
+ end
24
+ def is_completed #:nodoc:
25
+ @completed = true
26
+ end
27
+ alias completed completed?
28
+ def dependencies #:nodoc:
29
+ @dependencies || []
30
+ end
31
+ end
32
+ end
33
+ end
34
+
35
+ def run_with_dependencies(skip_dependencies=false) #:nodoc:
36
+ if skip_dependencies
37
+ logger.info("#{self.class.to_s} is skipping dependencies.")
38
+ run_without_dependencies
39
+ else
40
+ self.class.dependencies.each do |dependency|
41
+ migration = dependency.to_s.camelize.constantize
42
+ unless migration.completed?
43
+ logger.info("#{self.class.to_s} is running #{migration.to_s} as a dependency.")
44
+ migration.new.run
45
+ migration.is_completed
46
+ end
47
+ end
48
+ run_without_dependencies unless self.class.completed?
49
+ end
50
+ end
51
+
52
+ end
53
+ end