database_transform 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.rspec +1 -1
- data/README.md +84 -7
- data/Rakefile +12 -0
- data/database_transform.gemspec +8 -0
- data/lib/database_transform.rb +14 -2
- data/lib/database_transform/duplicate_error.rb +10 -0
- data/lib/database_transform/railtie.rb +5 -0
- data/lib/database_transform/schema.rb +123 -0
- data/lib/database_transform/schema_model_store.rb +7 -0
- data/lib/database_transform/schema_table.rb +224 -0
- data/lib/database_transform/schema_table_record_mapping.rb +31 -0
- data/lib/database_transform/schema_tables.rb +13 -0
- data/lib/database_transform/tasks/transform.rake +35 -0
- data/lib/database_transform/unsatisfied_dependency_error.rb +6 -0
- data/lib/database_transform/version.rb +1 -1
- metadata +95 -4
- data/bin/setup +0 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5260295aa827b58fa3504b901e310810dfb36a4a
|
4
|
+
data.tar.gz: f15c12e08f7b6b736af4dc86e1d97ae0b33cd394
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8c1d1982ff7532f8a7d21ab1c5541aa1169830bdcf07acecf14e8f5976d0a5d40ad498e4d48d17a3f0c32781a0657802ca1f937ce1c0cf79aeb6b737d657a069
|
7
|
+
data.tar.gz: 51a77ad4b478b66270af5f4e974da544a431886d49b146f154e485e1564314d37cf3c65e7dcfbb13155eb64f4a420cbfbe00f20fcf135f5f8568cf7d3aa098b0
|
data/.rspec
CHANGED
data/README.md
CHANGED
@@ -1,8 +1,18 @@
|
|
1
|
-
#
|
1
|
+
# Database Transform
|
2
|
+
[](https://travis-ci.org/lowjoel/database_transform)
|
3
|
+
[](https://coveralls.io/r/lowjoel/database_transform)
|
4
|
+
[](https://codeclimate.com/github/lowjoel/database_transform)
|
2
5
|
|
3
|
-
|
6
|
+
This gem will allow you to transform a database's contents across schemas with a simple DSL. This is useful when
|
7
|
+
migrating from an application written in another framework to Rails, allowing programmers to reason about how an old
|
8
|
+
schema maps to a new one.
|
4
9
|
|
5
|
-
|
10
|
+
Similar gems exist:
|
11
|
+
|
12
|
+
- [legacy_migrations](https://github.com/btelles/legacy_migrations)
|
13
|
+
- [super_migration](https://github.com/christian/super_migration)
|
14
|
+
|
15
|
+
However, they have not been updated for a few years.
|
6
16
|
|
7
17
|
## Installation
|
8
18
|
|
@@ -22,13 +32,80 @@ Or install it yourself as:
|
|
22
32
|
|
23
33
|
## Usage
|
24
34
|
|
25
|
-
|
35
|
+
Database Transform is built with ActiveRecord in mind. First, define a new database conndtion to the old database in
|
36
|
+
database.yml:
|
37
|
+
|
38
|
+
```yaml
|
39
|
+
my_old_app_production:
|
40
|
+
adapter: postgresql
|
41
|
+
host: old_server
|
42
|
+
database: my_old_app
|
43
|
+
```
|
26
44
|
|
27
|
-
|
45
|
+
Then, define the define a transform in `db/transforms/my_old_app_schema.rb`:
|
28
46
|
|
29
|
-
|
47
|
+
```ruby
|
48
|
+
class MyOldAppSchema < DatabaseTransform::Schema
|
49
|
+
transform_table :users, to: ::User, scope: proc { where('uid <> 0') } do
|
50
|
+
primary_key :uid
|
51
|
+
column :mail, to: :email
|
52
|
+
column :pass, to: :password do |password|
|
53
|
+
self.password_confirmation = password
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
transform_table :posts, to: ::Post do
|
58
|
+
primary_key :post_id
|
59
|
+
column :uid, to: :user, null: false do |uid|
|
60
|
+
Source::User.transform(uid)
|
61
|
+
end
|
62
|
+
column :content
|
63
|
+
save unless: proc { content.empty? }
|
64
|
+
end
|
65
|
+
end
|
66
|
+
```
|
30
67
|
|
31
|
-
|
68
|
+
A summary of methods:
|
69
|
+
|
70
|
+
- `transform_table` tells Database Transform to perform the given transform over records in the given source table.
|
71
|
+
- The first argument is the table to transform. This can be a symbol, string, or an ActiveRecord model.
|
72
|
+
- `to` specifies the new table to transform to. This can be a symbol, string, or an ActiveRecord model.
|
73
|
+
- If either argument is a symbol or string, an ActiveRecord model is generated which allows access to the record's
|
74
|
+
data.
|
75
|
+
- Source models are found in the Source namespace, and can be used as the `posts.uid` column above.
|
76
|
+
- Destination models are found in the Destination namespace.
|
77
|
+
- In all cases, the model will have extra accessory methods:
|
78
|
+
- `transform(old_primary_key)`: This takes a primary key in the source table, and returns the transformed object.
|
79
|
+
This only returns a valid result after the object has been transformed.
|
80
|
+
- `transformed?(old_primary_key)`: This checks if the object has been transformed.
|
81
|
+
- `default_scope` allows the programmer to specify the records to transform
|
82
|
+
- `primary_key` declares the name of column with the primary key. This allows later access when relations need to be
|
83
|
+
mapped.
|
84
|
+
- `column` declares how to transform the contents of that column from the old database to the new one.
|
85
|
+
- If `to:` is omitted, then it is assumed that the transfer function is the identity function, and the column would
|
86
|
+
map across as the same name.
|
87
|
+
- If `null: false` is specified, the value assigned to the column (in `to`) will be checked for nullity.
|
88
|
+
- A block can be provided.
|
89
|
+
- If so, then the data from the old record is passed to the block as the first argument
|
90
|
+
- In the context of the block, `self` refers to the new record.
|
91
|
+
- `self` has an additional attribute `source_record` which refers to the old record. (TODO)
|
92
|
+
- `self` has an additional attribute `schema` which refers to the transformation schema. (TODO)
|
93
|
+
- `save` declares whether the new record should be saved.
|
94
|
+
- `if` and `unless` accepts a block which will be evaluated to determine if the record should be saved.
|
95
|
+
- `validate` will allow the record to be saved bypassing validations. This defaults to `true`.
|
96
|
+
|
97
|
+
Finally, execute the Rake task:
|
98
|
+
|
99
|
+
$ rake db:transform my_old_app
|
100
|
+
|
101
|
+
And the schema (`MyOldAppSchema`) and database connection (via `my_old_app_production`) will be established for you. A
|
102
|
+
few variants of the schema name will be checked:
|
103
|
+
|
104
|
+
- my_old_app
|
105
|
+
- my_old_app_production
|
106
|
+
|
107
|
+
Only the *source* schema will be annotated to use the other connection. The *destination* schema will be used through
|
108
|
+
the application's normal configuration (i.e. depends on the value of `ENV['RAILS_ENV']`.)
|
32
109
|
|
33
110
|
## Contributing
|
34
111
|
|
data/Rakefile
CHANGED
@@ -1 +1,13 @@
|
|
1
1
|
require "bundler/gem_tasks"
|
2
|
+
begin
|
3
|
+
require 'bundler/setup'
|
4
|
+
rescue LoadError
|
5
|
+
puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
|
6
|
+
end
|
7
|
+
|
8
|
+
require "rspec/core/rake_task"
|
9
|
+
|
10
|
+
RSpec::Core::RakeTask.new
|
11
|
+
|
12
|
+
task :default => :spec
|
13
|
+
task :test => :spec
|
data/database_transform.gemspec
CHANGED
@@ -19,6 +19,14 @@ Gem::Specification.new do |spec|
|
|
19
19
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
20
20
|
spec.require_paths = ['lib']
|
21
21
|
|
22
|
+
spec.add_dependency 'activesupport'
|
23
|
+
spec.add_dependency 'activerecord'
|
24
|
+
|
22
25
|
spec.add_development_dependency 'bundler', '~> 1.9'
|
23
26
|
spec.add_development_dependency 'rake', '~> 10.0'
|
27
|
+
spec.add_development_dependency 'rspec', '~> 3'
|
28
|
+
spec.add_development_dependency 'sqlite3', '~> 1.3'
|
29
|
+
|
30
|
+
spec.add_development_dependency 'coveralls'
|
31
|
+
spec.add_development_dependency 'codeclimate-test-reporter'
|
24
32
|
end
|
data/lib/database_transform.rb
CHANGED
@@ -1,5 +1,17 @@
|
|
1
|
-
require
|
1
|
+
require 'database_transform/version'
|
2
2
|
|
3
3
|
module DatabaseTransform
|
4
|
-
|
4
|
+
extend ActiveSupport::Autoload
|
5
|
+
|
6
|
+
autoload :DuplicateError
|
7
|
+
autoload :UnsatisfiedDependencyError
|
8
|
+
|
9
|
+
autoload :SchemaTables
|
10
|
+
autoload :SchemaModelStore
|
11
|
+
autoload :Schema
|
12
|
+
|
13
|
+
autoload :SchemaTableRecordMapping
|
14
|
+
autoload :SchemaTable
|
15
|
+
|
16
|
+
require 'database_transform/railtie' if defined?(Rails)
|
5
17
|
end
|
@@ -0,0 +1,123 @@
|
|
1
|
+
class DatabaseTransform::Schema
|
2
|
+
extend DatabaseTransform::SchemaModelStore
|
3
|
+
extend DatabaseTransform::SchemaTables
|
4
|
+
|
5
|
+
# Transforms a table from the source database to the new database.
|
6
|
+
#
|
7
|
+
# Specify the source table to get entries from; a proc with the transform steps can be specified.
|
8
|
+
#
|
9
|
+
# @option args [String, Symbol, Class] to The name of the destination table; if this is not specified, no tables are
|
10
|
+
# copied. This can be a class, or a symbol which will be constantised; each column mapping proc will have access to
|
11
|
+
# one instance of this table when performing a transformation.
|
12
|
+
# @option args [Array<String, Symbol>] depends An array of symbols (source tables) which must be transformed before
|
13
|
+
# the specified source table can be transformed.
|
14
|
+
# @option args [Proc] default_scope The default scope of the old table to use when transforming.
|
15
|
+
def self.transform_table(source_table, args = {}, &proc)
|
16
|
+
raise ArgumentError.new if source_table.nil?
|
17
|
+
raise DatabaseTransform::DuplicateError.new(source_table) if tables.has_key?(source_table)
|
18
|
+
|
19
|
+
source_table, args[:to] = prepare_models(source_table, args[:to])
|
20
|
+
|
21
|
+
transform = DatabaseTransform::SchemaTable.new(source_table, args[:to], args[:default_scope])
|
22
|
+
tables[source_table] = { depends: args[:depends] || [], transform: transform }
|
23
|
+
transform.instance_eval(&proc) if proc
|
24
|
+
end
|
25
|
+
|
26
|
+
class << self
|
27
|
+
private
|
28
|
+
|
29
|
+
def prepare_models(source_table, destination_table)
|
30
|
+
source_table = generate_model(const_get(:Source), source_table) unless source_table.is_a?(Class)
|
31
|
+
set_connection_for_model(source_table, deduce_connection_name)
|
32
|
+
|
33
|
+
if !destination_table.nil? && !destination_table.is_a?(Class)
|
34
|
+
destination_table = generate_model(const_get(:Destination), destination_table)
|
35
|
+
end
|
36
|
+
|
37
|
+
[source_table, destination_table]
|
38
|
+
end
|
39
|
+
|
40
|
+
def generate_model(within, table_name)
|
41
|
+
class_name = ActiveSupport::Inflector.camelize(ActiveSupport::Inflector.singularize(table_name.to_s))
|
42
|
+
within.module_eval <<-EndCode, __FILE__, __LINE__ + 1
|
43
|
+
class #{class_name} < ActiveRecord::Base
|
44
|
+
self.table_name = '#{table_name.to_s}'
|
45
|
+
end
|
46
|
+
#{class_name.to_s.singularize.camelize}
|
47
|
+
EndCode
|
48
|
+
end
|
49
|
+
|
50
|
+
# Deduces the connection name from the name of the schema class.
|
51
|
+
#
|
52
|
+
# @return [String] The name of the connection to use.
|
53
|
+
def deduce_connection_name
|
54
|
+
deduced_connection_name = ActiveSupport::Inflector.underscore(name)
|
55
|
+
return deduced_connection_name if connection_name_exists?(deduced_connection_name)
|
56
|
+
|
57
|
+
deduced_connection_name << '_production'
|
58
|
+
end
|
59
|
+
|
60
|
+
# Checks if the given connection name exists
|
61
|
+
#
|
62
|
+
# @param [String] name The name of the connection to check.
|
63
|
+
# @return [Boolean] True if the connection exists.
|
64
|
+
def connection_name_exists?(name)
|
65
|
+
ActiveRecord::Base.configurations.has_key?(name)
|
66
|
+
end
|
67
|
+
|
68
|
+
# Sets the connection for the given model, using the given connection name.
|
69
|
+
#
|
70
|
+
# @param [Class] model The model to set the connection on.
|
71
|
+
# @param [String] connection_name The name of the connection to set.
|
72
|
+
# @return [Void]
|
73
|
+
def set_connection_for_model(model, connection_name)
|
74
|
+
model.establish_connection(connection_name.to_sym)
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
# Runs the transform.
|
79
|
+
#
|
80
|
+
# @raise [DatabaseTransform::UnsatisfiedDependencyError] When the dependencies for a table cannot be satisfied, and no
|
81
|
+
# progress can be made.
|
82
|
+
# @return [Void]
|
83
|
+
def transform!
|
84
|
+
# The tables have dependencies; we must run them in order.
|
85
|
+
transformed = Set.new
|
86
|
+
queue = Set.new(tables.keys)
|
87
|
+
|
88
|
+
# We try to run all the transforms we can until no more can be run.
|
89
|
+
# If no more can run and the input queue is empty, we are done.
|
90
|
+
# If no more can run and the input queue is not empty, we have a dependency cycle.
|
91
|
+
begin
|
92
|
+
transformed_this_pass = transform_pass(transformed, queue)
|
93
|
+
fail DatabaseTransform::UnsatisfiedDependencyError.new(queue.to_a) if transformed_this_pass.empty? && !queue.empty?
|
94
|
+
|
95
|
+
queue -= transformed_this_pass
|
96
|
+
transformed += transformed_this_pass
|
97
|
+
end until queue.empty?
|
98
|
+
end
|
99
|
+
|
100
|
+
private
|
101
|
+
|
102
|
+
# Performs a transform over all elements of the queue who has its dependencies satisfied.
|
103
|
+
#
|
104
|
+
# @param [Array<Symbol>] transformed The models which have been transformed.
|
105
|
+
# @param [Array<Symbol>] queue The models which need to be transformed.
|
106
|
+
# @return [Set<Symbol>] The set of models which were transformed this pass.
|
107
|
+
def transform_pass(transformed, queue)
|
108
|
+
transformed_this_pass = Set.new
|
109
|
+
queue.each do |table|
|
110
|
+
# Check that all dependencies are satisfied
|
111
|
+
table_config = tables[table]
|
112
|
+
unmet_dependencies = table_config[:depends].select do |s|
|
113
|
+
!transformed.include?(s) && !transformed_this_pass.include?(s)
|
114
|
+
end
|
115
|
+
next unless unmet_dependencies.empty?
|
116
|
+
|
117
|
+
table_config[:transform].run_transform
|
118
|
+
transformed_this_pass << table
|
119
|
+
end
|
120
|
+
|
121
|
+
transformed_this_pass
|
122
|
+
end
|
123
|
+
end
|
@@ -0,0 +1,224 @@
|
|
1
|
+
# Represents a transformation from a source table to a destination table.
|
2
|
+
class DatabaseTransform::SchemaTable
|
3
|
+
# Initialises the table definition for a particular schema
|
4
|
+
#
|
5
|
+
# @param [Class] source The model class to map source records from
|
6
|
+
# @param [Class] destination The model class to map destination records to
|
7
|
+
# @param [nil, Proc] default_scope The default scope for querying the source table.
|
8
|
+
def initialize(source, destination, default_scope)
|
9
|
+
@source = source
|
10
|
+
@source.extend(DatabaseTransform::SchemaTableRecordMapping)
|
11
|
+
@destination = destination
|
12
|
+
@destination.extend(DatabaseTransform::SchemaTableRecordMapping) if @destination
|
13
|
+
@default_scope = default_scope
|
14
|
+
|
15
|
+
@primary_key = nil
|
16
|
+
@save = nil
|
17
|
+
@columns = []
|
18
|
+
end
|
19
|
+
|
20
|
+
# Declare the primary key of the source table.
|
21
|
+
#
|
22
|
+
# @param [Symbol] id The name of the column in the source table which is the primary key.
|
23
|
+
def primary_key(id)
|
24
|
+
raise DatabaseTransform::DuplicateError.new(id) if @primary_key
|
25
|
+
@primary_key = id
|
26
|
+
@source.primary_key = id
|
27
|
+
end
|
28
|
+
|
29
|
+
# Declare the mapping for the source table to the new table.
|
30
|
+
#
|
31
|
+
# This function takes source columns to provide to the mapping proc as arguments.
|
32
|
+
#
|
33
|
+
# The destination column is specified as the to hash parameter.
|
34
|
+
# If no mapping block is specified, the source column is copied to the destination column without modification. This
|
35
|
+
# is not possible if more than one source column is specified.
|
36
|
+
# If the mapping block is specified, if the destination column is specified, the result of the block is used as the
|
37
|
+
# destination column's value.
|
38
|
+
# Otherwise, the result is unused. This can be used to execute the block purely for side-effects
|
39
|
+
#
|
40
|
+
# @option options [Symbol] to The column to map to.
|
41
|
+
# @option options [Boolean] null If to is specified, this can be false which enforces that nil is never added to the
|
42
|
+
# database.
|
43
|
+
def column(*args, &block)
|
44
|
+
raise ArgumentError if args.length < 1
|
45
|
+
|
46
|
+
# Get the columns
|
47
|
+
options = args.extract_options!
|
48
|
+
source_columns = args
|
49
|
+
to_column = options.delete(:to)
|
50
|
+
to_column ||= source_columns.first unless block
|
51
|
+
|
52
|
+
validate_column_options!(source_columns, to_column, options, block)
|
53
|
+
|
54
|
+
# Store the mapping
|
55
|
+
@columns << {
|
56
|
+
from: source_columns,
|
57
|
+
to: to_column,
|
58
|
+
null: options[:null].nil? ? true : options[:null],
|
59
|
+
block: block
|
60
|
+
}
|
61
|
+
end
|
62
|
+
|
63
|
+
# Specifies a save clause.
|
64
|
+
#
|
65
|
+
# @option options [Proc] unless The record will be saved if the proc returns a false value. This cannot be used
|
66
|
+
# together with if.
|
67
|
+
# @option options [Proc] if A proc to call. The record will be saved if the proc returns a true value. This cannot be
|
68
|
+
# used together with unless.
|
69
|
+
def save(options = {})
|
70
|
+
raise ArgumentError.new('unless and if cannot be both specified') if options[:unless] && options[:if]
|
71
|
+
raise ArgumentError.new('Cannot specify a save clause twice for the same table') if @save
|
72
|
+
|
73
|
+
if options[:unless]
|
74
|
+
clause = options.delete(:unless)
|
75
|
+
options[:if] = ->(*callback_args) { !self.instance_exec(*callback_args, &clause) }
|
76
|
+
end
|
77
|
+
|
78
|
+
@save = options
|
79
|
+
end
|
80
|
+
|
81
|
+
# @api private
|
82
|
+
# To be called only by Schema#run_transform
|
83
|
+
def run_transform
|
84
|
+
before_message =
|
85
|
+
if @destination
|
86
|
+
format("-- transforming '%s' to '%s'\n", @source.table_name, @destination.table_name)
|
87
|
+
else
|
88
|
+
format("-- transforming '%s'\n", @source.table_name)
|
89
|
+
end
|
90
|
+
|
91
|
+
time_block(before_message, " -> %fs\n", &method(:transform!))
|
92
|
+
end
|
93
|
+
|
94
|
+
private
|
95
|
+
|
96
|
+
# Validates the options given to the #column method.
|
97
|
+
def validate_column_options!(source_columns, to_column, options, block)
|
98
|
+
raise ArgumentError.new unless to_column.nil? || to_column.is_a?(Symbol)
|
99
|
+
raise ArgumentError.new if !block && source_columns.length > 1
|
100
|
+
raise ArgumentError.new if options[:null] == false && !to_column
|
101
|
+
end
|
102
|
+
|
103
|
+
# Performs the given operation, timing it and printing before and after messages for executing the block.
|
104
|
+
#
|
105
|
+
# @param [String] before The message to print before the operation.
|
106
|
+
# @param [String] after The message to print after the operation. One floating point format argument is available,
|
107
|
+
# which is the time taken for the operation.
|
108
|
+
# @return The result of executing the block.
|
109
|
+
# @yield The block to time.
|
110
|
+
def time_block(before, after, &proc)
|
111
|
+
start = Time.now
|
112
|
+
$stderr.puts(before)
|
113
|
+
|
114
|
+
result = proc.call
|
115
|
+
|
116
|
+
complete = Time.now - start
|
117
|
+
$stderr.printf(after, complete)
|
118
|
+
|
119
|
+
result
|
120
|
+
end
|
121
|
+
|
122
|
+
# Performs the transform with the given parameters.
|
123
|
+
#
|
124
|
+
# @return [Void]
|
125
|
+
def transform!
|
126
|
+
# For each item in the old model
|
127
|
+
default_scope = @default_scope || @source.method(:all)
|
128
|
+
@source.instance_exec(&default_scope).each do |record|
|
129
|
+
transform_record!(record)
|
130
|
+
end
|
131
|
+
end
|
132
|
+
|
133
|
+
# Transforms one record from the source model to the destination.
|
134
|
+
#
|
135
|
+
# @param [ActiveRecord::Base] old The record to map.
|
136
|
+
# @return [Void]
|
137
|
+
def transform_record!(old)
|
138
|
+
# Instantiate a new model record
|
139
|
+
new = @destination.new if @destination
|
140
|
+
|
141
|
+
# Map the columns over
|
142
|
+
transform_record_columns!(old, new)
|
143
|
+
return if new.nil? || new.frozen?
|
144
|
+
|
145
|
+
save_transformed_record(old, new)
|
146
|
+
end
|
147
|
+
|
148
|
+
# Applies the column transforms over the old record to the new record.
|
149
|
+
#
|
150
|
+
# @param [ActiveRecord::Base] old The record to map.
|
151
|
+
# @param [ActiveRecord::Base] new The record to map to.
|
152
|
+
# @return [Void]
|
153
|
+
def transform_record_columns!(old, new)
|
154
|
+
@columns.each do |column|
|
155
|
+
fail ArgumentError.new unless column.is_a?(Hash)
|
156
|
+
|
157
|
+
new_value = transform_record_field!(old, new, column[:from], column[:block])
|
158
|
+
|
159
|
+
unless new.nil?
|
160
|
+
break if new.frozen?
|
161
|
+
next if column[:to].nil?
|
162
|
+
|
163
|
+
assign_record_field!(old, new, column, new_value)
|
164
|
+
end
|
165
|
+
end
|
166
|
+
end
|
167
|
+
|
168
|
+
# Transforms one record's field.
|
169
|
+
#
|
170
|
+
# @param [ActiveRecord::Base] source The source row to map the values for.
|
171
|
+
# @param [ActiveRecord::Base] destination The destination row to map the values to.
|
172
|
+
# @param [Array<Symbol>] from The source columns to be used to map to the destination column.
|
173
|
+
# @param [Proc, nil] block The block to transform the source values to the destination value.
|
174
|
+
# @return The result of applying the transform over the input values.
|
175
|
+
def transform_record_field!(source, destination, from = nil, block = nil)
|
176
|
+
# Get the old record column values (can be a block taking multiple arguments)
|
177
|
+
new_values = from.map { |k| source.send(k) }
|
178
|
+
|
179
|
+
if block.nil?
|
180
|
+
# We have to ensure the value is a scalar
|
181
|
+
new_values.first
|
182
|
+
elsif destination
|
183
|
+
# Map the value if necessary.
|
184
|
+
# TODO: Provide the schema instance to the callback.
|
185
|
+
# We should ensure that the block has a way to access the schema.
|
186
|
+
destination.instance_exec(*new_values, &block)
|
187
|
+
else
|
188
|
+
# Call the proc
|
189
|
+
block.call(*new_values)
|
190
|
+
end
|
191
|
+
end
|
192
|
+
|
193
|
+
# Assigns the transformed value to the new record, raising an argument error if the field was declared to not be
|
194
|
+
# nullable and the value is null.
|
195
|
+
#
|
196
|
+
# @param [ActiveRecord::Base] old The source row to map the values for.
|
197
|
+
# @param [ActiveRecord::Base] new The destination row to map the values to.
|
198
|
+
# @param [Hash] column The column mapping definition being used.
|
199
|
+
# @param new_value The new value to assign to the field.
|
200
|
+
# @return [Void]
|
201
|
+
def assign_record_field!(old, new, column, new_value)
|
202
|
+
if new_value.nil? && column[:null] == false
|
203
|
+
old_value = @primary_key ? old.send(@primary_key) : old.inspect
|
204
|
+
raise ArgumentError.new("Key #{column[:from]} for row #{old_value} in #{@source.table_name} maps to null for "\
|
205
|
+
'non-nullable column')
|
206
|
+
end
|
207
|
+
|
208
|
+
new.send("#{column[:to]}=", new_value)
|
209
|
+
end
|
210
|
+
|
211
|
+
# Saves the newly transformed record, and then memoizes the transformed value.
|
212
|
+
#
|
213
|
+
# @param [ActiveRecord::Base] old The source row to map the values for.
|
214
|
+
# @param [ActiveRecord::Base] new The destination row to map the values to.
|
215
|
+
# @return [Void]
|
216
|
+
def save_transformed_record(old, new)
|
217
|
+
# Save. Skip if the conditional callback is given
|
218
|
+
return if @save && @save[:if] && !new.instance_exec(&@save[:if])
|
219
|
+
|
220
|
+
# TODO: Make validation optional using the save clause.
|
221
|
+
new.save!(validate: false) unless new.destroyed?
|
222
|
+
@source.memoize_transform(old.send(@primary_key), new) if @primary_key
|
223
|
+
end
|
224
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
module DatabaseTransform::SchemaTableRecordMapping
|
2
|
+
# Obtains the result of transforming the record with the given primary key.
|
3
|
+
#
|
4
|
+
# @param old_primary_key The primary key of the record to obtain the result for.
|
5
|
+
# @raise [ActiveRecord::RecordNotFound] When the primary has not been transformed, or the primary key does not exist.
|
6
|
+
# @return The new record after transformation.
|
7
|
+
def transform(old_primary_key)
|
8
|
+
@transformed ||= {}
|
9
|
+
unless @transformed.has_key?(old_primary_key)
|
10
|
+
raise ActiveRecord::RecordNotFound.new("Key #{old_primary_key} in #{table_name}")
|
11
|
+
end
|
12
|
+
|
13
|
+
@transformed[old_primary_key]
|
14
|
+
end
|
15
|
+
|
16
|
+
# Checks if the given primary key has been transformed.
|
17
|
+
#
|
18
|
+
# @param old_primary_key The primary key of the record to obtain the result for.
|
19
|
+
# @return [Boolean] True if the record has been transformed.
|
20
|
+
def transformed?(old_primary_key)
|
21
|
+
@transformed ||= {}
|
22
|
+
@transformed.has_key?(old_primary_key)
|
23
|
+
end
|
24
|
+
|
25
|
+
# @api private
|
26
|
+
# Called by TableTransform#run_transform
|
27
|
+
def memoize_transform(old_primary_key, result)
|
28
|
+
@transformed ||= {}
|
29
|
+
@transformed[old_primary_key] = result
|
30
|
+
end
|
31
|
+
end
|
@@ -0,0 +1,35 @@
|
|
1
|
+
# This class just loads the file containing the schema definition and hands off control to the schema.
|
2
|
+
class DatabaseTransform::Transform
|
3
|
+
def initialize(args)
|
4
|
+
@schema = args.schema_name
|
5
|
+
@extra_args = args.extras
|
6
|
+
run
|
7
|
+
end
|
8
|
+
|
9
|
+
def run
|
10
|
+
import_schema
|
11
|
+
schema = @schema.constantize.new
|
12
|
+
|
13
|
+
ActiveRecord::Base.logger = Logger.new('log/import.log')
|
14
|
+
schema.transform!
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def import_schema
|
20
|
+
schema_file = @schema.underscore
|
21
|
+
begin
|
22
|
+
return require(File.join(Rails.root, 'db', 'transforms', schema_file))
|
23
|
+
rescue LoadError
|
24
|
+
end
|
25
|
+
|
26
|
+
require (File.join(Rails.root, 'db', 'transforms', schema_file, schema_file))
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
namespace :db do
|
31
|
+
desc 'Transform old database schemas'
|
32
|
+
task :transform, [:schema_name] => :environment do |_, args|
|
33
|
+
DatabaseTransform::Transform.new(args)
|
34
|
+
end
|
35
|
+
end
|
metadata
CHANGED
@@ -1,15 +1,43 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: database_transform
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Joel Low
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-05-
|
11
|
+
date: 2015-05-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: activesupport
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '0'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '0'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: activerecord
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
13
41
|
- !ruby/object:Gem::Dependency
|
14
42
|
name: bundler
|
15
43
|
requirement: !ruby/object:Gem::Requirement
|
@@ -38,12 +66,67 @@ dependencies:
|
|
38
66
|
- - "~>"
|
39
67
|
- !ruby/object:Gem::Version
|
40
68
|
version: '10.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: rspec
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '3'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - "~>"
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '3'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: sqlite3
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - "~>"
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '1.3'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - "~>"
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '1.3'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: coveralls
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
type: :development
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - ">="
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
name: codeclimate-test-reporter
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
type: :development
|
119
|
+
prerelease: false
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
121
|
+
requirements:
|
122
|
+
- - ">="
|
123
|
+
- !ruby/object:Gem::Version
|
124
|
+
version: '0'
|
41
125
|
description: Transforms the data of a database from an old schema to a new one.
|
42
126
|
email:
|
43
127
|
- joel@joelsplace.sg
|
44
128
|
executables:
|
45
129
|
- console
|
46
|
-
- setup
|
47
130
|
extensions: []
|
48
131
|
extra_rdoc_files: []
|
49
132
|
files:
|
@@ -55,9 +138,17 @@ files:
|
|
55
138
|
- README.md
|
56
139
|
- Rakefile
|
57
140
|
- bin/console
|
58
|
-
- bin/setup
|
59
141
|
- database_transform.gemspec
|
60
142
|
- lib/database_transform.rb
|
143
|
+
- lib/database_transform/duplicate_error.rb
|
144
|
+
- lib/database_transform/railtie.rb
|
145
|
+
- lib/database_transform/schema.rb
|
146
|
+
- lib/database_transform/schema_model_store.rb
|
147
|
+
- lib/database_transform/schema_table.rb
|
148
|
+
- lib/database_transform/schema_table_record_mapping.rb
|
149
|
+
- lib/database_transform/schema_tables.rb
|
150
|
+
- lib/database_transform/tasks/transform.rake
|
151
|
+
- lib/database_transform/unsatisfied_dependency_error.rb
|
61
152
|
- lib/database_transform/version.rb
|
62
153
|
homepage: https://github.com/lowjoel/database_transform
|
63
154
|
licenses:
|